# Technical Documentation: grafana/tempo > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/grafana/tempo) ### Overview (README.md)

Tempo Logo

Latest Release License Docker Pulls Slack Community Forum Go Report Card

Grafana Tempo is an open source, easy-to-use, and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Prometheus, and Loki. ## Business value of distributed tracing Distributed tracing helps teams quickly pinpoint performance issues and understand the flow of requests across services. The Traces Drilldown UI simplifies this process by offering a user-friendly interface to view and analyze trace data, making it easier to identify and resolve issues without needing to write complex queries. Refer to [Use traces to find solutions](https://grafana.com/docs/tempo/latest/introduction/solutions-with-traces/) to learn more about how you can use distributed tracing to investigate and solve issues. ## Traces Drilldown UI: A better way to get value from your tracing data We are excited to introduce the [Traces Drilldown](https://github.com/grafana/traces-drilldown) (formerly Explore Traces) app as part of the Grafana Explore suite. This app provides a queryless and intuitive experience for analyzing tracing data, allowing teams to quickly identify performance issues, latency bottlenecks, and errors without needing to write complex queries or use TraceQL. Key Features: - **Intuitive Trace Analysis**: Spot slow or error-prone traces with easy, point-and-click interactions. - **RED Metrics Overview**: Use Rate, Errors, and Duration metrics to highlight performance issues. - **Automated Comparison**: Identify problematic attributes with automatic trace comparison. - **Simplified Visualizations**: Access rich visual data without needing to construct TraceQL queries. To learn more see the following links: - [Traces Drilldown repo](https://github.com/grafana/traces-drilldown) - [Traces Drilldown documentation](https://grafana.com/docs/grafana/latest/explore/simplified-exploration/traces/) - [Demo video](https://www.youtube.com/watch?v=a3uB1C2oHA4) ## TraceQL Tempo implements [TraceQL](https://grafana.com/docs/tempo/latest/traceql/), a traces-first query language inspired by LogQL and PromQL, which enables targeted queries or rich UI-driven analyses. ### TraceQL metrics [TraceQL metrics](https://grafana.com/docs/tempo/latest/traceql/metrics-queries/) is an experimental feature in Grafana Tempo that creates metrics from traces. Metric queries extend trace queries by applying a function to trace query results. This powerful feature allows for ad hoc aggregation of any existing TraceQL query by any dimension available in your traces, much in the same way that LogQL metric queries create metrics from logs. Tempo is Jaeger, Zipkin, Kafka and OpenTelemetry compatible. It ingests batches in any of the mentioned formats, buffers them, and then writes them to Azure, GCS, S3, or local disk. As such, it's robust, cheap, and easy to operate. ## Getting started with Tempo - [Get started documentation](https://grafana.com/docs/tempo/latest/getting-started/) - [Deployment Examples](./example) - [Docker Compose](./example/docker-compose) - [Helm](./example/helm) - [Jsonnet](./example/tk) ## Further reading To learn more about Tempo, consult the following documents & talks: - [How to get started with Tempo with Joe Elliott (video)](https://www.youtube.com/watch?v=zDrA7Ly3ovU) - [Grafana blog posts about Tempo](https://grafana.com/tags/tempo/) - [New in Grafana Tempo 2.0: Apache Parquet as the default storage format, support for TraceQL][tempo_20_announce] - [Get to know TraceQL: A powerful new query language for distributed tracing][traceql-post] [tempo_20_announce]: https://grafana.com/blog/2023/02/01/new-in-grafana-tempo-2.0-apache-parquet-as-the-default-storage-format-support-for-traceql/ [traceql-post]: https://grafana.com/blog/2023/02/07/get-to-know-traceql-a-powerful-new-query-language-for-distributed-tracing/ ## Getting help If you have any questions or feedback regarding Tempo: - Grafana Labs hosts a [forum](https://community.grafana.com/c/grafana-tempo/40) for Tempo. This is a great place to post questions and search for answers. - Ask a question on the [Tempo Slack channel](https://grafana.slack.com/archives/C01D981PEE5). - [File an issue](https://github.com/grafana/tempo/issues/new/choose) for bugs, issues and feature suggestions. - UI issues should be filed with [Grafana](https://github.com/grafana/grafana/issues/new/choose). ## OpenTelemetry Tempo's receiver layer, wire format and storage format are all based directly on [standards](https://github.com/open-telemetry/opentelemetry-proto) and [code](https://github.com/open-telemetry/opentelemetry-collector) established by [OpenTelemetry](https://opentelemetry.io/). We support open standards at Grafana! Check out the [Integration Guides](https://grafana.com/docs/tempo/latest/guides/instrumentation/) to see examples of OpenTelemetry instrumentation with Tempo. ## Other components ### tempo-vulture [tempo-vulture](https://github.com/grafana/tempo/tree/main/cmd/tempo-vulture) is Tempo's bird themed consistency checking tool. It writes traces to Tempo and then queries them back in a variety of ways. ### tempo-cli [tempo-cli](https://github.com/grafana/tempo/tree/main/cmd/tempo-cli) is the place to put any utility functionality related to Tempo. See [Documentation](https://grafana.com/docs/tempo/latest/operations/tempo_cli/) for more info. ## License Grafana Tempo is distributed under [AGPL-3.0-only](LICENSE). For Apache-2.0 exceptions, see [LICENSING.md](LICENSING.md). --- ### README (`docs/README.md`) # Tempo Documentation

Tempo Logo
Please see the [Documentation Site](https://grafana.com/docs/tempo/latest/) The files in this directory are used to generate it but consequently the links don't work in Github. To contribute to these docs, please check `Documentation` section in [CONTRIBUTING.md](/CONTRIBUTING.md). --- ### Design Proposals README (`docs/design-proposals/README.md`) # Design Proposals This directory contains design proposals to modify or extend Grafana Tempo. Everyone is allowed and encouraged to create new design proposals. A design proposal is especially recommended for: - Controversial changes - Large new features - Changes with a wide impact Smaller changes can still be suggested using [the feature request template](https://github.com/grafana/tempo/issues/new?template=feature_request.md). ## How to create a new design proposal We strive to establish a lightweight process: it should be easy to propose a change and facilitate a discussion. The process should help guide this discussion and ensure document are well archived, but it should not hinder it in any way. If you are considering starting a design proposal but aren't sure yet if it will be useful, feel free to open a GitHub issue to start a discussion or leave a message in the #tempo Slack channel. Process: - Create a new document in this directory - The new document should be named `- `. For example `2022-01 Metrics-generator` - Please add at least the following front-matter: ```markdown --- Author: <name> (@<github handle>) Created: <date this document was created> Last updated: <date this document was last updated> --- # <title> ... ``` - Once the document is ready to be shared, create a Pull Request. --- ### Sources Tempo Index (`docs/sources/tempo/_index.md`) --- title: Grafana Tempo description: Grafana Tempo is an open source distributed tracing backend. aliases: - /docs/tempo/ cascade: GRAFANA_VERSION: next hero: title: Grafana Tempo level: 1 image: /static/assets/img/blog/tempo.png width: 110 height: 110 description: >- Grafana Tempo is an open-source, easy-to-use, and high-scale distributed tracing backend. Tempo lets you search for traces, generate metrics from spans, and link your tracing data with logs and metrics. cards: title_class: pt-0 lh-1 items: - title: Learn about tracing href: ./introduction/ description: What is distributed tracing? Learn about traces and how you can use them, how you can instrument your app for tracing, and how you can visualize tracing data in Grafana. - title: Solutions and use cases href: ./solutions-with-traces/ description: Learn how tracing data can help you understand application insights and performance as well as help triage issues in your services and applications. - title: Set up for tracing href: ./set-up-for-tracing/ description: Plan your deployment to meet your needs, deploy Tempo, test your installation, and instrument your app or services for traces. - title: Manage Tempo href: ./operations/ description: Learn about Tempo architecture, best practices, Parquet backend, dedicated attribute columns, and more. - title: Metrics and tracing href: ./metrics-from-traces/ description: Learn about metrics created from traces, including the metrics-generator processor and TraceQL metrics. - title: Query with TraceQL href: ./traceql/ description: Inspired by PromQL and LogQL, TraceQL is a query language designed for selecting traces in Tempo. This query language lets you precisely and easily select spans and jump directly to the spans fulfilling the specified conditions. --- {{< docs/hero-simple key="hero" >}} --- ## Overview Distributed tracing visualizes the lifecycle of a request as it passes through a set of applications. Tempo is cost-efficient and only requires an object storage to operate. Tempo is deeply integrated with Grafana, Mimir, Prometheus, and Loki. You can use Tempo with open source tracing protocols, including Jaeger, Zipkin, or OpenTelemetry. {{< figure src="/media/docs/tempo/intro/trace_custom_metrics_dash.png" alt="Trace visualization in Grafana" class="w-100p" link-class="w-fit mx-auto d-flex flex-direction-column" >}} Tempo integrates well with a number of open source tools: - **Grafana** ships with native support using the built-in [Tempo data source](/docs/grafana/latest/datasources/tempo/). - **Grafana Loki**, with its powerful query language LogQL v2 lets you filter requests that you care about, and jump to traces using the [Derived fields support in Grafana](/docs/grafana/latest/datasources/loki). - **Prometheus exemplars** let you jump from Prometheus metrics to Tempo traces by clicking on recorded exemplars. {{< docs/play title="Traces Drilldown" url="https://play.grafana.org/a/grafana-exploretraces-app/explore" >}} ## Explore {{< card-grid key="cards" type="simple" >}} --- ### Modules Frontend Docs Config Overview (`modules/frontend/docs/config-overview.md`) # Tempo Configuration Overview This is an orientation map of Tempo's configuration. It explains what each top-level configuration block controls so you can find the right area quickly. For exact key names, nesting, and default values, request the "config-reference" doc, which lists every option. ## How Tempo configuration works Tempo ships as a single binary that supports two deployment modes. In monolithic mode (`-target=all`, the default), all components run in one process. In microservices mode, each component runs as its own process with its own `-target` (for example, `-target=distributor` or `-target=querier`). The `target` option selects which components a process runs. All components read from the same configuration file, so a block such as `storage` applies wherever it's relevant. Configuration is set in a YAML file and may be overridden by command-line flags. Flag names mirror the YAML path, for example `query_frontend.mcp_server.enabled` becomes `--query-frontend.mcp-server.enabled`. ## Top-level configuration blocks - `target`: Which component(s) this process runs. Use `all` for monolithic mode. - `http_api_prefix`: Optional prefix applied to Tempo's HTTP API paths. - `memory`: Process memory management, including automatic memory limit settings. - `server`: HTTP and gRPC listen addresses, ports, TLS, and logging for the public API. - `internal_server`: An optional separate server for internal endpoints. - `distributor`: Receives incoming spans. Configures the OTLP, Jaeger, and Zipkin receivers and any forwarders. - `querier`: Executes query jobs dispatched by the query frontend, fetching trace data from live-stores (recent data) and object storage (historical data). - `query_frontend`: Splits, queues, and shards incoming queries. - `metrics_generator`: Generates metrics from incoming spans, such as service graphs and span metrics, and remote-writes them. Configures processors and generator storage. - `ingest`: Configures the ingestion queue used between components. Microservices mode only. - `block_builder`: Consumes ingested trace data and builds storage blocks. Microservices mode only. - `live_store`: Holds and serves recent trace data before it's flushed to long-term storage. - `live_store_client`: Client settings for querying the live store. - `storage`: The trace backend (object storage such as S3, GCS, or Azure, or local), plus the write-ahead log, connection pool, and caching. - `overrides`: Per-tenant limits and settings, such as ingestion rate limits, read limits, and metrics-generator options. Supports runtime-loaded user-configurable overrides. - `memberlist`: Gossip-based ring membership shared by components that use rings. - `usage_report`: Anonymous usage reporting settings. - `cache`: Named cache backends (for example, Memcached or Redis) referenced elsewhere. - `backend_scheduler`: Schedules background maintenance work such as compaction. - `backend_scheduler_client`: Client settings for connecting to the backend scheduler. - `backend_worker`: Runs background maintenance jobs assigned by the scheduler. ## Finding exact options Request the "config-reference" doc for the complete list of options with defaults. Most installations set only 10 to 20 options; start from the examples and override what you need. --- ### Sources Helm Charts Tempo Distributed Index (`docs/sources/helm-charts/tempo-distributed/_index.md`) --- title: "Grafana tempo-distributed Helm chart documentation" menuTitle: "Grafana tempo-distributed Helm chart" description: "Technical documentation for the tempo-distributed Helm chart" weight: 1 keywords: - Grafana Tempo - Grafana traces --- # Grafana tempo-distributed Helm chart documentation {{< admonition type="note" >}} The `tempo-distributed` Helm chart is now maintained by the community. The chart has moved to the [grafana-community/helm-charts](https://github.com/grafana-community/helm-charts/tree/main/charts/tempo-distributed) repository. For support and contributions, refer to the [community Helm charts repository](https://github.com/grafana-community/helm-charts). {{< /admonition >}} The tempo-distributed Helm chart for Grafana Tempo allows you to configure, install, and upgrade Grafana Tempo within a Kubernetes cluster. The chart is maintained by the community. {{< section menuTitle="true" >}} --- ### Sources Tempo Api Docs Index (`docs/sources/tempo/api_docs/_index.md`) --- title: Tempo HTTP API description: Grafana Tempo exposes an API for pushing and querying traces, and operating the cluster itself. menuTitle: API weight: 800 --- # Tempo HTTP API Tempo exposes an API for pushing and querying traces, and operating the cluster itself. For the sake of clarity, API endpoints are grouped by service. These endpoints are exposed both when running Tempo in microservices and monolithic mode: - **microservices**: each service exposes its own endpoints - **monolithic**: the Tempo process exposes all API endpoints for the services running internally For externally supported gRPC API, [refer to Tempo gRPC API](#tempo-grpc-api). ## Endpoints | API | Service | Type | Endpoint | | ------------------------------------------------------------------------------------- | ----------------------------------------- | ---- | --------------------------------------------------------- | | [Readiness probe](#readiness-probe) | _All services_ | HTTP | `GET /ready` | | [Metrics](#metrics) | _All services_ | HTTP | `GET /metrics` | | [Pprof](#pprof) | _All services_ | HTTP | `GET /debug/pprof` | | [Ingest traces](#ingest) | Distributor | - | See section for details | | [Querying traces by id](#query) | Query-frontend | HTTP | `GET /api/traces/<traceID>` | | [Querying traces by id V2](#query-v2) | Query-frontend | HTTP | `GET /api/v2/traces/<traceID>` | | [Searching traces](#search) | Query-frontend | HTTP | `GET /api/search?<params>` | | [Search tag names](#search-tags) | Query-frontend | HTTP | `GET /api/search/tags` | | [Search tag names V2](#search-tags-v2) | Query-frontend | HTTP | `GET /api/v2/search/tags` | | [Search tag values](#search-tag-values) | Query-frontend | HTTP | `GET /api/search/tag/<tag>/values` | | [Search tag values V2](#search-tag-values-v2) | Query-frontend | HTTP | `GET /api/v2/search/tag/<tag>/values` | | [TraceQL Metrics](#traceql-metrics) | Query-frontend | HTTP | `GET /api/metrics/query_range` | | [TraceQL Metrics (instant)](#instant) | Query-frontend | HTTP | `GET /api/metrics/query` | | [Traces diff](#traces-diff) (\*) | Query-frontend | HTTP | `POST /api/v2/traces/diff` | | [Query Echo Endpoint](#query-echo-endpoint) | Query-frontend | HTTP | `GET /api/echo` | | [Overrides API](#overrides-api) | Query-frontend | HTTP | `GET,POST,PATCH,DELETE /api/overrides` | | Memberlist | Distributor, Querier, Live store | HTTP | `GET /memberlist` | | [Prepare live store partition downscale](#prepare-live-store-partition-downscale) | Live store | HTTP | `GET,POST,DELETE /live-store/prepare-partition-downscale` | | [Prepare live store downscale](#prepare-live-store-downscale) | Live store | HTTP | `GET,POST,DELETE /live-store/prepare-downscale` | | [Usage Metrics](#usage-metrics) | Distributor | HTTP | `GET /usage_metrics` | | [Distributor ring status](#distributor-ring-status) (\*) | Distributor | HTTP | `GET /distributor/ring` | | [Live-store ring status](#live-store-ring-status) | Distributor, Querier | HTTP | `GET /live-store/ring` | | [Partition ring status](#partition-ring-status) | Distributor, Querier, Live store | HTTP | `GET /partition-ring` | | [Status](#status) | Status | HTTP | `GET /status` | | [List build information](#list-build-information) | Status | HTTP | `GET /api/status/buildinfo` | | [Backend scheduler job status](#backend-scheduler-job-status) | Backend scheduler | HTTP | `GET /status/backendscheduler` | | [MCP Server](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/mcp-server) (\*) | MCP | | `/api/mcp` | _(\*) This endpoint isn't always available, check the specific section for more details._ ### Readiness probe ``` GET /ready ``` Returns status code 200 when Tempo is ready to serve traffic. ### Metrics ``` GET /metrics ``` Returns the metrics for the running Tempo service in the Prometheus exposition format. ### Pprof ``` GET /debug/pprof/heap GET /debug/pprof/block GET /debug/pprof/profile GET /debug/pprof/trace GET /debug/pprof/goroutine GET /debug/pprof/mutex ``` Returns the runtime profiling data in the format expected by the pprof visualization tool. There are many things which can be profiled using this including heap, trace, goroutine, etc. For more information, refer to the official documentation of [pprof](https://golang.org/pkg/net/http/pprof/). ### Ingest The Tempo distributor uses the OpenTelemetry Collector receivers as a foundation to ingest trace data. These APIs are meant to be consumed by the corresponding client SDK or pipeline component, such as Grafana Alloy, OpenTelemetry Collector, or Jaeger Agent. | Protocol | Type | Docs | | ------------- | -------------- | -------------------------------------------------------------------------- | | OpenTelemetry | gRPC | [Link](https://opentelemetry.io/docs/specs/otlp/) | | OpenTelemetry | HTTP | [Link](https://opentelemetry.io/docs/specs/otlp/) | | Jaeger | Thrift Compact | [Link](https://www.jaegertracing.io/docs/latest/apis/#span-reporting-apis) | | Jaeger | Thrift Binary | [Link](https://www.jaegertracing.io/docs/latest/apis/#span-reporting-apis) | | Jaeger | Thrift HTTP | [Link](https://www.jaegertracing.io/docs/latest/apis/#span-reporting-apis) | | Jaeger | gRPC | [Link](https://www.jaegertracing.io/docs/latest/apis/#span-reporting-apis) | | Zipkin | HTTP | [Link](https://zipkin.io/zipkin-api/) | For information on how to push spans using OTLP/HTTP or OTLP/gRPC (for debugging purposes), refer to [Push spans](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/pushing-spans-with-http/). If you are using Grafana Enterprise Traces (GET), then it only supports OpenTelemetry (OTLP): | Protocol | Type | Docs | | ------------- | ---- | ------------------------------------------------- | | OpenTelemetry | gRPC | [Link](https://opentelemetry.io/docs/specs/otlp/) | | OpenTelemetry | HTTP | [Link](https://opentelemetry.io/docs/specs/otlp/) | ### Query The following request is used to retrieve a trace from the query frontend service in a microservices deployment or the Tempo endpoint in a monolithic mode deployment. ``` GET /api/traces/<traceid>?start=<start>&end=<end> ``` Parameters: - `start = (unix epoch seconds)` Optional. Along with `end` define a time range from which traces should be returned. - `end = (unix epoch seconds)` Optional. Along with `start` define a time range from which traces should be returned. Providing both `start` and `end` includes traces for the specified time range only. If the parameters aren't provided then Tempo checks for the trace across all blocks in backend. If the parameters are provided, it only checks in the blocks within the specified time range, this can result in trace not being found or partial results if it doesn't fall in the specified time range. The following query API is also provided on the querier service for _debugging_ purposes. ``` GET /querier/api/traces/<traceid>?mode=xxxx&blockStart=0000&blockEnd=FFFF&start=<start>&end=<end> ``` Parameters: - `mode = (blocks|ingesters|all)` Specifies whether the querier should look for the trace in blocks, live stores, or both (all). The `ingesters` mode queries live stores for recent data. Default = `all` - `blockStart = (GUID)` Specifies the blockID start boundary. If specified, the querier only searches blocks with IDs > blockStart. Default = `00000000-0000-0000-0000-000000000000` Example: `blockStart=12345678-0000-0000-1235-000001240000` - `blockEnd = (GUID)` Specifies the blockID finish boundary. If specified, the querier only searches blocks with IDs < `blockEnd`. Default = `FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF` Example: `blockStart=FFFFFFFF-FFFF-FFFF-FFFF-456787652341` - `start = (unix epoch seconds)` Optional. Along with `end` define a time range from which traces should be returned. - `end = (unix epoch seconds)` Optional. Along with `start` define a time range from which traces should be returned. Providing both `start` and `end` includes blocks for the specified time range only. This API isn't meant to be used directly unless for debugging the sharding functionality of the query frontend. **Returns** By default, this endpoint returns a mostly compatible [OpenTelemetry](https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/trace/v1) JSON, but if it can also send OpenTelemetry proto if `Accept: application/protobuf` is passed. ### Query V2 The following request is used to retrieve a trace from the query frontend service in a microservices deployment or the Tempo endpoint in a monolithic mode deployment. ``` GET /api/v2/traces/<traceid>?start=<start>&end=<end> ``` Parameters: - `start = (unix epoch seconds)` Optional. Along with `end` define a time range from which traces should be returned. - `end = (unix epoch seconds)` Optional. Along with `start` define a time range from which traces should be returned. Providing both `start` and `end` includes traces for the specified time range only. If the parameters aren't provided then Tempo checks for the trace across all blocks in backend. If the parameters are provided, it only checks in the blocks within the specified time range, this can result in trace not being found or partial results if it doesn't fall in the specified time range. {{< admonition type="warning" >}} Span pruning is experimental. The `span_pruning*` parameters, their behavior, and the response format may change in future releases. {{< /admonition >}} - `span_pruning = (boolean)` Optional. Enables span pruning post-processing on the returned trace, collapsing similar leaf spans into a single summary span. If the query-frontend has `span_pruning_enabled_by_default: true` set under `trace_by_id`, pruning defaults to enabled when this parameter is omitted, but an explicit `span_pruning` value in the request (true or false) always takes precedence. Pruning is applied even if the returned trace is partial (for example, because it exceeds the maximum trace size); the resulting summary spans reflect only the spans present in the partial trace. Default = `false` - `span_pruning_group_by = (comma-separated list of attribute glob patterns)` Optional. Overrides the attribute patterns used to decide which leaf spans belong in the same aggregation group, for example `db.*,http.method`. Spans must share the span name and have identical values for every matched attribute to be grouped. Only used when pruning is enabled for the request, either via `span_pruning=true` or `span_pruning_enabled_by_default`. - `span_pruning_min_spans = (integer)` Optional. Overrides the minimum number of similar spans required in a group before they're aggregated into a single summary span. Groups smaller than this threshold are left unpruned. Only used when pruning is enabled for the request, either via `span_pruning=true` or `span_pruning_enabled_by_default`. Default = `5` - `span_pruning_max_parent_depth = (integer)` Optional. Overrides how many ancestor levels above the aggregated leaf spans can also be aggregated. Use `0` to aggregate only leaves, or `-1` for unlimited depth. Only used when pruning is enabled for the request, either via `span_pruning=true` or `span_pruning_enabled_by_default`. Default = `1` - `q = (TraceQL filter)` Optional. A single TraceQL spanset filter (for example `{ span.http.status_code = 500 }`) that returns only the matching spans. When it drops spans, the response status is `PARTIAL` to signal a subset. Only a single `{ ... }` filter is supported: pipelines, structural operators, metrics queries, trace-level intrinsics, and trace-scoped attributes return a `400`, and an absent or empty `q` returns the full trace. - `keep_hierarchy = (bool)` Optional. When `true`, the response also includes the ancestor path from the root spans to each matched span, so the result is still a complete hierarchy that can be rendered as a waterfall. Defaults to `false` (only the spans matching `q`). Ignored when `q` isn't set. - `match_depth = (int)` Optional. How many levels of descendants to keep below each span matched by `q`, independent of `keep_hierarchy`. Use `-1` for the full subtree, `0` for the matched spans alone, or `n` (`n >= 1`) to keep levels 1 (direct children) through `n`. Values below `-1` are invalid and return a `400`. Ignored when `q` isn't set. Default = `0` - `ancestor_depth = (int)` Optional. How many levels of ancestors to keep above each matched span. Use `-1` for the whole path to the root, `0` for none, or `n` (`n >= 1`) to keep levels 1 (immediate parent) through `n`. Values below `-1` are invalid and return a `400`. Only read when `keep_hierarchy` is `true`; otherwise it's ignored without being validated. Default = `-1` The following query API is also provided on the querier service for _debugging_ purposes. ``` GET /querier/api/v2/traces/<traceid>?mode=xxxx&blockStart=0000&blockEnd=FFFF&start=<start>&end=<end> ``` Parameters: - `mode = (blocks|ingesters|all)` Specifies whether the querier should look for the trace in blocks, live stores, or both (all). The `ingesters` mode queries live stores for recent data. Default = `all` - `blockStart = (GUID)` Specifies the blockID start boundary. If specified, the querier only searches blocks with IDs > blockStart. Default = `00000000-0000-0000-0000-000000000000` Example: `blockStart=12345678-0000-0000-1235-000001240000` - `blockEnd = (GUID)` Specifies the blockID finish boundary. If specified, the querier only searches blocks with IDs < blockEnd. Default = `FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF` Example: `blockStart=FFFFFFFF-FFFF-FFFF-FFFF-456787652341` - `start = (unix epoch seconds)` Optional. Along with `end` define a time range from which traces should be returned. - `end = (unix epoch seconds)` Optional. Along with `start` define a time range from which traces should be returned. Providing both `start` and `end` includes blocks for the specified time range only. **Returns** By default, this endpoint returns Query response with a [OpenTelemetry](https://github.com/open-telemetry/opentelemetry-proto/tree/main/opentelemetry/proto/trace/v1) JSON trace. Other formats can be requested using the `Accept` header: - `Accept: application/protobuf` - Returns OpenTelemetry proto format - `Accept: application/vnd.grafana.llm` - Returns a simplified JSON format optimized for LLM consumption. This format is subject to change and shouldn't be relied on for programmatic use. ### Search The Tempo Search API finds traces based on span and process attributes (tags and values). When performing a search, Tempo does a massively parallel search over the given time range, and takes the first N results. Even identical searches differs due to things like machine load and network latency. TraceQL follows the same behavior. The API is available in the query frontend service in a microservices deployment, or the Tempo endpoint in a monolithic mode deployment. The following request is used to find traces containing spans from service `myservice` and the URL contains `api/myapi`. ``` GET /api/search?tags=service.name%3Dmyservice%20http.url%3Dapi%2Fmyapi ``` The URL query parameters support the following values: **Parameters for TraceQL Search** - `q = (TraceQL query)`: URL encoded [TraceQL query](../traceql). **Parameters for Tag Based Search** - `tags = (logfmt)`: `logfmt` encoding of any span-level or process-level attributes to filter on. The value is matched as a case-insensitive substring. Key-value pairs are separated by spaces. If a value contains a space, it should be enclosed within double quotes. - `minDuration = (go duration value)` Optional. Find traces with at least this duration. Duration values are of the form `10s` for 10 seconds, `100ms`, `30m`, etc. - `maxDuration = (go duration value)` Optional. Find traces with no greater than this duration. Uses the same form as `minDuration`. **Parameters supported for all searches** - `limit = (integer)` Optional. Limit the number of search results. Default is 20, but this is configurable in the querier. Refer to [Configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#querier). - `start = (unix epoch seconds)` Optional. Along with `end` define a time range from which traces should be returned. - `end = (unix epoch seconds)` Optional. Along with `start`, define a time range from which traces should be returned. Providing both `start` and `end` changes the way that Tempo searches. If the parameters aren't provided, then Tempo searches the recent trace data stored in the live store. If the parameters are provided, it searches the backend as well. - `spss = (integer)` Optional. Limit the number of spans per span-set. Default value is 3. #### Example of TraceQL search Example of how to query Tempo using curl. This query returns all traces that have their status set to error. ```bash curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ status=error }' | jq { "traces": [ { "traceID": "2f3e0cee77ae5dc9c17ade3689eb2e54", "rootServiceName": "shop-backend", "rootTraceName": "update-billing", "startTimeUnixNano": "1684778327699392724", "durationMs": 557, "spanSets": [ { "spans": [ { "spanID": "563d623c76514f8e", "startTimeUnixNano": "1684778327735077898", "durationNanos": "446979497", "attributes": [ { "key": "status", "value": { "stringValue": "error" } } ] } ], "matched": 1 } ] ], "metrics": { "totalBlocks": 13 } } ``` #### Example of tags-based search Example of how to query Tempo using curl. This query returns all traces that have a tag `service.name` containing `cartservice` and a minimum duration of 600 ms. ```bash curl -G -s http://localhost:3200/api/search --data-urlencode 'tags=service.name=cartservice' --data-urlencode minDuration=600ms | jq { "traces": [ { "traceID": "d6e9329d67b6146a", "rootServiceName": "frontend", "rootTraceName": "/cart", "startTimeUnixNano": "1634727903545000000", "durationMs": 611 }, { "traceID": "1b1ba462b409200d", "rootServiceName": "frontend", "rootTraceName": "/cart", "startTimeUnixNano": "1634727775935000000", "durationMs": 611 } ], "metrics": { "inspectedTraces": 3100, "inspectedBytes": "3811736", "totalBlocks": 3, "backendReads": "47", "backendBytes": "2415680", "additionalMetrics": { "cacheHits": "124", "cacheMisses": "18", "cacheBytes": "1396224" } } } ``` The `metrics` object on search responses includes read-side counters that help explain query cost: - `backendReads` is the number of read operations issued against object storage. - `backendBytes` is the bytes read from object storage, excluding cache hits. - `additionalMetrics` is an open-ended map keyed by stable strings (today: `cacheHits`, `cacheMisses`, `cacheBytes`, plus row-group and page inspection counters as they are populated). The keys in `additionalMetrics` are stable and additive: clients should treat unknown keys as forward compatible. ### Search tags Live store configuration `complete_block_timeout` affects how long tags are available for search. Tag search is supported in blocks as well, based on the start and end query parameters. Tag value search is also supported. This endpoint retrieves all discovered tag names that can be used in search. The endpoint is available in the query frontend service in a microservices deployment, or the Tempo endpoint in a monolithic mode deployment. The tags endpoint takes a scope that controls the kinds of tags or attributes returned. If nothing is provided, the endpoint returns all resource and span tags. ``` GET /api/search/tags?scope=<resource|span|intrinsic> ``` #### Example Example of how to query Tempo using curl. This query returns all discovered tag names. ```bash curl -G -s http://localhost:3200/api/search/tags?scope=span | jq { "tagNames": [ "host.name", "http.method", "http.status_code", "http.url", "ip", "load_generator.seq_num", "name", "region", "root_cause_error", "sampler.param", "sampler.type", "service.name", "starter", "version" ] "metrics": { "inspectedBytes": "630188" } } ``` Parameters: - `scope = (resource|span|intrinsic|event|link|instrumentation)` Optional. Specifies the scope of the tags. If not specified, it means all scopes. Default = `all` - `start = (unix epoch seconds)` Optional. Along with `end`, defines a time range from which tags should be returned. - `end = (unix epoch seconds)` Optional. Along with `start`, defines a time range from which tags should be returned. Providing both `start` and `end` includes blocks for the specified time range only. - `limit = (integer)` Optional. Limits the maximum number of tags values. - `maxStaleValues = (integer)` Optional. Limits the search for tags names. If the number of stale (already known) values reaches or exceeds this limit, the search stops. If Tempo processes `maxStaleValues` matches without finding a new tag name, the search is returned early. ### Search tags V2 Live store configuration `complete_block_timeout` affects how long tags are available for search. If the start or end aren't specified, it only fetches recent data from the live store. This endpoint retrieves all discovered tag names that can be used in search. The endpoint is available in the query frontend service in a microservices deployment, or the Tempo endpoint in a monolithic mode deployment. The tags endpoint takes a scope that controls the kinds of tags or attributes returned. If nothing is provided, the endpoint returns all resource and span tags. ```bash GET /api/v2/search/tags?scope=<resource|span|intrinsic|event|link|instrumentation> ``` Parameters: - `scope = (resource|span|intrinsic|event|link|instrumentation)` Specifies the scope of the tags, this is an optional parameter, if not specified it means all scopes. Default = `all` - `q = (traceql query)` Optional. A TraceQL query to filter tag names by. Supports `&&` and `||` operators within a single spanset. For example: `{ span.foo = "bar" && resource.baz = "bat" }` or `{ span.foo = "bar" || resource.baz = "bat" }`. See also [Filtered tag values](#filtered-tag-values). - `start = (unix epoch seconds)` Optional. Along with `end` define a time range from which tags should be returned. - `end = (unix epoch seconds)` Optional. Along with `start` define a time range from which tags should be returned. Providing both `start` and `end` includes blocks for the specified time range only. - `limit = (integer)` Optional. Sets the maximum number of tags names allowed per scope. The query stops once this limit is reached for any scope. - `maxStaleValues = (integer)` Optional. Limits the search for tag values. The search stops if the number of stale (already known) values reaches or exceeds this limit. #### Example Example of how to query Tempo using curl. This query returns all discovered tag names. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Search tag values Live store configuration `complete_block_timeout` affects how long tags are available for search. If start or end aren't specified, it only fetches recent data from the live store. This endpoint retrieves all discovered values for the given tag, which can be used in search. The endpoint is available in the query frontend service in a microservices deployment, or the Tempo endpoint in a monolithic mode deployment. The following request returns all discovered service names. ```bash GET /api/search/tag/service.name/values ``` #### Example Example of how to query Tempo using curl. This query returns all discovered values for the tag `service.name`. ```bash curl -G -s http://localhost:3200/api/search/tag/service.name/values | jq { "tagValues": [ "article-service", "auth-service", "billing-service", "cart-service", "postgres", "shop-backend" ], "metrics": { "inspectedBytes": "431380" } } ``` Parameters: - `start = (unix epoch seconds)` Optional. Along with `end`, defines a time range from which tags should be returned. - `end = (unix epoch seconds)` Optional. Along with `start`, defines a time range from which tags should be returned. Providing both `start` and `end` includes blocks for the specified time range only. - `limit = (integer)` Optional. Limits the maximum number of tags values. - `maxStaleValues = (integer)` Optional. Limits the search for tags values. If the number of stale (already known) values reaches or exceeds this limit, the search stops. If Tempo processes `maxStaleValues` matches without finding a new tag name, the search is returned early. ### Search tag values V2 This endpoint retrieves all discovered values and their data types for the given TraceQL identifier. The endpoint is available in the query frontend service in a microservices deployment, or the Tempo endpoint in a monolithic mode deployment. This endpoint is similar to `/api/search/tag/<tag>/values` but operates on TraceQL identifiers and types. Refer to [TraceQL](../traceql) documentation for more information. #### Example This example queries Tempo using curl and returns all discovered values for the tag `service.name`. ```bash curl -G -s http://localhost:3200/api/v2/search/tag/.service.name/values | jq { "tagValues": [ { "type": "string", "value": "article-service" }, { "type": "string", "value": "postgres" }, { "type": "string", "value": "cart-service" }, { "type": "string", "value": "billing-service" }, { "type": "string", "value": "shop-backend" }, { "type": "string", "value": "auth-service" } ], "metrics": { "inspectedBytes": "502756" } } ``` Parameters: - `start = (unix epoch seconds)` Optional. Along with `end`, defines a time range from which tags values should be returned. - `end = (unix epoch seconds)` Optional. Along with `start`, defines a time range from which tags values should be returned. Providing both `start` and `end` includes blocks for the specified time range only. - `q = (traceql query)` Optional. A TraceQL query to filter tag values by. Supports `&&` and `||` operators within a single spanset. For example: `{ span.foo = "bar" && resource.baz = "bat" }` or `{ span.foo = "bar" || resource.baz = "bat" }`. Refer to [Filtered tag values](#filtered-tag-values). - `limit = (integer)` Optional. Limits the maximum number of tags values - `maxStaleValues = (integer)` Optional. Limits the search for tags values. If the number of stale (already known) values reaches or exceeds this limit, the search stops. If Tempo processes `maxStaleValues` matches without finding a new tag name, the search is returned early. **Returns** By default, this endpoint returns a JSON response with tag values and their types. Other formats can be requested using the `Accept` header: - `Accept: application/vnd.grafana.llm` - Returns a simplified JSON format optimized for LLM consumption. This format is subject to change and shouldn't be relied on for programmatic use. #### Filtered tag values You can pass an optional URL query parameter, `q`, to your request. The `q` parameter is a URL-encoded [TraceQL query](../traceql). If provided, the tag values returned by the API are filtered to only return values seen on spans matching your filter parameters. Queries can be incomplete: for example, `{ resource.cluster = }`. Tempo extracts only the valid matchers and builds a valid query. If an input is invalid, Tempo doesn't provide an error. Instead, you'll see the whole list when a failure of parsing input. This behavior helps with backwards compatibility. Queries must use a single spanset selector `{}`. Both `&&` and `||` operators are supported within the selector. - Example supported: `{ resource.cluster = "us-east-1" && resource.service = "frontend" }` - Example supported: `{ resource.cluster = "us-east-1" || resource.service = "frontend" }` - Example unsupported: `{ resource.cluster = "us-east-1" } && { resource.cluster = "us-east-2" }` Unscoped attributes are supported for filtered tag values. When you use an unscoped attribute name, Tempo looks for matches in both span and resource scopes. The following request returns all discovered service names on spans with `span.http.method=GET`: ``` GET /api/v2/search/tag/resource.service.name/values?q="{span.http.method='GET'}" ``` If a particular service name (for example, `shopping-cart`) is only present on spans with `span.http.method=POST`, it won't be included in the list of values returned. ### TraceQL Metrics The TraceQL Metrics API returns Prometheus-like time-series for a given metrics query. Metrics queries are those using metrics functions like `rate()` and `quantile_over_time()`. Refer to the [TraceQL metrics documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/) for more information list. Parameters: - `q = (traceql query)` The TraceQL metrics query to process. - `start = (unix epoch seconds | unix epoch nanoseconds | RFC3339 string)` Optional. Along with `end` defines the time range. - `end = (unix epoch seconds | unix epoch nanoseconds | RFC3339 string)` Optional. Along with `start` define the time range. Providing both `start` and `end` includes blocks for the specified time range only. - `since = (duration string)` Optional. Can be used instead of `start` and `end` to define the time range in relative values. For example, `since=15m` queries the last 15 minutes. Default is the last 1 hour. - `step = (duration string)` Optional. Defines the granularity of the returned time-series. For example, `step=15s` returns a data point every 15s within the time range. If not specified, then the default behavior chooses a dynamic step based on the time range. - `exemplars = (integer)` Optional. Defines the maximum number of exemplars for the query. It's trimmed to `max_exemplars` if it exceeds it. The API is available in the query frontend service in a microservices deployment, or the Tempo endpoint in a monolithic mode deployment. For example, the following request computes the rate of spans received for `myservice` over the last three hours, at 1 minute intervals. {{< admonition type="note" >}} Actual API parameters must be URL-encoded. This example is left unencoded for readability. {{< /admonition >}} ``` GET /api/metrics/query_range?q={resource.service.name="myservice"} | min_over_time() with(exemplars=true) &since=3h&step=1m&exemplars=100 ``` #### Instant The instant version of the metrics API is similar to the range version, but instead returns a single value for the query. This version is useful when you don't need the granularity of a full time-series, but instead want a total sum, or single value computed across the whole time range. The parameters are identical to the range version except there is no `step`. Parameters: - `q = (traceql query)` The TraceQL metrics query to process. - `start = (unix epoch seconds | unix epoch nanoseconds | RFC3339 string)` Optional. Along with `end` defines the time range. - `end = (unix epoch seconds | unix epoch nanoseconds | RFC3339 string)` Optional. Along with `start` define the time range. Providing both `start` and `end` includes blocks for the specified time range only. - `since = (duration string)` Optional. Can be used instead of `start` and `end` to define the time range in relative values. For example, `since=15m` queries the last 15 minutes. Default is last 1 hour. The API is available in the query frontend service in a microservices deployment, or the Tempo endpoint in a monolithic mode deployment. For example the following request computes the total number of failed spans over the last hour per service. {{< admonition type="note" >}} Actual API parameters must be URL-encoded. This example is left unencoded for readability. {{< /admonition >}} ``` GET /api/metrics/query?q={status=error}|count_over_time()by(resource.service.name) ``` ### Traces diff {{< admonition type="warning" >}} This endpoint is experimental. The request and response formats may change in future releases. {{< /admonition >}} This endpoint compares two complete traces. Send a `POST` request with a JSON body that identifies both traces by their IDs. Partial traces are rejected. This endpoint uses the same `max_bytes_per_trace` limit as [querying a trace by ID](#query-v2). ``` POST /api/v2/traces/diff ``` Request body: ```json { "base": { "traceId": "<BASE_TRACE_ID>", "start": 1700000000, "end": 1700003600 }, "compare": { "traceId": "<COMPARE_TRACE_ID>", "start": 1700100000, "end": 1700103600 }, "format": "trace-summary-v0-composed" } ``` Parameters: - `base.traceId` Trace ID for the baseline trace, as a hexadecimal string. - `base.start` Optional. Start of the time range to search for the baseline trace (UNIX epoch seconds). - `base.end` Optional. End of the time range to search for the baseline trace (UNIX epoch seconds). - `compare.traceId` Trace ID for the comparison trace, as a hexadecimal string. - `compare.start` Optional. Start of the time range to search for the comparison trace (UNIX epoch seconds). - `compare.end` Optional. End of the time range to search for the comparison trace (UNIX epoch seconds). - `format` Optional. Output format. The default is `trace-patch-v0`. Use `trace-summary-v0-native` for only the compact summary or `trace-summary-v0-composed` for the summary with a patch attachment limited to 64 KiB. The composed response always includes a native summary. If the full patch is larger than 64 KiB, `patchOmitted` reports its serialized size and the reason `over_budget`. Request `trace-patch-v0` only when span-level evidence is required and the client can accept the reported patch size. The 64 KiB limit applies only to the attached patch; neither the complete composed response nor the full patch response has a hard output size limit. When `start` and `end` are provided, the request validates that `start` is before `end` and limits the block search to that time range. If omitted, Tempo searches across all blocks. Only `POST` is allowed. Other methods return `405 Method Not Allowed`. ### Query Echo endpoint ``` GET /api/echo ``` Returns status code 200 and body `echo` when the query frontend is up and ready to receive requests. {{< admonition type="note" >}} Meant to be used in a Query Visualization UI like Grafana to test that the Tempo data source is working. {{< /admonition >}} ### Overrides API For more information about user-configurable overrides API, refer to the [user-configurable overrides](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/user-configurable-overrides/#api) documentation. ### Prepare live store downscale ``` GET,POST,DELETE /live-store/prepare-downscale ``` This endpoint prepares the live store for downscaling by configuring whether it should remove itself from the ring on shutdown. A `GET` call returns set if the live-store is prepared for downscale, unset otherwise. A `POST` call enables prepare downscale mode (remove the live store from the ring owners on shutdown). The partition must be in `INACTIVE` state. A `DELETE` call disables prepare downscale mode (do not remove the live store from the ring owners on shutdown). ### Prepare live store partition downscale ``` GET,POST,DELETE /live-store/prepare-partition-downscale ``` This endpoint prepares the live store's partition for downscaling by setting it to the `INACTIVE` state. A `GET` call to this endpoint returns a timestamp of when the partition was switched to the `INACTIVE` state, or 0, if the partition is not in the `INACTIVE` state. A `POST` call switches this live store's partition to the `INACTIVE` state, if it isn't `INACTIVE` already, and returns the timestamp of when the switch to the `INACTIVE` state occurred. A `DELETE` call sets the partition back from the `INACTIVE` to the `ACTIVE` state. ### Usage metrics {{< admonition type="note" >}} This endpoint is only available when one or more usage trackers are enabled in [the distributor](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#distributor). {{< /admonition >}} ``` GET /usage_metrics ``` Special metrics scrape endpoint that provides per-tenant metrics on ingested data. Per-tenant grouping rules are configured in [the per-tenant overrides](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#overrides) Example: ``` curl http://localhost:3200/usage_metrics # HELP tempo_usage_tracker_bytes_received_total bytes total received with these attributes # TYPE tempo_usage_tracker_bytes_received_total counter tempo_usage_tracker_bytes_received_total{service="auth-service",tenant="single-tenant",tracker="cost-attribution"} 96563 tempo_usage_tracker_bytes_received_total{service="cache",tenant="single-tenant",tracker="cost-attribution"} 81904 tempo_usage_tracker_bytes_received_total{service="gateway",tenant="single-tenant",tracker="cost-attribution"} 164751 tempo_usage_tracker_bytes_received_total{service="identity-service",tenant="single-tenant",tracker="cost-attribution"} 85974 tempo_usage_tracker_bytes_received_total{service="service-A",tenant="single-tenant",tracker="cost-attribution"} 92799 ``` ### Distributor ring status {{< admonition type="note" >}} This endpoint is only available when Tempo is configured with [the global ingestion rate strategy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#ingestion-rate-strategy). {{< /admonition >}} ``` GET /distributor/ring ``` Displays a web page with the distributor hash ring status, including the state, healthy, and last heartbeat time of each distributor. For more information, refer to [consistent hash ring](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/consistent_hash_ring/). ### Live-store ring status ``` GET /live-store/ring ``` Displays a web page with the live-store hash ring status, including the state, health, and last heartbeat time of each live-store instance. For more information, refer to [consistent hash ring](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/consistent_hash_ring/). ### Partition ring status ``` GET /partition-ring ``` Displays a web page with the partition ring status, showing partition ownership across live-store instances, including state, ownership percentage, and available actions. For more information, refer to [consistent hash ring](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/consistent_hash_ring/). ### Status ``` GET /status ``` Print all available information by default. ``` GET /status/version ``` Print the version information. ``` GET /status/services ``` Displays a list of services and their status. If a service failed it shows the failure case. ``` GET /status/endpoints ``` Displays status information about the API endpoints. ``` GET /status/config ``` Displays the configuration. Displays the configuration currently applied to Tempo (in YAML format), including default values and settings via CLI flags. Sensitive data is masked. Be aware that the exported configuration **doesn't include the per-tenant overrides**. Optional query parameter: - `mode = (diff|defaults)`: `diff` shows the difference between the default values and the current configuration. `defaults` shows the default values. ``` GET /status/runtime_config ``` Displays the override configuration. Query parameter: - `mode = (diff)`: Show the difference between defaults and overrides. ``` GET /status/overrides ``` Displays all tenants that have non-default overrides configured. ``` GET /status/overrides/{tenant} ``` Displays all overrides configured for the specified tenant. ``` GET /status/usage-stats ``` Displays anonymous usage stats data that's reported back to Grafana Labs. ### List build information ``` GET /api/status/buildinfo ``` Exposes the build information in a JSON object. The fields are `version`, `revision`, `branch`, `buildDate`, `buildUser`, and `goVersion`. ### Backend scheduler job status ``` GET /status/backendscheduler ``` Displays the current state of all jobs in the backend scheduler work cache and the pending redaction queue. The response is a plain-text table with two sections: - Active Jobs: all jobs in the work cache, sorted by creation time. Use the `status` column to interpret each row. A non-empty `worker` field indicates the job is currently assigned to a worker. - Pending Jobs: redaction jobs in the pending queue. Some may already be eligible to run; others may still be waiting for the rescan or compaction preconditions to clear. This endpoint is only available when the backend scheduler component is running. ## Tempo gRPC API Tempo uses [gRPC](https://grpc.io) to internally communicate with itself, but only has one externally supported client. The query-frontend component implements the streaming querier interface defined below. [Refer here](https://github.com/grafana/tempo/blob/main/pkg/tempopb/) for the complete proto definition and generated code. By default, this service is only offered over the gRPC port. You can use streaming service over the HTTP port as well, which Grafana expects. To enable the streaming service over the HTTP port for use with Grafana, set the following: ```yaml stream_over_http_enabled: true ``` The query frontend segments streamed diffs and final responses into smaller packets. The default packet size is 1 MiB. To change the packet size, configure `query_frontend.max_grpc_streaming_packet_size`. For details, refer to the [query frontend configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#query-frontend). The query frontend supports the following interface. Refer to [`tempo.proto`](https://github.com/grafana/tempo/blob/main/pkg/tempopb/tempo.proto) for complete details of all objects. ```protobuf service StreamingQuerier { rpc Search(SearchRequest) returns (stream SearchResponse); rpc SearchTags(SearchTagsRequest) returns (stream SearchTagsResponse) {} rpc SearchTagsV2(SearchTagsRequest) returns (stream SearchTagsV2Response) {} rpc SearchTagValues(SearchTagValuesRequest) returns (stream SearchTagValuesResponse) {} rpc SearchTagValuesV2(SearchTagValuesRequest) returns (stream SearchTagValuesV2Response) {} rpc MetricsQueryRange(QueryRangeRequest) returns (stream QueryRangeResponse) {} } ``` {{< admonition type="note" >}} gRPC compression is disabled by default. Refer to [gRPC compression configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#grpc-compression) for more information. {{< /admonition >}} --- ### Sources Tempo Community Index (`docs/sources/tempo/community/_index.md`) --- title: Community description: Participate in and contribute to the Tempo open source community. weight: 900 --- # Community If you are interested in distributed tracing, consider participating and contributing to the [open source Tempo project](https://github.com/grafana/tempo). Want to learn more about open source communities? Read David Allen's blog post, [What makes a good open source community?](/blog/2023/09/01/what-makes-a-good-open-source-community/) on the Grafana website. ## Communicate - [Grafana Slack](https://slack.grafana.com/) #tempo channel - [Community Forum](https://community.grafana.com/c/grafana-tempo/40) - for questions/feedback. - [Community Call](https://docs.google.com/document/d/1yGsI6ywU-PxZBjmq3p3vAXr9g5yBXSDk4NU8LGo8qeY/edit#) - Monthly on the second Thursday at 1630 UTC. Recordings available [online](https://www.youtube.com/playlist?list=PLDGkOdUX1Ujqe8WZ8T1h2pNjpll0t-KLw). ## Contribute This page lists resources for developers who want to contribute to the Tempo software ecosystem. - [Governance](https://github.com/grafana/tempo/blob/main/GOVERNANCE.md) - [Contributing](https://github.com/grafana/tempo/blob/main/CONTRIBUTING.md) - [Code of Conduct](https://github.com/grafana/tempo/blob/main/CODE_OF_CONDUCT.md) - [License](https://github.com/grafana/tempo/blob/main/LICENSE) --- ### Sources Tempo Configuration Index (`docs/sources/tempo/configuration/_index.md`) --- title: Configure Tempo menuTitle: Configure description: Learn about available options in Tempo and how to configure them. weight: 400 --- # Configure Tempo This document explains the configuration options for Tempo as well as the details of what they impact. {{< admonition type="tip" >}} Instructions for configuring Tempo data sources are available in the [Grafana Cloud](/docs/grafana-cloud/send-data/traces/) and [Grafana](/docs/grafana/latest/datasources/tempo/) documentation. {{< /admonition >}} The Tempo configuration options include: - [Configure Tempo](#configure-tempo) - [Use environment variables in the configuration](#use-environment-variables-in-the-configuration) - [Deployment modes](#deployment-modes) - [Configuration by deployment mode](#configuration-by-deployment-mode) - [Server](#server) - [Memory](#memory) - [Distributor](#distributor) - [Set max attribute size to help control out of memory errors](#set-max-attribute-size-to-help-control-out-of-memory-errors) - [gRPC compression](#grpc-compression) - [Ingest](#ingest) - [Block-builder](#block-builder) - [Live-store](#live-store) - [Metrics-generator](#metrics-generator) - [Query-frontend](#query-frontend) - [Limit query size to improve performance and stability](#limit-query-size-to-improve-performance-and-stability) - [Limit the spans per spanset](#limit-the-spans-per-spanset) - [Cap the maximum query length](#cap-the-maximum-query-length) - [Querier](#querier) - [Backend scheduler](#backend-scheduler) - [Backend worker](#backend-worker) - [Storage](#storage) - [Local storage recommendations](#local-storage-recommendations) - [Hedged requests](#hedged-requests) - [Storage block configuration example](#storage-block-configuration-example) - [Memberlist](#memberlist) - [Configuration blocks](#configuration-blocks) - [Block](#block) - [Compaction](#compaction) - [Filter policies](#filter-policies) - [Filter policy](#filter-policy) - [Policy match](#policy-match) - [Examples](#examples) - [GRPC client](#grpc-client) - [KVStore](#kvstore) - [Search](#search) - [WAL](#wal) - [Overrides](#overrides) - [Ingestion limits](#ingestion-limits) - [Standard overrides](#standard-overrides) - [Tenant-specific overrides](#tenant-specific-overrides) - [Runtime overrides](#runtime-overrides) - [User-configurable overrides](#user-configurable-overrides) - [Ingestion rate strategy](#ingestion-rate-strategy) - [Examples](#examples-1) - [Usage-report](#usage-report) - [Configure usage-reporting](#configure-usage-reporting) - [Cache](#cache) - [Configure authentication](#configure-authentication) Additionally, you can review [TLS](network/tls/) to configure the cluster components to communicate over TLS, or receive traces over TLS. {{< admonition type="tip" >}} Throughout the configuration, the `duration` values support the following units: `ns`, `us` (or `µs`), `ms`, `s`, `m`, `h`. See the [Go time package](https://pkg.go.dev/time#ParseDuration) for more information. {{< /admonition >}} ## Use environment variables in the configuration You can use environment variable references in the configuration file to set values that need to be configurable during deployment. To do this, pass `-config.expand-env=true` and use: ``` ${VAR} ``` Where `VAR` is the name of the environment variable. Each variable reference is replaced at startup by the value of the environment variable. The replacement is case-sensitive and occurs before the YAML file is parsed. References to undefined variables are replaced by empty strings unless you specify a default value or custom error text. To specify a default value, use: ``` ${VAR:-default_value} ``` where `default_value` is the value to use if the environment variable is undefined. You can find more about other supported syntax [here](https://github.com/drone/envsubst/blob/master/readme.md). ## Deployment modes Tempo supports two deployment modes: monolithic and microservices. * Monolithic mode: The required components run in a single process using `-target=all`, which is the default. No Kafka is required. * Microservices mode: Each component runs as a separate process with its own `-target` flag. For example, `-target=distributor` or `-target=querier`. This mode requires a Kafka-compatible system, such as Apache Kafka, Redpanda, or WarpStream, as the durable queue between the distributor and downstream components. Refer to the [Deployment modes](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/plan/deployment-modes/) documentation for more information on when to use each mode. ### Configuration by deployment mode Not all configuration blocks apply to both deployment modes. In monolithic mode, the most important configuration blocks are: | Config block | Description | |---|---| | `distributor` | Configure receivers (OTLP, Jaeger, Zipkin) and ingestion limits. | | `storage` | Configure the backend used to flush and store trace blocks. | | `metrics_generator` | Optional. Configure span-metrics and service-graph generation. | | `query_frontend` | Configure query splitting, caching, and result streaming. | | `overrides` | Set per-tenant rate limits and trace size limits. | The `ingest` and `block_builder` blocks are only used in microservices mode. Other blocks, including `live_store_client`, `backend_scheduler`, `backend_worker`, `memberlist`, and `cache`, apply in both modes but run in-process in monolithic mode. For the complete mapping of all configuration blocks to deployment modes, refer to the [Components by deployment mode](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/deployment-modes/#components-by-deployment-mode) table. ## Server Tempo uses the server from `dskit/server`. For the full list of available server options, refer to the [dskit server configuration](https://github.com/grafana/dskit/blob/main/server/server.go#L66) and the [manifest](/docs/tempo/<TEMPO_VERSION>/configuration/manifest/). For details on how server settings apply across deployment modes, refer to the [Deployment modes](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/deployment-modes/) documentation. Additional root-level options such as `target`, `shutdown_delay`, `auth_enabled`, `enable_go_runtime_metrics`, and `span_profiling` are available as [command-line flags](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/command-line-flags/). ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Internal server Tempo can expose a separate HTTP server for internal endpoints (health checks, metrics, readiness) on a dedicated port. This keeps internal traffic isolated from the public API. The `internal_server` block accepts the same fields as `server` but defaults to port `3101` with instrumentation disabled. The internal server is disabled by default; set `internal_server.enable: true` to activate it. For the full list of `internal_server` options, refer to the [manifest](/docs/tempo/<TEMPO_VERSION>/configuration/manifest/). ## Memory Tempo supports automatic GOMEMLIMIT configuration using the [automemlimit](https://github.com/KimMachineGun/automemlimit) library. When enabled, it automatically sets Go's memory limit based on available container (via CGroups) or system memory every 15 seconds. For information on memory considerations across deployment modes, refer to the [Deployment modes](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/deployment-modes/) documentation. NOTE: enabling this will override value set in GOMEMLIMIT environment variable ```yaml memory: # Enable automatic GOMEMLIMIT configuration based on cgroup/system memory. # When enabled, Tempo will automatically detect available memory from cgroups (v2 or v1) # or system memory and set GOMEMLIMIT accordingly. [automemlimit_enabled: <bool> | default = false] # Ratio of available memory to use for GOMEMLIMIT. # For example, 0.8 means 80% of available memory will be used for GOMEMLIMIT. [automemlimit_ratio: <float> | default = 0.8] ``` ## Distributor For more information on configuration options, refer to [this file](https://github.com/grafana/tempo/blob/main/modules/distributor/config.go). For architectural details, refer to the [Distributor architecture](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/distributor/) documentation. The distributor is the entry point for all trace data into Tempo. It receives spans from instrumented applications, validates them against configured [ingestion limits](#ingestion-limits), and forwards them for processing. How the distributor forwards data depends on the [deployment mode](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/deployment-modes/): - **Microservices mode**: The distributor shards traces by trace ID and writes them to [Kafka](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/kafka/). Kafka settings are configured in the [Ingest](#ingest) section. Downstream, [block-builders](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/block-builder/) and [live-stores](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/live-store/) consume from Kafka independently. - **Monolithic mode** (`target: all`): The distributor pushes data in-process directly to the live-store and metrics-generator. No Kafka is required in this mode. It's suitable for local development, testing, and small installations. The following configuration enables all available receivers with their default configuration. For a production deployment, enable only the receivers you need. Additional documentation and more advanced configuration options are available in [the receiver README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/README.md). ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Set max attribute size to help control out of memory errors Tempo queriers can run out of memory when fetching traces that have spans with very large attributes. This issue has been observed when trying to fetch a single trace using the [`tracebyID` endpoint](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#query). While a trace might not have a lot of spans (roughly 500), it can have a larger size (approximately 250KB). Some of the spans in that trace had attributes whose values were very large in size. To avoid these out-of-memory crashes, use `max_attribute_bytes` to limit the maximum allowable size of any individual attribute. Any key or values that exceed the configured limit are truncated before storing. The default value is `2048`. Use the `tempo_distributor_attributes_truncated_total` metric to track how many attributes are truncated. This metric includes `tenant` and `scope` labels, where `scope` is one of `resource`, `scope`, `span`, `event`, or `link`. Use the `tempo_distributor_attribute_size_bytes` histogram to observe the size distribution of attributes as they are received. Like the counter, it includes `tenant` and `scope` labels, and each observation records the combined size of an attribute's key and string value. Every attribute is measured, whether or not it is truncated. When truncation occurs, the distributor also emits a rate-limited log line (at most one per second) with diagnostic details including the tenant, total truncated count, configured limit, and an example of the first truncated attribute (scope, name, field, and original size). For additional information, refer to [Troubleshoot out-of-memory errors](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/out-of-memory-errors/). ### gRPC compression By default, gRPC compression between all components uses `snappy`. This provides a balanced approach to compression that works for most installations. If you prefer a different balance of CPU/memory and bandwidth, consider disabling compression or using `zstd`. Disabling compression may reduce CPU and memory usage on queriers and distributors, but can increase network traffic and [live-store](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/live-store/) data volume, especially in larger clusters. Increased data can impact billing for Grafana Cloud. You can configure the gRPC compression in the `live_store_client` and `querier.frontend_worker` gRPC clients. To disable compression, remove `snappy` from the `grpc_compression` lines. To set the compression, use `snappy` with the following settings: ```yaml live_store_client: grpc_client_config: grpc_compression: "snappy" querier: frontend_worker: grpc_client_config: grpc_compression: "snappy" ``` ## Ingest In distributed mode, the ingest configuration controls the Kafka-compatible system that Tempo uses as a durable write-ahead log for trace data. Distributors write to Kafka, and downstream consumers (block-builders, live-stores, and metrics-generators) each consume from it independently. In monolithic mode (`target: all`), distributors bypass Kafka entirely, so these settings don't apply. For more information on configuration options, refer to [this file](https://github.com/grafana/tempo/blob/main/pkg/ingest/config.go). For architectural details, refer to the [Kafka architecture](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/kafka/) documentation. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Block-builder The block-builder consumes trace data from Kafka, organizes it into Parquet blocks, and flushes them to object storage for long-term retention and querying. For more information on configuration options, refer to [this file](https://github.com/grafana/tempo/blob/main/modules/blockbuilder/config.go). For architectural details, refer to the [Block-builder architecture](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/block-builder/) documentation. ```yaml # Block-builder configuration block block_builder: # Instance id. [instance_id: <string> | default = <hostname>] # Map of instance names to partition IDs assigned to each block builder. [assigned_partitions: <map of string to list of int>] # Number of partitions assigned to this block builder. [partitions_per_instance: <int> | default = 0] # Interval between consumption cycles. [consume_cycle_duration: <duration> | default = 5m] # Maximum number of bytes that can be consumed in a single cycle. 0 to disable. [max_consuming_bytes: <uint64> | default = 5000000000] # Block configuration for the block builder. block: <Block config> [max_block_bytes: <uint64> | default = 20971520] # Write ahead log configuration for the block builder. wal: <WAL config> [path: <string> | default = "/var/tempo/block-builder/traces"] ``` ## Live-store The live-store holds recent trace data in memory and serves queries during the window between ingestion and block availability in object storage. It periodically flushes traces to a local WAL in Parquet format for TraceQL search and metrics queries. For more information on configuration options, refer to [this file](https://github.com/grafana/tempo/blob/main/modules/livestore/config.go). For architectural details, refer to the [Live-store architecture](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/live-store/) documentation. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Metrics-generator The metrics-generator processes spans and writes metrics using the Prometheus remote write protocol. For more information on the metrics-generator, refer to the [Metrics-generator documentation](../metrics-from-traces/metrics-generator/). Metrics-generator processors are disabled by default. To enable it for a specific tenant, set `metrics_generator.processors` in the [overrides](#overrides) section. For more information on configuration options, refer to [this file](https://github.com/grafana/tempo/blob/main/modules/generator/config.go). For architectural details, refer to the [Metrics-generator architecture](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/metrics-generator/) documentation. {{< admonition type="note" >}} If you want to enable metrics-generator for your Grafana Cloud account, refer to the [Metrics-generator in Grafana Cloud](https://grafana.com/docs/grafana-cloud/send-data/traces/metrics-generator/) documentation. {{< /admonition >}} Spans with end times older than `metrics_ingestion_time_range_slack` are excluded from metrics generation. The default value is 30 seconds, which means spans that ended more than 30 seconds ago are discarded. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Use `http_headers` with `secrets` to send sensitive header values Header values that you set under `headers` in a `remote_write` endpoint are stored and displayed in plaintext. They appear in the configuration that the [`/status/config` endpoint](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#status) returns, which can expose secrets such as authentication tokens. To send sensitive header values, use `http_headers` with `secrets` instead of `headers`. Each header under `http_headers` supports three optional fields: ```yaml # Custom HTTP headers to send with each remote write request. http_headers: # Header name. [<string>]: # Plaintext header values. Stored and displayed in plaintext. [values: <list of strings>] # Header values that Tempo masks as <secret> in the displayed configuration. [secrets: <list of secrets>] # Files that Tempo reads header values from. [files: <list of strings>] ``` Use `secrets` for sensitive values, such as authentication tokens. Tempo masks values that you configure under `secrets` as `<secret>` in the displayed configuration. Use `values` for non-sensitive values, and `files` to read a value from a file on disk. For example: ```yaml metrics_generator: storage: remote_write: - url: https://prometheus.example.com/api/v1/write http_headers: # Masked as <secret> in the displayed configuration. X-Custom-Token: secrets: - <SECRET_VALUE> # Stored and displayed in plaintext. X-Custom-Header: values: - custom-value # Read from a file on disk. X-Token-From-File: files: - /etc/tempo/custom-token ``` ## Query-frontend The query frontend is responsible for sharding incoming requests for faster processing in parallel by the queriers. For more information on configuration options, refer to [this file](https://github.com/grafana/tempo/blob/main/modules/frontend/config.go). For architectural details, refer to the [Query frontend architecture](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/query-frontend/) documentation. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Limit query size to improve performance and stability Querying large tracing data presents several challenges. Span sets with large number of spans impact query performance and stability. In a similar manner, excessive queries result size can also negatively impact query performance. #### Limit the spans per spanset You can control spans per spanset behavior using two configuration options: - `default_spans_per_span_set`: Sets the default number of spans returned when not specified in the query (default: 3). Set to `0` to return unlimited spans by default. - `max_spans_per_span_set`: Sets the maximum allowed value (default: 100). Set to `0` to disable the limit entirely. In Grafana or Grafana Cloud, you can use the **Span Limit** field in the [TraceQL query editor](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/data-sources/tempo/query-editor/) in Grafana Explore. This field sets the number of spans to return for each span set (the `spss` query parameter). If not specified, the value from `default_spans_per_span_set` is used. The maximum value that you can set for **Span Limit** (or the `spss` query parameter) is controlled by `max_spans_per_span_set`. To disable the maximum spans per span set limit, set `max_spans_per_span_set` to `0`. When set to `0`, there is no maximum and users can request any number of spans, including unlimited spans by setting `spss=0`. #### Cap the maximum query length You can set the maximum length of a query using `query_frontend.max_query_expression_size_bytes` configuration parameter for the query-frontend. The default value is 128 KB. This limit protects system stability from potential abuse or mistakes when running large, potentially expensive queries. Tempo enforces this limit before parsing the TraceQL expression, so queries that exceed the configured size are rejected immediately with a clear error message. You can set the value lower or higher in the `query_frontend` configuration section, for example: ``` query_frontend: max_query_expression_size_bytes: 10000 ``` ## Querier The querier executes query jobs dispatched by the query frontend. It fetches trace data from both live-stores (for recent data) and object storage (for historical data), then returns results to the query frontend for merging. For more information on configuration options, refer to [this file](https://github.com/grafana/tempo/blob/main/modules/querier/config.go). For architectural details, refer to the [Querier architecture](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/querier/) documentation. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The querier also queries compacted blocks that fall within a range of `2 * storage.trace.blocklist_poll`, where `storage.trace.blocklist_poll` is the blocklist poll duration configured in the storage section below. ## Backend scheduler For architectural details, refer to the [Compaction architecture](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/) documentation. The backend scheduler is responsible for scheduling and tracking jobs which are assigned to backend workers for processing. Only one scheduler should be running at a time. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Backend worker For architectural details, refer to the [Compaction architecture](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/) documentation. The backend worker connects to the backend scheduler to receive and process jobs. Workers are responsible for executing compaction and retention and other jobs, and updating the scheduler on job status. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Storage Tempo supports Amazon S3, GCS, Azure, and local file system for storage. In addition, you can use Memcached or Redis for increased query performance. For more information on configuration options, refer to [this file](https://github.com/grafana/tempo/blob/main/tempodb/config.go). ### Local storage recommendations While you can use local storage, object storage is recommended for production workloads. A local backend won't correctly retrieve traces with a distributed deployment unless all components have access to the same disk. Tempo is designed for object storage more than local storage. At Grafana Labs, we've run Tempo with SSDs when using local storage. Hard drives haven't been tested. You can estimate how much storage space you need by considering the ingested bytes and retention. For example, ingested bytes per day _times_ retention days = stored bytes. You can not use both local and object storage in the same Tempo deployment. ### Hedged requests Each storage backend (GCS, S3, and Azure) supports hedged requests. Hedging reduces long-tail read latency by issuing a duplicate backend request after a configured delay. Tempo uses whichever response arrives first and discards the other. Configure hedging with two parameters in each backend block: - `hedge_requests_at` -- the delay before sending a duplicate request. Set this to approximately the p99 latency of your backend requests. Default is `0` (disabled). - `hedge_requests_up_to` -- the maximum number of requests to issue, including the original. Default is `2`. Requires `hedge_requests_at` to be set. Hedging is most effective on querier nodes, where read latency directly affects query performance. It has minimal impact on other components. ### Storage block configuration example The storage block configures TempoDB. The following example shows common options. For further platform-specific information, refer to the following: - [GCS](hosted-storage/gcs/) - [S3](hosted-storage/s3/) - [Azure](hosted-storage/azure/) - [Parquet](parquet/) ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Memberlist [Memberlist](https://github.com/hashicorp/memberlist) is the default mechanism for all of the Tempo pieces to coordinate with each other. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Configuration blocks Defines re-used configuration blocks. ### Block ```yaml # block format version. options: vParquet4, vParquet5 [version: <string> | default = vParquet5] # bloom filter false positive rate. lower values create larger filters but fewer false positives [bloom_filter_false_positive: <float> | default = 0.01] # maximum size of each bloom filter shard [bloom_filter_shard_size_bytes: <int> | default = 100KiB] # an estimate of the number of bytes per row group when cutting Parquet blocks. lower values will # create larger footers but will be harder to shard when searching. It is difficult to calculate # this field directly and it may vary based on workload. This is roughly a lower bound. [parquet_row_group_size_bytes: <int> | default = 250MB] # Configures attributes to be stored in dedicated columns within the parquet file, rather than in the # generic attribute key-value list. This allows for more efficient searching of these attributes. # Up to 20 string and 5 int attributes can be configured per scope (span, resource, event). # Requires vParquet4 or later. parquet_dedicated_columns: <list of columns> # name of the attribute - [name: <string>] # type of the attribute. options: string, int [type: <string>] # scope of the attribute. # options: resource, span [scope: <string>] ``` ### Compaction The `compaction` configuration block is used by the scheduler and worker. ```yaml # Optional. Duration to keep blocks. [block_retention: <duration> | default=336h] # Optional # Duration to keep blocks that have been compacted elsewhere. [compacted_block_retention: <duration> | default=1h] # Optional # Blocks in this time window will be compacted together. [compaction_window: <duration> | default=1h] # Optional # Maximum number of traces in a compacted block. # WARNING: Deprecated. Use max_block_bytes instead. [max_compaction_objects: <int> | default=6000000] # Optional # Maximum size of a compacted block in bytes. [max_block_bytes: <int> | default=107374182400] # Optional # Number of tenants to process in parallel during retention. [retention_concurrency: <int> | default=10] # Optional # Number of blocks to clear in parallel within a single tenant's retention pass. [retention_block_concurrency: <int> | default=4] # Optional # The maximum amount of time to spend compacting a single tenant before moving to the next. [max_time_per_tenant: <duration> | default=5m] # Optional # The time between compaction cycles. # Note: The default will be used if the value is set to 0. [compaction_cycle: <duration> | default=30s] ``` ### Filter policies Span filter configuration policies block #### Filter policy ```yaml # Exclude filters (positive matching) [include: <policy match>] # Exclude filters (negative matching) [exclude: <policy match>] ``` #### Policy match ```yaml # How to match the value of attributes # Options: "strict", "regex" [match_type: <string>] # List of attributes to match attributes: <list of policy attributes> # Attribute key - [key: <string>] # Attribute value [value: <any>] ``` #### Examples ```yaml exclude: match_type: "regex" attributes: - key: "resource.service.name" value: "unknown_service:myservice" ``` ```yaml include: match_type: "strict" attributes: - key: "foo.bar" value: "baz" ``` ### GRPC client These settings are used to configure various gRPC clients used throughout Tempo. ``` [max_recv_msg_size: <int> | default = 104857600] [max_send_msg_size: <int> | default = 104857600] [grpc_compression: <string> | default = "snappy"] [rate_limit: <float> | default = 0] [rate_limit_burst: <int> | default = 0] [backoff_on_ratelimits: <bool> | default = false] backoff_config: [min_period: <duration> | default = 100ms] [max_period: <duration> | default = 10s] [max_retries: <int> | default = 10] [initial_stream_window_size: <int>] [initial_connection_window_size: <int>] [tls_enabled: <bool> | default = false] [tls_cert_path: <string>] [tls_key_path: <string>] [tls_ca_path: <string>] [tls_server_name: <string>] [tls_insecure_skip_verify: <bool> | default = false] [tls_cipher_suites: <string>] [tls_min_version: <string>] [connect_timeout: <duration> | default = 5s] [connect_backoff_base_delay: <duration> | default = 1s] [connect_backoff_max_delay: <duration> | default = 5s] ``` ### KVStore The `kvstore` configuration block ```yaml # Set backing store to use [store: <string> | default = "consul"] # What prefix to use for keys [prefix: <string> | default = "ring."] # Store specific configs consul: [host: <string> | default = "localhost:8500"] [acl_token: <secret string> | default = "" ] [http_client_timeout: <duration> | default = 20s] [consistent_reads: <bool> | default = false] [watch_rate_limit: <float64> | default = 1.0] [watch_burst_size: <int> | default = 1] [cas_retry_delay: <duration> | default 1s] etcd: [endpoints: <list of string> | default = [] ] [dial_timeout: <duration> | default = 10s] [max_retries: <int> | default = 10 ] [tls_enabled: <bool> | default = false] # TLS config [tls_cert_path: <string> | default = ""] [tls_key_path: <string> | default = ""] [tls_ca_path: <string> | default = ""] [tls_server_name: <string> | default = ""] [tls_insecure_skip_verify: <bool> | default = false] [tls_cipher_suites: <string> | default = ""] [tls_min_version: <string> | default = ""] [username: <string> | default = ""] [password: <secret string> | default = ""] multi: [primary: <string> | default = ""] [secondary: <string> | default = ""] [mirror_enabled: <bool> | default = false] [mirror_timeout: <bool> | default = 2s] ``` ### Search ```yaml # Target number of bytes per GET request while scanning blocks. Default is 1MB. Reducing # this value could positively impact trace search performance at the cost of more requests # to object storage. [chunk_size_bytes: <uint32> | default = 1000000] # Number of traces to prefetch while scanning blocks. Default is 1000. Increasing this value # can improve trace search performance at the cost of memory. [prefetch_trace_count: <int> | default = 1000] # Number of read buffers used when performing search on a vparquet block. This value times the read_buffer_size_bytes # is the total amount of bytes used for buffering when performing search on a parquet block. [read_buffer_count: <int> | default = 32] # Size of read buffers used when performing search on a vparquet block. This value times the read_buffer_count # is the total amount of bytes used for buffering when performing search on a parquet block. [read_buffer_size_bytes: <int> | default = 2097152] # Granular cache control settings for parquet metadata objects # Deprecated. See [Cache](#cache) section. cache_control: # Specifies if footer should be cached [footer: <bool> | default = false] # Specifies if column index should be cached [column_index: <bool> | default = false] # Specifies if offset index should be cached [offset_index: <bool> | default = false] ``` ### WAL The storage WAL configuration block. ```yaml # Where to store the wal files while they are being appended to. # Must be set. # Example: "/var/tempo/wal [path: <string> | default = ""] # When a span is written to the WAL it adjusts the start and end times of the block it is written to. # This block start and end time range is then used when choosing blocks for search. # This is also used for querying traces by ID when the start and end parameters are specified. To prevent spans too far # in the past or future from impacting the block start and end times we use this configuration option. # This option only allows spans that occur within the configured duration to adjust the block start and # end times. # This can result in trace not being found if the trace falls outside the slack configuration value as the # start and end times of the block will not be updated in this case. [ingestion_time_range_slack: <duration> | default = 2m] ``` ## Overrides Tempo provides an overrides module for users to set global or per-tenant override settings. ### Ingestion limits Tempo enforces ingestion limits at different points in the write path. For an overview of where each component sits in the pipeline, refer to [About the Tempo architecture](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/about-tempo-architecture/). The default limits in Tempo may not be sufficient in high-volume tracing environments. Errors including `RATE_LIMITED`/`TRACE_TOO_LARGE`/`LIVE_TRACES_EXCEEDED` occur when these limits are exceeded. See below for how to override these limits globally or per tenant. #### Standard overrides You can create an `overrides` section to configure ingestion limits that apply to all tenants of the cluster. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` #### Tenant-specific overrides There are two types of tenant-specific overrides: - runtime overrides - user-configurable overrides ##### Runtime overrides You can set tenant-specific overrides settings in a separate file and point `per_tenant_override_config` to it. This overrides file is dynamically loaded. It can be changed at runtime and reloaded by Tempo without restarting the application. These override settings can be set per tenant. ```yaml # /conf/tempo.yaml # Overrides configuration block overrides: per_tenant_override_config: /conf/overrides.yaml --- # /conf/overrides.yaml # Tenant-specific overrides configuration overrides: "<tenant-id>": ingestion: [rate_limit_bytes: <int>] # Honored by both global and local strategies. [burst_size_bytes: <int>] # Always local, regardless of rate strategy. [max_traces_per_user: <int>] global: [max_bytes_per_trace: <int>] metrics_generator: # Enable metrics-generator processors for this tenant. For all available fields, # refer to the metrics_generator block in the overrides defaults above. [processors: <list of strings>] # Optional per-tenant remote write headers, for example for per-tenant authentication. remote_write_headers: [<header-name>: <string>] # External forwarder names must match distributor.forwarders entries. [forwarders: <list of strings>] forwarder: [queue_size: <int> | default = 100] [workers: <int> | default = 2] # A "wildcard" override can be used that will apply to all tenants if a match is not found otherwise. "*": ingestion: [burst_size_bytes: <int>] [rate_limit_bytes: <int>] [max_traces_per_user: <int>] global: [max_bytes_per_trace: <int>] ``` When a tenant matches a per-tenant override entry, Tempo doesn't merge that entry field-by-field with `defaults`. Most fields require you to set every value a tenant needs within its own block (some settings, such as `ingestion.rate_strategy`, always use `defaults`). A few metrics-generator fields fall back to `defaults` when unset, but most, including `processors` and `remote_write_headers`, don't. For a complete per-tenant `metrics_generator` example, including `remote_write_headers` for per-tenant authentication and environment variable expansion, refer to [Multi-tenancy support](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-generator/multitenancy/). ##### User-configurable overrides These tenant-specific overrides are stored in an object store and can be modified using API requests. User-configurable overrides have priority over runtime overrides. Refer to [user-configurable overrides](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/user-configurable-overrides/) for more details. #### External forwarder queues Configure `overrides.defaults.forwarder`, or `forwarder` in a tenant runtime override: - `queue_size`: maximum buffered batches per queue (default: `100`). - `workers`: concurrent forwarding workers per queue (default: `2`). Each distributor creates a separate queue per tenant and enabled forwarder. These operator-only settings don't affect metrics-generator forwarding. Omitted or zero values use the built-in defaults; negative values are invalid. Tenant and wildcard entries don't inherit custom values from `overrides.defaults.forwarder` for omitted fields. After runtime overrides reload, queue settings update on the next 10-second check. Old queues drain asynchronously while replacement queues accept new batches. For example, use default queue settings and larger queues for one tenant. The `external-otlp` name must match a forwarder defined in `distributor.forwarders`: ```yaml # /conf/tempo.yaml overrides: defaults: forwarders: [external-otlp] forwarder: queue_size: 100 workers: 2 per_tenant_override_config: /conf/overrides.yaml --- # /conf/overrides.yaml overrides: tenant-a: forwarders: [external-otlp] forwarder: queue_size: 1000 workers: 4 ``` #### Ingestion rate strategy Ingestion limits are enforced at different points in the write path. The distributor enforces rate and burst limits before writing to Kafka. Live-stores and block-builders enforce per-trace size and active trace count limits asynchronously after consuming from Kafka. The `rate_strategy` setting controls how the distributor's rate limit scales across instances. It only affects `rate_limit_bytes`. | Strategy | When to use | How it works | |---|---|---| | **`local`** (default) | You want each distributor to independently handle a fixed rate, and you accept that the effective cluster rate grows as you add distributors. | Each distributor enforces the full configured `rate_limit_bytes` value. With 5 distributors at `30 MB/s`, the cluster allows up to `150 MB/s`. | | **`global`** | You need a predictable cluster-wide ingestion budget that stays constant regardless of how many distributors you run. | The configured `rate_limit_bytes` is divided across healthy distributors. With 5 distributors at `30 MB/s`, each allows `6 MB/s`. | ```yaml overrides: defaults: ingestion: [rate_strategy: <global|local> | default = local] ``` The following table shows where each ingestion limit is enforced and whether it is affected by `rate_strategy`: | Setting | Enforced by | Affected by `rate_strategy`? | |---|---|---| | `rate_limit_bytes` | Distributor | Yes | | `burst_size_bytes` | Distributor | No (always per instance) | | `max_traces_per_user` | Live-store | No (always per instance) | | `max_global_traces_per_user` | Not enforced at runtime | No (metric only, disabled by default) | | `max_bytes_per_trace` | Live-store, block-builder | No | ##### Examples Each distributor instance independently allows `30 MB/s`: ```yaml overrides: defaults: ingestion: rate_strategy: local rate_limit_bytes: 30000000 ``` All distributors share a total cluster rate of `30 MB/s`. With 5 distributors, each instance allows `6 MB/s`: ```yaml overrides: defaults: ingestion: rate_strategy: global rate_limit_bytes: 30000000 ``` For guidance on sizing these limits for your workload, refer to [Manage trace ingestion](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-trace-ingestion/). ## Usage-report By default, Tempo reports anonymous usage data about the shape of a deployment to Grafana Labs. This data is used to determine how common the deployment of certain features are, if a feature flag has been enabled, and which replication factors or block formats are used. By providing information on how people use Tempo, usage reporting helps the Tempo team decide where to focus their development and documentation efforts. No private information is collected, and all reports are completely anonymous. The following configuration values are used: - Receivers enabled - Frontend concurrency and version - Storage cache, backend, and configured block format - Ring replication factor, and `kvstore` - Features toggles enabled No performance data is collected. You can view the report by visiting this address on your Tempo instance: `http://localhost:3200/status/usage-stats` Refer to [Anonymous usage reporting](../configuration/anonymous-usage-reporting/) for detailed information on the information included in the report. ### Configure usage-reporting Reporting is controlled by a configuration option. You can disable the automatic reporting of this generic information using the following configuration: ```yaml usage_report: reporting_enabled: false ``` If you are using a Helm chart, you can enable or disable usage reporting by changing the `reportingEnabled` value. This value is available in the [tempo-distributed](https://github.com/grafana-community/helm-charts/tree/main/charts/tempo-distributed) and the [tempo](https://github.com/grafana/helm-charts/tree/main/charts/tempo) Helm charts. ```yaml # -- If true, Tempo will report anonymous usage data about the shape of a deployment to Grafana Labs reportingEnabled: true ``` ## Cache Use this block to configure caches available throughout the application. Multiple caches can be created and assigned roles which determine how they are used by Tempo. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Example configuration: ```yaml cache: background: writeback_goroutines: 5 caches: - roles: - parquet-footer memcached: host: memcached-instance - roles: - bloom redis: endpoint: redis-instance ``` ## Configure authentication Grafana Tempo doesn't come with any included authentication layer. You must run an authenticating reverse proxy in front of your services to prevent unauthorized access to Tempo (for example, nginx). Refer to [Manage authentication](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/authentication/) for more details. --- ### Sources Tempo Introduction Index (`docs/sources/tempo/introduction/_index.md`) --- aliases: - /docs/tempo/getting-started/traces - /docs/tempo/traces - /docs/grafana-cloud/send-data/traces/what-are-traces description: "What are traces? Learn about traces and how you can use them." keywords: - Grafana - traces - tracing title: Introduction weight: 120 --- # Introduction A trace represents the whole journey of a request or an action as it moves through all the nodes of a distributed system, especially containerized applications or microservices architectures. Traces are the ideal observability signal for discovering bottlenecks and interconnection issues. Traces are composed of one or more spans. A span is a unit of work within a trace that has a start time relative to the beginning of the trace, a duration, and an operation name for the unit of work. It usually has a reference to a parent span, unless it's the first, or root, span in a trace. It frequently includes key/value attributes that are relevant to the span itself, for example the HTTP method used in the request, as well as other metadata such as the service name, sub-span events, or links to other spans. {{< docs/box >}} {{< docs/icon-heading heading="## Learn about tracing through a strategy game" >}} This blog post walks you through Game of Traces, a grand strategy game you can play to learn the key concepts of OpenTelemetry tracing — and, of course, heroically defend a few kingdoms along the way. [Read more about the Game of Traces](https://grafana.com/blog/2025/08/11/learn-opentelemetry-tracing-through-a-grand-strategy-game-introducing-game-of-traces/) {{< figure src="https://a-us.storyblok.com/f/1022730/350x350/55b219645d/game-of-traces-logo_2.png" alt="A screenshot of the Game of Traces logo" max-width="none" >}} {{< /docs/box >}} By definition, traces are never complete. You can always push another batch of spans, even if days have passed since the last one. When receiving a query requesting a stored trace, tracing backends like Tempo find all the spans for that specific trace and collate them into a returned result. Retrieving trace data can have issues if traces are extremely large. {{< youtube id="ZirbR0ZJIOs" >}} ## Example of traces Firstly, a user on your website enters their email address into a form to sign up for your mailing list. They click **Enter**. This initial transaction has a trace ID that's subsequently associated with every interaction in the chain of processes within the system. Next, the user's email address is data that flows through your system. In a cloud computing world, it's possible that clicking that one button triggers many downstream processes on various microservices operating across many different nodes in your compute infrastructure. As a result, the email address goes to a microservice responsible for verification. If the email passes this check, then the database stores the address. Along the way, an anonymizing microservice strips personally identifying data from the address and adds additional metadata before sending it along to a marketing qualifying microservice. This microservice determines whether the request came from a targeted part of the internet. Services respond and data flows back from each, sometimes triggering additional events across the system. Along the way, nodes write logs on which those services run with a time stamp showing when the info passed through. Finally, the request and response activity end. No other spans append to that trace ID. ## Traces and trace IDs Setting up tracing adds an identifier, or trace ID, to all of these events. The trace ID generates when the request initiates. That same trace ID applies to every span as the request and response generate activity across the system. The trace ID lets you trace, or follow, a request as it flows from node to node, service to microservice to lambda function to wherever it goes in your chaotic, cloud computing system and back again. This is recorded and displayed as spans. Here's an example showing two pages in Grafana. The first, numbered 1, shows a query using the **Explore** feature. In the query results, you can see a **TraceID** field that was added to an application. That field contains a **Tempo** trace ID. The second page, numbered 2, uses the same **Explore** feature to perform a Tempo search using that **TraceID**. It then shows a set of spans as horizontal bars, each bar denoting a different part of the system. ## What are traces used for? Traces can help you find bottlenecks. Applications like Grafana can visualize traces to give a graphic representation of how long it takes for each step in the data flow pathway to complete. It can show where additional requests initiate and end, and how your system responds. This data helps you locate problem areas, often in places you never would have anticipated or found without this ability to trace the request flow. {{< youtube id="0tlp7QCPu0k" >}} ## Learn more To continue learning about traces and Tempo, refer to: - [Traces and telemetry](./telemetry/): how traces relate to metrics, logs, and profiles. - [Trace structure](./trace-structure/): the parts of a span and the attributes you can query. - [Tempo architecture](./architecture/): how Tempo ingests, stores, and queries traces. - [Tempo and AI](./tempo-and-ai/): query traces with AI agents through the MCP server and LLM-optimized APIs. - [Glossary](./glossary/): definitions of common tracing terms. For real-world examples, refer to [User journeys: How tracing can help you](../solutions-with-traces/) elsewhere in the Tempo documentation. --- ### Sources Tempo Metrics From Traces Index (`docs/sources/tempo/metrics-from-traces/_index.md`) --- title: Metrics from traces description: Learn about how we can correlate traces and metrics. weight: 700 aliases: - ./getting-started/metrics-from-traces/ # /docs/tempo/next/getting-started/metrics-from-traces/ --- # Metrics from traces Metrics provide a powerful insight into the systems you are monitoring with your observability strategy. Instead of running an additional service to generate metrics, you can use Grafana Tempo to generate metrics from traces. Grafana Tempo can generate metrics from tracing data using the metrics-generator and TraceQL metrics. Refer to the table for a summary of these metrics and their capabilities. | | Metrics-generator | TraceQL metrics | | -------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Functionality | An optional component within Tempo that processes incoming spans to produce predefined metrics: RED (Rate, Error, Duration) metrics, service graphs, and host info. | A feature in Tempo that allows for on-the-fly computation of metrics directly from trace data using the TraceQL query language, without the need for a separate metrics storage backend. | | Capabilities | **Span metrics:** Calculates the total count and duration of spans based on dimensions like service name, operation, span kind, status code, and other span attributes. <br> **Service graphs**: Analyzes traces to map relationships between services, identifying transactions and recording metrics related to request counts and durations. | Ad-hoc aggregation and analysis of trace data by applying functions to trace query results, similar to how LogQL operates with logs. | | Output | The generated metrics are written to a Prometheus-compatible database, enabling integration with time-series databases for storage and analysis. | Generates metrics dynamically at query time, facilitating flexible and detailed investigations into specific behaviors or patterns within the trace data. | | Use case | Ideal for continuous monitoring and alerting, leveraging predefined metrics that are stored in a time-series database. Less expressive for trace-specific analysis as it focuses on standard telemetry dimensions and RED metrics. | More expressive and flexible for analyzing trace data directly, enabling complex trace-based queries and fine-grained exploration. Suited for exploratory analysis and debugging, allowing users to derive insights from trace data without prior metric definitions or storage considerations. | | Setup | Configure the metrics-generator in the Tempo configuration file, enable processors like span metrics or service graphs, and send metrics to a Prometheus-compatible database. | TraceQL metrics work out of the box. Configure a Tempo data source in Grafana. | | Query language | Metrics are consumed using PromQL via Prometheus/Grafana. | Uses TraceQL which has a PromQL-inspired syntax, but not all PromQL features are supported; it's a similar but distinct subset with different semantics. | ## Why metrics-generator and TraceQL metrics can show different results The metrics-generator and TraceQL metrics use different data paths. If you query the same spans both ways, the results may not match. The following factors cause differences: | Factor | What happens | How to detect | | --- | --- | --- | | **Different pipelines** | The metrics-generator processes spans at ingestion time in a streaming processor and writes pre-computed metrics to Prometheus. TraceQL metrics query raw stored spans at query time. These are independent data paths with different processing models. | Compare span counts from `traces_spanmetrics_calls_total` with a `count_over_time()` TraceQL query for the same service and time range. | | **Late-arriving spans** | Spans that arrive after the `metrics_ingestion_time_range_slack` window (default: 30s) are stored in Tempo but never counted by the metrics-generator. TraceQL queries can still find them. | Query `tempo_metrics_generator_spans_discarded_total` by reason. In Grafana Cloud, query `grafanacloud_traces_instance_metrics_generator_discarded_spans_per_second{reason="outside_metrics_ingestion_slack"}`. Refer to [Discarded spans](/docs/tempo/<TEMPO_VERSION>/troubleshooting/metrics-generator/#discarded-spans-in-the-generator). | | **Active series limits** | When the per-tenant active series limit is reached, the metrics-generator routes new metric series to overflow buckets (labeled `metric_overflow="true"`) instead of tracking them individually. There's no customer-visible error. Metrics appear incomplete because detail is collapsed into the overflow series. | Query `tempo_metrics_generator_registry_series_limited_total` to detect limited series. Use `tempo_metrics_generator_registry_active_series_demand_estimate` to see true demand. Refer to [Max active series](/docs/tempo/<TEMPO_VERSION>/troubleshooting/metrics-generator/#max-active-series). | If you see unexpected differences, start by checking the late-span discard metric and the active series limit metric. These two causes account for the majority of discrepancies in environments where traces reach Tempo unsampled. ## Metrics-generator Tempo can generate metrics from ingested traces using the metrics-generator, an optional Tempo component. The metrics-generator runs processors including [service graphs](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/), [span metrics](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics), and host info. The metrics-generator looks at incoming spans, and calculates rate, error, and duration (RED) metrics from them, which it then writes to a time series database like Prometheus. By querying Prometheus, you can see the overall request rate, erroring request rate, and distribution of request latency in your system. By using the labels on those metrics, you can get a more granular view of request rate, error rate, and latency at a per-service, per-namespace, or per-operation level. | Useful for investigating | Metric | Meaning | | ---------------------------------------- | -------- | -------------------------------------------------------------- | | Unusual spikes in activity | Rate | Number of requests per second | | Overall issues in your tracing ecosystem | Error | Number of those requests that are failing | | Response times and latency issues | Duration | Amount of time those requests take, represented as a histogram | The metrics-generator generates metrics from tracing data using the `service_graphs` and `span_metrics` processors, which are written to a Prometheus-compatible backend. The metrics-generator processes spans and writes metrics using the Prometheus remote write protocol. For more information, refer to [Metrics generator](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-generator/). ### Use-cases for span metrics Span metrics are of particular interest if your system isn't monitored with metrics but it has distributed tracing implemented. You get out-of-the-box metrics from your tracing pipeline. {{< admonition type="note" >}} In Grafana Cloud, the metrics-generator is disabled by default. Contact Grafana Support to enable metrics generation in your organization. {{< /admonition >}} After the metrics-generator is enabled in your organization, refer to [Metrics-generator configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#metrics-generator) for information about metrics-generator options. These metrics exist in your Hosted Metrics instance and can also be used to generate powerful custom dashboards. <p align="center"><img src="/media/docs/tempo/intro/trace_custom_metrics_dash.png" alt="Trace custom metrics dashboard"></p> The metrics-generator automatically generates exemplars as well which allows easy metrics to trace linking. [Exemplars](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/) are available in Grafana Cloud. {{< figure src="/media/docs/grafana/exemplars/screenshot-exemplar-span-details.png" class="docs-image--no-shadow" max-width= "600px" caption="Span details" >}} ## TraceQL metrics Traces are a unique observability signal that contain causal relationships between the components in your system. - Do you want to know how many database calls across all systems are downstream of your application? - What services beneath a given endpoint are currently failing? - What services beneath an endpoint are currently slow? TraceQL metrics can answer all these questions by parsing your traces in aggregate. You can query data generated by TraceQL metrics in a similar way that you would query results stored in Prometheus, Grafana Mimir, or other Prometheus-compatible Time-Series-Database (TSDB). TraceQL metrics queries allows you to calculate metrics on trace span data on-the-fly with Tempo (your tracing database), without requiring a time-series-database like Prometheus. TraceQL metrics, powered by the API of the same name, return Prometheus-like time series for a given metrics query. Metrics queries apply a function to trace query results. TraceQL metrics power the [Grafana Traces Drilldown app](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/simplified-exploration/traces/). You can explore the power of visualizing your metrics in the Grafana Traces Drilldown app using Grafana Play. {{< docs/play title="the Grafana Play site" url="https://play.grafana.org/a/grafana-exploretraces-app/explore" >}} Refer to these resources for additional information: - [Solves problems with TraceQL metrics queries](https://grafana.com/docs/tempo/<TEMPO_VERSION>/solutions-with-traces/solve-problems-metrics-queries/) - [Configure TraceQL metrics](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/configure-traceql-metrics/) - [TraceQL metrics queries](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/) - [TraceQL metrics functions](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/functions/) --- ### Sources Tempo Operations Index (`docs/sources/tempo/operations/_index.md`) --- title: Manage Tempo menuTitle: Manage description: Learn how to manage and tune Tempo. weight: 450 aliases: - ./ingester_pvcs/ # https://grafana.com/docs/tempo/next/operations/ingester_pvcs/ --- # Manage Tempo This section provides resources for managing and tuning Tempo. {{< section >}} In addition, the [Tempo runbooks](https://github.com/grafana/tempo/blob/main/operations/tempo-mixin/runbook.md) can help with remediating operational issues. --- ### Sources Tempo Reference Tempo Architecture Index (`docs/sources/tempo/reference-tempo-architecture/_index.md`) --- title: Tempo architecture reference menuTitle: Architecture reference description: Detailed reference for the Tempo architecture, components, and design decisions. weight: 600 topicType: introduction versionDate: 2026-03-20 --- # Tempo architecture reference This section provides a detailed technical reference for the Grafana Tempo architecture. For a high-level overview, refer to the [Tempo architecture](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/architecture/) page. {{< section menuTitle="true" >}} --- ### Sources Tempo Release Notes Index (`docs/sources/tempo/release-notes/_index.md`) --- title: Release notes description: Release notes for Grafana Tempo weight: 850 --- # Release notes Release notes for Grafana Tempo are in the CHANGELOG for the release and listed here by version number. Packages are linked in the [Assets section for each Tempo release](https://github.com/grafana/tempo/releases). {{< section >}} --- ### Sources Tempo Set Up For Tracing Index (`docs/sources/tempo/set-up-for-tracing/_index.md`) --- title: Set up for tracing menuTitle: Set up for tracing description: Instructions for setting up Tempo for traces weight: 300 aliases: - ./getting-started/ # /docs/tempo/next/getting-started/ refs: examples: - pattern: /docs/tempo/ destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/example-demo-app/ - pattern: /docs/enterprise-traces/ destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/setup/ setup: - pattern: /docs/tempo/ destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/ - pattern: /docs/enterprise-traces/ destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/setup/ deploy: - pattern: /docs/tempo/ destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/ - pattern: /docs/enterprise-traces/ destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/setup/hardware-requirements/ configure-alloy: - pattern: /docs/tempo/ destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/grafana-alloy/ - pattern: /docs/enterprise-traces/ destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/setup/set-up-get-tenants/ --- # Set up for tracing Grafana Tempo is an open source and high-scale distributed tracing backend. Tempo lets you search for traces, generate metrics from spans, and link your tracing data with logs and metrics. Tempo also powers Grafana Cloud Traces and Grafana Enterprise Traces. Distributed tracing visualizes the lifecycle of a request as it passes through a set of applications. For more information about traces, refer to [Introduction to traces](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/). To set up for tracing, you need to: 1. Plan your Tempo deployment to meet your needs. Refer to the [Plan your deployment](./setup-tempo/plan/) documentation. Check out the [examples](ref:examples) for ideas on how to get started. 1. Deploy Tempo using the [Set up Tempo documentation](ref:setup). Tempo offers different deployment modes. Refer to the [Deployment documentation](ref:deploy) section for more information. You can download the latest version from the [Tempo Releases page](https://github.com/grafana/tempo/releases). 1. Instrument your application or service to emit traces. Use the [Set up instrumentation](./instrument-send/set-up-instrumentation/) documentation. 1. Set up a collector, like Grafana Alloy or the OpenTelemetry Collector, to offload traces from your application and forward them to Tempo. Refer to the [Set up a collector](./instrument-send/set-up-collector/) documentation. ## Requirements for tracing Tracing requires client instrumentation, a pipeline for gathering and sending traces, and a backend. - Client instrumentation enables your application or service to emit traces. - The pipeline component offloads traces from your application and forwards them to a backend. You can use Grafana Alloy or the OpenTelemetry Collector. - The backend is the storage and retrieval system for traces, which in this case is Tempo. Refer to the [Tracing pipeline components](#tracing-pipeline-components) section for more information about these components. {{< youtube id="zDrA7Ly3ovU" >}} ### Tracing pipeline components To build a tracing pipeline, you need four major components: client instrumentation, pipeline, backend, and visualization. This diagram illustrates a tracing system configuration: <p align="center"><img src="/media/docs/tempo/intro/tempo-get-started-overview.svg" alt="Tracing pipeline overview"></p> ### Client instrumentation Client instrumentation (1 in the diagram) is the first building block to a functioning distributed tracing visualization pipeline. Client instrumentation is the process of adding instrumentation points in the application that create and offload spans. {{< admonition type="note" >}} To learn more about instrumentation, refer to the [Set up instrumentation](./instrument-send/set-up-instrumentation/) documentation to learn how to instrument your favorite language for distributed tracing. {{< /admonition >}} ### Pipeline (Grafana Alloy) After you instrument your application for tracing, the traces are sent to a backend for storage and visualization. You can build a tracing pipeline that offloads spans from your application, buffers them, and forwards them to a backend. While applications can send traces directly to Tempo, using a pipeline (like Grafana Alloy or the OpenTelemetry Collector) is strongly recommended for production deployments. Pipelines provide important benefits such as batching, retry logic, and decoupling your application from backend availability. Grafana Alloy is a service that's deployed close to the application, either on the same node or within the same cluster (in Kubernetes) to quickly offload traces from the application and forward them to a storage backend. Alloy also abstracts features like trace batching to a remote trace backend store, including retries on write failures. To learn more about Grafana Alloy and how to set it up for tracing with Tempo, refer to [Grafana Alloy configuration for tracing](ref:configure-alloy). {{< admonition type="note" >}} The [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector) / [Jaeger Agent](https://www.jaegertracing.io/docs/latest/deployment/) can also be used at the agent layer. Refer to [this blog post](/blog/2021/04/13/how-to-send-traces-to-grafana-clouds-tempo-service-with-opentelemetry-collector/) to see how to use the OpenTelemetry Collector with Tempo. {{< /admonition >}} ### Backend (Tempo) The tracing backend stores and retrieves traces on demand. Grafana Tempo is the distributed tracing backend used to store and query traces. ## Visualize tracing data with Grafana Grafana and Grafana Cloud have a built-in Tempo data source that you can use to query Tempo and visualize traces. For more information, refer to the [Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/) and the [Visualize tracing data in Grafana](https://grafana.com/docs/tempo/<TEMPO_VERSION>/visualize-traces/) topics. --- ### Sources Tempo Shared Index (`docs/sources/tempo/shared/index.md`) --- description: No description necessary for a shared file index. headless: true --- --- ### Sources Tempo Solutions With Traces Index (`docs/sources/tempo/solutions-with-traces/_index.md`) --- description: 'Learn how tracing data can help you understand application insights and performance as well as triaging issues in your services and applications.' keywords: - Grafana - traces - tracing title: Use traces to find solutions menuTitle: Use cases aliases: - ./introduction/solutions-with-traces/ # /docs/tempo/<TEMPO_VERSION>/introduction/solutions-with-traces/ weight: 150 --- # Use traces to find solutions {{< shared id="traces-intro-2" >}} Tracing is best used for analyzing the performance of your system, identifying bottlenecks, monitoring latency, and providing a complete picture of requests processing. - Decrease mean time to repair and mean time to identify an issue by pinpointing exactly where errors or latency are occurring within a transaction across multiple services. - Optimize bottlenecks and long-running code by visualizing the path and duration of requests. Tracing can help identify bottleneck operations and long-running pieces of code that could benefit from optimization. - Detect issues with generated metrics. Tracing generates metrics related to request rate, error rate, and duration of requests. You can set alerts against these high-level signals to detect problems. - Seamless telemetry correlation. Use tracing in conjunction with logs and metrics for a comprehensive view of events over time, during an active incident, or for root-cause analysis. Tracing shows relationships between services and dependencies. - Monitor compliance with policies. Business policy adherence ensures that services are correctly isolated using generated metrics and generated service graphs. {{< /shared >}} Each use case provides real-world examples, including the background of the use case and how tracing highlighted and helped resolve any issues. --- ### Sources Tempo Traceql Index (`docs/sources/tempo/traceql/_index.md`) --- title: TraceQL menuTitle: Query with TraceQL description: Learn about TraceQL, the query language for traces weight: 600 aliases: - /docs/tempo/latest/traceql/ keywords: - Tempo query language - query language - TraceQL --- # TraceQL TraceQL is a query language designed for selecting traces in Tempo. Distributed traces contain a wealth of information, and tools like auto-instrumentation simplify the process of capturing data. Extracting value from traces can be much harder. For example, Tempo metrics-generator can aggregate traces into service graphs and span metrics, and exemplars allow you to navigate from a spike in API latency to a trace that contributed to that spike. But traces can do so much more. Traces are the flow of events throughout your components. They have a tree structure—-with a root, branches, and leaves—-arbitrary key/value data at any location, and of course timestamps. For example, you can use traces to perform root cause analyses (RCA) on a service outage and use TraceQL to pinpoint the root cause. Refer to [Diagnose errors with traces](/docs/tempo/<TEMPO_VERSION>/solutions-with-traces/traces-diagnose-errors/#diagnose-errors-with-traces) for a use case example. ## Get started with TraceQL Use these references to get started with TraceQL: - Determine the information you want to query by understanding the [relationship of queries to trace structure](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/construct-traceql-queries/) and spans. - [Construct a query to locate the information](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/construct-traceql-queries/). Use the examples and language reference to learn about the syntax and semantics of TraceQL. TraceQL uses similar syntax and semantics as [PromQL](/blog/2020/02/04/introduction-to-promql-the-prometheus-query-language/) and [LogQL](/docs/loki/latest/logql/), where possible. {{< vimeo 796408188 >}} ## How can you use TraceQL? You can use TraceQL queries using the command line or in Grafana with the query editor and query builder. The query editor and builder are available in the [Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/) for Grafana Explore. In addition, you can use Traces Drilldown to investigate your tracing data without writing TraceQL queries. For more information, refer to the [Traces Drilldown](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/simplified-exploration/traces/) documentation. For more information, refer to [Write TraceQL queries in Grafana](http://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/query-editor/). ### Metrics from traces with TraceQL TraceQL metrics generate metrics from traces and let you use TraceQL to query metrics. Refer to [TraceQL metrics](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/) for more information. ## Resources TraceQL requires the Parquet columnar format, which is the default block format for Tempo. Refer to the [Apache Parquet backend](http://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/parquet) documentation. Refer to the [Tempo release notes](https://grafana.com/docs/tempo/<TEMPO_VERSION>/release-notes/) for the latest updates to TraceQL. --- ### Sources Tempo Troubleshooting Index (`docs/sources/tempo/troubleshooting/_index.md`) --- title: Troubleshoot Tempo menuTitle: Troubleshoot description: Learn how to troubleshoot operational issues for Grafana Tempo. weight: 700 aliases: - ../operations/troubleshooting/ # https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/troubleshooting/ --- # Troubleshoot Tempo This section helps with day zero operational issues that may come up when getting started with Tempo. The documents walk you through debugging each part of the write and query pipeline to diagnose issues. In addition, the [Tempo runbook](https://github.com/grafana/tempo/blob/main/operations/tempo-mixin/runbook.md) can help with remediating operational issues. ## Sending traces - [Spans are being refused with "pusher failed to consume trace data"](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/send-traces/max-trace-limit-reached/) - [Is Grafana Alloy sending to the backend?](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/send-traces/alloy/) ## Querying - [Unable to find my traces in Tempo](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/unable-to-see-trace/) - [Error message "Too many jobs in the queue"](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/too-many-jobs-in-queue/) - [Queries fail with 500 and "error using pageFinder"](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/bad-blocks/) - [I can search traces, but there are no service name or span name values available](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/search-tag) - [Error message `response larger than the max (<number> vs <limit>)`](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/response-too-large/) - [Search results don't match trace lookup results with long-running traces](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/long-running-traces/) ## Metrics-generator - [Metrics or service graphs seem incomplete](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/metrics-generator/) ## Out-of-memory errors - [Set the max attribute size to help control out of memory errors](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/out-of-memory-errors/) --- ### Sources Tempo Visualize Traces Index (`docs/sources/tempo/visualize-traces/_index.md`) --- title: Visualize tracing data in Grafana menuTitle: Visualize data description: Use Grafana to visualize, query, and explore your tracing data with the built-in Tempo data source. weight: 650 aliases: - /docs/tempo/latest/introduction/tempo-in-grafana/ - /docs/tempo/latest/getting-started/tempo-in-grafana/ --- # Visualize tracing data in Grafana Grafana has a built-in Tempo data source that can be used to query Tempo and visualize traces. This page describes the high-level features and their availability. Use the latest versions for best compatibility and stability. [//]: # 'Shared content for best practices for traces' [//]: # 'This content is located in /tempo/docs/sources/shared/tempo-in-grafana.md' {{< docs/shared source="tempo" lookup="tempo-in-grafana.md" version="<TEMPO_VERSION>" >}} --- ### Sources Helm Charts Tempo Distributed Get Started Helm Charts Index (`docs/sources/helm-charts/tempo-distributed/get-started-helm-charts/_index.md`) --- description: Learn how to get started with Grafana Tempo using the Helm chart. menuTitle: Get started title: Get started with Grafana Tempo using the Helm chart weight: 20 keywords: - Helm chart - Kubernetes - Grafana Tempo topicType: task versionDate: 2026-07-15 --- # Get started with Grafana Tempo using the Helm chart {{< admonition type="note" >}} The `tempo-distributed` Helm chart is now maintained by the community. The chart has moved to the [grafana-community/helm-charts](https://github.com/grafana-community/helm-charts/tree/main/charts/tempo-distributed) repository. {{< /admonition >}} {{< admonition type="note" >}} Grafana Enterprise Traces (GET) is no longer maintained in this chart. The enterprise templates (`admin-api`, `enterprise-gateway`, `enterprise-federation-frontend`, `provisioner`, and `tokengen`) have been removed. For enterprise deployments, use the [grafana/helm-charts](https://github.com/grafana/helm-charts) repository. {{< /admonition >}} The `tempo-distributed` Helm chart allows you to configure, install, and upgrade Grafana Tempo within a Kubernetes cluster. Tempo 3.0 uses a Kafka-based architecture: distributors write spans to a Kafka-compatible broker, block-builders consume from Kafka to write blocks to object storage, and live-stores serve recent-data queries. As a result, this procedure requires an external Kafka-compatible broker and an external S3-compatible object store, neither of which the chart deploys for you. Before deploying the chart, refer to [Configure a Kafka-compatible backend](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/configure-kafka/) for topic sizing, retention, durability, security, and monitoring guidance. Using this procedure, you need to: - Create a custom namespace within your Kubernetes cluster - Install Helm and the Grafana Community `helm-charts` repository - Provide an external S3-compatible object store for traces - Provide a Kafka-compatible broker for the ingest path - Install Tempo using Helm To learn more about Helm, read the [Helm documentation](https://helm.sh/). ## Before you begin These instructions are common across any flavor of Kubernetes. They also assume that you know how to install, configure, and operate a Kubernetes cluster. It also assumes that you have an understanding of what the `kubectl` command does. {{< admonition type="warning" >}} This procedure is primarily aimed at local or development setups. {{< /admonition >}} ### Hardware requirements The Tempo 3.0 microservices deployment runs more components than earlier versions, including block-builders, live-stores, and a backend-scheduler and backend-worker, alongside an external Kafka broker and object store. - The main example uses the chart default of three Kafka partitions, which runs three block-builder and three live-store replicas. For a smaller single-partition footprint suitable for a single node (one block-builder and one live-store replica), use the [Optional: Quick test with a local S3-compatible store and Kafka](#optional-quick-test-with-a-local-s3-compatible-store-and-kafka) section and a Kubernetes node with a minimum of 6 cores and 16 GB RAM. - For production deployments, start with approximately one active live-store replica for every 10 MB/s of peak ingestion and validate that estimate against your workload. With the default `blockBuilder.config.partitions_per_instance: 1`, use the same number of block-builder replicas. The Kafka topic can contain significantly more partitions than Tempo currently has active, so provision extra partitions for future growth. Refer to [Plan your deployment](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/plan/) for more sizing guidance. Tempo 3.0 can run more specialized pods than the previous architecture, but pod count alone doesn't determine total cost of ownership. The architecture lowers total cost by eliminating duplicate block building, operating at replication factor 1 (RF1) to avoid duplicate blocks in object storage, and supporting autoscaling. ### Software requirements - Kubernetes 1.25 or later (refer to [Kubernetes installation documentation](https://kubernetes.io/docs/setup/)) - The `kubectl` command for your version of Kubernetes - Helm 3 or later (refer to [Helm installation documentation](https://helm.sh/docs/intro/install/)) - A Kafka-compatible broker reachable from the cluster (for example, Apache Kafka, Redpanda, or WarpStream). The chart doesn't deploy Kafka. ### Additional requirements Verify that you have: - Access to the Kubernetes cluster. - Enabled persistent storage in the Kubernetes cluster, which has a default storage class setup. - Access to an external S3-compatible object store, such as Amazon S3, Azure Blob Storage, or Google Cloud Platform. The chart no longer bundles MinIO. For local testing, you can run an S3-compatible store yourself using one of the options in [S3-compatible local stores for testing](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/hosted-storage/s3/#s3-compatible-local-stores-for-testing). Refer to the [Optional: Other storage options](#optional-other-storage-options) section for more information. - A reachable Kafka-compatible broker for the ingest path. - DNS service works in the Kubernetes cluster. Refer to [Debugging DNS resolution](https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/) in the Kubernetes documentation. - Optional: Set up an ingress controller in the Kubernetes cluster, for example [ingress-nginx](https://kubernetes.github.io/ingress-nginx/). {{< admonition type="note" >}} If you want to access Tempo from outside of the Kubernetes cluster, you may need an ingress. Ingress-related procedures are optional. Note that [ingress-nginx](https://github.com/kubernetes/ingress-nginx) is being retired and shouldn't be used in production environments. {{< /admonition >}} ## Create a custom namespace and add the Helm repository Using a custom namespace solves problems later on because you don't have to overwrite the default namespace. 1. Create a unique Kubernetes namespace, for example, `tempo-test`: ```bash kubectl create namespace tempo-test ``` For more details, refer to the Kubernetes documentation about [Creating a namespace](https://kubernetes.io/docs/tasks/administer-cluster/namespaces/#creating-a-new-namespace). 1. Set up a Helm repository using the following commands: ```bash helm repo add grafana-community https://grafana-community.github.io/helm-charts helm repo update ``` {{< admonition type="note" >}} The Helm chart at [https://grafana-community.github.io/helm-charts](https://grafana-community.github.io/helm-charts) is maintained by the community. {{< /admonition >}} ## Set Helm chart values The Helm chart for Tempo includes a file called `values.yaml`, which contains default configuration options. In this procedure, you create a local file called `custom.yaml` in a working directory. When you use Helm to deploy the chart, you can specify that Helm uses your `custom.yaml` to augment the default `values.yaml` file. The `custom.yaml` file sets the storage, ingest, and traces options. The `storage` section points Tempo at your object store, the `ingest` section connects Tempo to your Kafka broker, and the `traces` section configures the distributor's receiver protocols. After creating the file, you have the option to make changes in that file as needed for your deployment environment. To customize your Helm chart values: 1. Create a `custom.yaml` file in your working directory. 1. From the example below, copy and paste the Tempo Helm chart values into your file. 1. Save your `custom.yaml` file. 1. Set the `storage` section to point at your external S3-compatible object store. Further down this page are instructions for customizing your trace storage configuration options. 1. Set the `ingest.kafka` section to point at your Kafka broker. The chart's default Kafka topic name is `tempo-traces`. If your topic has a different name, set `ingest.kafka.topic` to that exact name. Set `liveStore.replicas` to the initial number of active Tempo partitions. With the default `blockBuilder.config.partitions_per_instance: 1`, set `blockBuilder.replicas` to the same value. The Kafka topic can contain significantly more partitions than these replica counts. 1. Set your traces values to configure the receivers on the Tempo distributor. 1. Save the changes to your file. {{< admonition type="note" >}} Tempo exposes partition lifecycle APIs that autoscaling controllers can use for safe scale-down. KEDA can determine the desired replica count, but safe scale-down also requires a controller to orchestrate the partition lifecycle. The community chart doesn't yet implement this integration for live-stores and block-builders. Until that integration is available, don't attach a generic HPA directly to these StatefulSets or reduce their replicas without first draining the affected Tempo partitions. Refer to [Scale down a Helm deployment](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/configure-kafka/#scale-down-a-helm-deployment) for the manual procedure, and check the [upstream chart](https://github.com/grafana-community/helm-charts/tree/main/charts/tempo-distributed) for current capabilities. {{< /admonition >}} ### Tempo Helm chart values This sample file contains example values for installing Tempo using Helm. {{< collapse title="Example Tempo values file" >}} ```yaml --- # Point Tempo at your external S3-compatible object store. storage: trace: backend: s3 s3: access_key: "grafana-tempo" secret_key: "supersecret" bucket: "tempo-traces" endpoint: "<s3-endpoint>:9000" insecure: true # remove once TLS is configured # Connect Tempo to your Kafka-compatible broker. ingest: kafka: address: "<kafka-broker>:9092" topic: tempo-traces auto_create_topic_enabled: true # Number of partitions for the auto-created topic. This example uses the chart # default and initially activates all three partitions. A pre-created production # topic can have more partitions than the initial Tempo replica counts. auto_create_topic_default_partitions: 3 # One block-builder replica per active partition with the default # blockBuilder.config.partitions_per_instance value of 1. blockBuilder: replicas: 3 # One live-store replica per active Tempo partition. liveStore: replicas: 3 # Specifies which trace protocols the distributor accepts. traces: otlp: grpc: enabled: true http: enabled: true zipkin: enabled: false jaeger: thriftHttp: enabled: false ``` {{< /collapse >}} ### Set your storage option Before you run the Helm chart, you need to configure where to store trace data. The `storage` block defined in the `values.yaml` file configures the storage that Tempo uses for trace storage. {{< admonition type="warning" >}} The chart no longer bundles MinIO. Setting `minio.enabled: true` now fails the install. You must point Tempo at an externally managed S3-compatible object store through the `storage.trace.s3` block. {{< /admonition >}} Configure the `storage.trace.s3` block in your `custom.yaml` to point at your object store: ```yaml --- storage: trace: backend: s3 s3: access_key: "grafana-tempo" secret_key: "supersecret" bucket: "tempo-traces" endpoint: "<s3-endpoint>:9000" insecure: true # remove once TLS is configured ``` The Amazon S3 example is identical, except the `endpoint` and `insecure` options are dropped. For other providers, refer to the [Optional: Other storage options](#optional-other-storage-options) section. #### Optional: Quick test with a local S3-compatible store and Kafka For local testing only, you can run an S3-compatible object store and a Kafka broker yourself, then point Tempo at them. Neither is suitable for production. 1. Stand up a local S3-compatible object store using one of the options in [S3-compatible local stores for testing](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/hosted-storage/s3/#s3-compatible-local-stores-for-testing). SeaweedFS is the recommended option. Create a bucket named `tempo-traces`, then set `storage.trace.s3.endpoint`, `access_key`, and `secret_key` in your `custom.yaml` to match. 1. Run a single-node Kafka broker in the cluster. The following manifest deploys a plain Kafka broker in a `kafka-test` namespace, suitable for evaluation: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` 1. Set `ingest.kafka.address` in your `custom.yaml` to `kafka.kafka-test.svc.cluster.local:9092`. For this single-node local test, set `blockBuilder.replicas`, `liveStore.replicas`, and `ingest.kafka.auto_create_topic_default_partitions` all to `1` to reduce the resource footprint. For production, provision the Kafka topic for your maximum expected active partition count and set the initial Tempo replica counts for the partitions you want active. The Kafka topic can have significantly more partitions than the current block-builder and live-store replica counts. #### Optional: Other storage options You can enable persistent storage in the Kubernetes cluster, which has a default storage class setup. To change the default, refer to the [StorageClass using Kubernetes documentation](https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/). You can use any S3-compatible store, or a bucket like Amazon S3, Azure Blob Storage, or Google Cloud Platform. Each storage provider has a different configuration stanza. You need to update your configuration based upon your storage provider. Refer to the [`storage` configuration block](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration#storage) for information on storage options. Update the `storage` configuration options based upon your requirements: - [Amazon S3 configuration documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/hosted-storage/s3). The Amazon S3 example is identical to the S3-compatible example above, except the `endpoint` and `insecure` options are dropped. - [Azure Blob Storage configuration documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/hosted-storage/azure) - [Google Cloud Storage configuration documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/hosted-storage/gcs) #### Azure with the `local_blocks` and metrics-generator processors [//]: # "Shared content for local_blocks and metrics-generator in Azure blob storage when using Helm" [//]: # "This content is located in /tempo/docs/sources/shared/azure-metrics-generator.md" {{< docs/shared source="tempo" lookup="azure-metrics-generator.md" version="<TEMPO_VERSION>" >}} For more information about the local-blocks processor, refer to [Configure TraceQL metrics](https://grafana.com/docs/tempo/next/metrics-from-traces/metrics-queries/configure-traceql-metrics). ### Set traces receivers The Helm chart values in your `custom.yaml` file are configured to use OTLP. If you are using other receivers, then you need to configure them. You can configure Tempo to receive data from OTLP, Jaeger, Zipkin and Kafka. The following example enables OTLP on the distributor. For other options, refer to the [distributor documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration#distributor) The example used in this procedure has OTLP enabled. Enable any other protocols based on your requirements. ```yaml traces: otlp: grpc: enabled: true http: enabled: true ``` ### Optional: Add custom configurations There are many configuration options available in the `tempo-distributed` Helm chart. This procedure only covers the minimum configuration required to launch Tempo in a basic deployment. You can add values to your `custom.yaml` file to set custom configuration options that override the defaults present in the Helm chart. The [`tempo-distributed` Helm chart's README](https://github.com/grafana-community/helm-charts/blob/main/charts/tempo-distributed/README.md) contains a list of available options. The `values.yaml` files provides the defaults for the Helm chart. Use the following command to see all of the configurable parameters for the `tempo-distributed` Helm chart: ```bash helm show values grafana-community/tempo-distributed ``` Add the configuration sections to the `custom.yaml` file. Include this file when you install or upgrade the Helm chart. ### Optional: Configure an ingress An ingress lets you externally access a Kubernetes cluster. Replace `<ingress-host>` with a suitable hostname that DNS can resolve to the external IP address of the Kubernetes cluster. For more information, refer to [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/). {{< admonition type="note" >}} If you are using a Linux system and it's not possible for you set up local DNS resolution, use the `--add-host=<ingress-host>:<kubernetes-cluster-external-address>` command-line flag to define the `<ingress-host>` local address for the Docker commands in the examples that follow. {{< /admonition >}} 1. Open your `custom.yaml` or create a YAML file of Helm values called `custom.yaml`. 1. Add the following configuration to the file: ```yaml gateway: enabled: true ingress: enabled: true ingressClassName: nginx hosts: - host: <ingress-host> paths: - path: / pathType: Prefix tls: {} # empty, disabled. ``` 1. Save the changes. ### Optional: Configure TLS with Helm Tempo can be configured to communicate between the components using Transport Layer Security, or TLS. To configure TLS with the Helm chart, you must have a TLS key-pair and CA certificate stored in a Kubernetes secret. For instructions, refer to [Configure TLS with Helm](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/network/tls/). ### Optional: Use global or per-tenant overrides The `tempo-distributed` Helm chart provides a module for users to set global or per-tenant override settings: - Global overrides come under the `overrides` property, which pertain to the standard overrides - Per-tenant overrides come under the `per_tenant_overrides` property, and allow specific tenants to alter configuration associated with them as per tenant-specific runtime overrides. The Helm chart generates a `/runtime/overrides.yaml` configuration file for all per-tenant configuration. These overrides correlate to the standard (global) and tenant-specific (`per_tenant_overide_config`)overrides in Tempo configuration. For more information about overrides, refer to the [Overrides configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#overrides) documentation. The following example configuration sets some global configuration options, as well as a set of options for a specific tenant: ```yaml overrides: defaults: ingestion: rate_limit_bytes: 5 * 1000 * 1000 burst_size_bytes: 5 * 1000 * 1000 max_traces_per_user: 1000 global: max_bytes_per_trace: 10 * 1000 * 1000 metrics_generator: processors: ["service-graphs", "span-metrics"] per_tenant_overrides: "1234": ingestion: rate_limit_bytes: 2 * 1000 * 1000 burst_size_bytes: 2 * 1000 * 1000 max_traces_per_user: 400 global: max_bytes_per_trace: 5 * 1000 * 1000 ``` This configuration: - Enables the Span Metrics and Service Graph metrics-generator processors for all tenants - An ingestion rate and burst size limit of 5MB/s, a maximum trace size of 10MB and a maximum of 1000 live traces per tenant for all tenants - Overrides the '1234' tenant with a rate and burst size limit of 2MB/s, a maximum trace size of 5MB and a maximum of 400 live traces per tenant {{< admonition type="note" >}} Runtime configurations should include all options for a specific tenant. {{< /admonition >}} {{< admonition type="note" >}} The `local-blocks` processor was removed in Tempo 3.0. Block building is handled by the block-builder component, so you can no longer enable `local-blocks` through `metrics_generator.processors`. {{< /admonition >}} ## Install Tempo using the Helm chart Use the following command to install Tempo using the configuration options you've specified in the `custom.yaml` file: ```bash helm -n tempo-test install tempo grafana-community/tempo-distributed -f custom.yaml ``` {{< admonition type="note" >}} The output of the command contains the write and read URLs necessary for the following steps. {{< /admonition >}} If the installation is successful, the output should be similar to this: {{< collapse title="Installation block example" >}} ```bash > helm -n tempo-test install tempo grafana-community/tempo-distributed -f custom.yaml NAME: tempo LAST DEPLOYED: Fri May 31 15:02:08 2026 NAMESPACE: tempo-test STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: *********************************************************************** Welcome to Grafana Tempo Chart version: 3.0.5 Tempo version: 3.0.2 *********************************************************************** Installed components: * distributor * querier * query-frontend * backend-scheduler * backend-worker * block-builder * live-store * memcached ``` {{< /collapse >}} {{< admonition type="note" >}} If you update your `values.yaml` or `custom.yaml`, run the same helm install command and replace `install` with `upgrade`. {{< /admonition >}} Check the statuses of the Tempo pods: ```bash kubectl -n tempo-test get pods ``` The results look similar to this: ```bash NAME READY STATUS RESTARTS AGE tempo-backend-scheduler-0 1/1 Running 0 22h tempo-backend-worker-0 1/1 Running 0 22h tempo-block-builder-0 1/1 Running 0 22h tempo-block-builder-1 1/1 Running 0 22h tempo-block-builder-2 1/1 Running 0 22h tempo-distributor-bbf4889db-v8l8r 1/1 Running 0 22h tempo-live-store-0 1/1 Running 0 22h tempo-live-store-1 1/1 Running 0 22h tempo-live-store-2 1/1 Running 0 22h tempo-memcached-0 1/1 Running 0 8d tempo-querier-777c8dcf54-fqz45 1/1 Running 0 22h tempo-query-frontend-7f7f686d55-xsnq5 1/1 Running 0 22h ``` Wait until all of the pods have a status of Running or Completed, which might take a few minutes. ## Test your installation The next step is to test your Tempo installation by sending trace data to Grafana. You can use the [Set up a test application for a Tempo cluster](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/test/set-up-test-app/) document for step-by-step instructions. If you already have Grafana available, you can add a Tempo data source using the URL fitting to your environment, for example: `http://tempo-query-frontend.tempo-test.svc.cluster.local:3200` {{< admonition type="note" >}} If you enable the gateway (`gateway.enabled: true`), point the data source at the gateway service instead of the query-frontend. {{< /admonition >}} ## Set up metamonitoring Metamonitoring provides observability for your Tempo deployment by collecting metrics and logs from the Tempo components themselves. This helps you monitor the health and performance of your tracing infrastructure. Setting up metamonitoring for Tempo uses the `k8s-monitoring` Helm chart. For more information about this Helm chart, refer to [k8s-monitoring README](https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/README.md). ### Configure metamonitoring To configure metamonitoring, you need to create a `metamonitoring-values.yaml` file and use the Kubernetes Monitoring Helm chart. 1. Create a `metamonitoring-values.yaml` file for the Kubernetes Monitoring Helm chart configuration. Replace the following values with your monitoring backend details: - `tempo`: A descriptive name for your cluster and namespace - `<url>`: Your Prometheus and Loki endpoint URLs - `<username>`: Your username/instance ID - `<password>`: Your password/API key ```yaml cluster: name: tempo # Name of the cluster, this will populate the cluster label integrations: tempo: instances: - name: "tempo" # This is the name for the instance label that will be reported. namespaces: - tempo # This is the namespace that will be searched for tempo instances, change this accordingly metrics: enabled: true portName: prom-metrics logs: enabled: true labelSelectors: app.kubernetes.io/name: tempo alloy: name: "alloy-tempo" destinations: - name: "tempo-metrics" type: prometheus url: "<url>" # Enter Prometheus URL auth: type: basic username: "<username>" # Enter username password: "<password>" # Enter password - name: "tempo-logs" type: loki url: "<url>" # Enter Loki URL auth: type: basic username: "<username>" # Enter username password: "<password>" # Enter password alloy-metrics: enabled: true podLogs: enabled: true gatherMethod: kubernetesApi namespaces: [tempo] # Set to namespace collector: alloy-singleton alloy-singleton: enabled: true alloy-metrics: enabled: true # This will send Grafana Alloy metrics to ensure the monitoring is working properly. ``` 1. Install the k8s-monitoring Helm chart: ```bash helm install k8s-monitoring grafana/k8s-monitoring \ --namespace monitoring \ --create-namespace \ -f metamonitoring-values.yaml ``` 1. Verify the installation: ```bash kubectl -n monitoring get pods ``` You should see pods for the k8s-monitoring components running. ### Verify metamonitoring in Grafana Verify that metamonitoring is working correctly by checking metrics and logs in Grafana. 1. Navigate to your Grafana instance (Grafana Cloud or self-managed). 1. Check that metrics are being collected: - Go to **Explore** > **Prometheus**. - Query for Tempo metrics like `tempo_build_info` or `tempo_distributor_spans_received_total`. 1. Check that logs are being collected: - Go to **Explore** > **Loki** - Filter logs by your cluster name and look for Tempo component logs 1. Set up Tempo monitoring dashboards: - For pre-built dashboards and alerts, refer to the [Tempo mixin documentation](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) Your Tempo deployment now includes comprehensive metamonitoring, giving you visibility into the health and performance of your tracing infrastructure. --- ### Sources Helm Charts Tempo Distributed Release Notes Index (`docs/sources/helm-charts/tempo-distributed/release-notes/_index.md`) --- description: Release notes for tempo-distributed Helm chart menuTitle: Release notes title: Grafana Tempo Helm chart release notes weight: 20 keywords: - Helm chart - Grafana Tempo --- # Grafana Tempo Helm chart release notes The release notes for the `tempo-distributed` Helm chart reside in the [`tempo-distributed` README file](https://github.com/grafana-community/helm-charts/blob/main/charts/tempo-distributed/README.md). --- ### Sources Tempo Configuration Hosted Storage Index (`docs/sources/tempo/configuration/hosted-storage/_index.md`) --- title: Hosted storage menuTitle: Hosted storage description: Learn about available hosted storage options and how to configure them. weight: 400 aliases: - /docs/tempo/latest/configuration/ --- # Hosted storage Tempo provides additional hosted storage configuration options discussed on the pages below. These options relate to providers such as Google Cloud, AWS S3, and Azure. For additional details about storage configuration, refer to [Storage configuration](../#storage). {{< section withDescriptions="true">}} --- ### Sources Tempo Configuration Network Index (`docs/sources/tempo/configuration/network/_index.md`) --- title: Network options menuTitle: Network options description: Learn about available network options and how to configure them. weight: 500 --- # Network options Tempo provides additional network configuration options discussed on the pages below. {{< section withDescriptions="true" >}} --- ### Sources Tempo Metrics From Traces Metrics Generator Index (`docs/sources/tempo/metrics-from-traces/metrics-generator/_index.md`) --- aliases: - ../operations/server_side_metrics # https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/server_side_metrics/ - ../metrics-generator/ # /docs/tempo/next/metrics-generator/ title: Metrics-generator description: Metrics-generator is an optional Tempo component that derives metrics from ingested traces. weight: 300 --- # Metrics-generator Metrics-generator is an optional Tempo component that derives metrics from ingested traces. The metrics-generator consumes trace data from Kafka and writes metrics to a Prometheus data source using the Prometheus remote-write protocol. If you sample traces before they reach Tempo, where you generate metrics depends on how you sample. Refer to [Choose where to generate metrics from traces](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/where-to-generate-metrics/). ## Architecture Metrics-generator consumes trace data from Kafka to generate metrics from traces. The metrics-generator internally runs a set of **processors**. Each processor ingests spans and produces metrics. Every processor derives different metrics. Currently, the following processors are available: - Service graphs - Span metrics - Host info {{< admonition type="note" >}} Tempo 3.0 removed the `local-blocks` processor. Remove any `local-blocks` entries from your `metrics_generator.processors` configuration. The live-store component handles TraceQL metrics queries on recent data. {{< /admonition >}} Instrumented applications send traces to the distributor, which writes them to Kafka. The metrics-generator consumes trace data from Kafka and runs it through its configured processors (span metrics, service graphs, and host info). Each processor derives a different set of metrics, which the metrics-generator then remote-writes to a Prometheus-compatible backend such as Prometheus or Grafana Mimir. <p align="center"><img src="tempo-metrics-gen-overview.svg" alt="Service metrics architecture"></p> ### Service graphs Service graphs are the representations of the relationships between services within a distributed system. This service graphs processor builds a map of services by analyzing traces, with the objective to find _edges_. Edges are spans with a parent-child relationship, that represent a jump (for example, a request) between two services. The processor records the request count and duration as metrics and uses them to represent the graph. To learn more about this processor, refer to the [service graph](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/) documentation. ### Span metrics The span metrics processor derives RED (Request, Error, and Duration) metrics from spans. The span metrics processor computes the total count and the duration of spans for every unique combination of dimensions. Dimensions can be the service name, the operation, the span kind, the status code and any tag or attribute present in the span. The more dimensions you enable, the higher the cardinality of the generated metrics. To learn more about this processor, refer to the [span metrics](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics/) documentation. ### Host info The host info processor emits a `traces_host_info` gauge metric derived from resource attributes on incoming spans. It identifies hosts sending traces using configurable identifiers (by default, `k8s.node.name` and `host.id`) and produces labels `grafana_host_id` and `host_source`. This processor is useful for correlating trace data with host-level infrastructure metrics. To learn more about the configuration, refer to the [Metrics-generator](/docs/tempo/<TEMPO_VERSION>/configuration/#metrics-generator) section of the Tempo Configuration documentation. ## Remote writing metrics The metrics-generator runs a Prometheus Agent that periodically sends metrics to a `remote_write` endpoint. The `remote_write` endpoint is configurable and can be any [Prometheus-compatible endpoint](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write). To learn more about the endpoint configuration, refer to the [Metrics-generator](http://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#metrics-generator) section of the Tempo Configuration documentation. Use `metrics_generator.registry.collection_interval` to control the writing interval. When you enable multi-tenancy, the metrics-generator forwards the `X-Scope-OrgID` header of the original request to the `remote_write` endpoint. To disable this behavior, set `remote_write_add_org_id_header` to false. ## Native histograms [Native histograms](https://grafana.com/docs/mimir/<MIMIR_VERSION>/visualize/native-histograms/) are a data type in Prometheus that can produce, store, and query high-resolution histograms of observations. It usually offers higher resolution and more straightforward instrumentation than classic histograms. The metrics-generator supports the ability to produce native histograms for high-resolution data. Users must [update the receiving endpoint](https://grafana.com/docs/mimir/<MIMIR_VERSION>/configure/configure-native-histograms-ingestion/) to ingest native histograms, and [update histogram queries](https://grafana.com/docs/mimir/<MIMIR_VERSION>/visualize/native-histograms/) in their dashboards. To learn more about the configuration, refer to the [Metrics-generator](/docs/tempo/<TEMPO_VERSION>/configuration/#metrics-generator) section of the Tempo Configuration documentation. ## Use metrics-generator in Grafana Cloud If you want to enable metrics-generator for your Grafana Cloud account, refer to the [Metrics-generator in Grafana Cloud](https://grafana.com/docs/grafana-cloud/send-data/traces/metrics-generator/) documentation. Enabling metrics generation and remote writing them to Grafana Cloud Metrics produces extra active series that could impact your billing. For more information on billing, refer to [Understand your invoice](/docs/grafana-cloud/cost-management-and-billing/understand-your-invoice/). ## Partition handoff during rollouts The metrics-generator uses Kafka static membership (`instance_id`) so that a pod restarting with the same identity can rejoin the consumer group immediately (without waiting for the session timeout) and reclaim its previously held partitions. This is the right behaviour for **StatefulSet** deployments, where pod names are stable across restarts (for example, `metrics-generator-0`). For **Deployment** deployments, pod names change on every restart (for example, `metrics-generator-7f9d4b6c5-xk2pj`). Because the new pod has a different name, it registers as a new static member, and the old member slot holds its partitions until the session timeout expires (typically several minutes). During that window the partitions are idle and no metrics are generated from the corresponding trace data. To avoid this delay, set `leave_consumer_group_on_shutdown: true` on the metrics-generator. When enabled, the shutting-down pod explicitly sends a `LeaveGroup` request to the Kafka coordinator before closing. The coordinator can then reassign the partitions immediately to the new pod, rather than waiting for the session timeout. ```yaml metrics_generator: leave_consumer_group_on_shutdown: true # recommended for Deployment rollouts ``` Set `leave_consumer_group_on_shutdown: false` (the default) for StatefulSet deployments. With a stable `instance_id`, sending `LeaveGroup` on shutdown would cause an unnecessary leave-and-rejoin rebalance on every restart, briefly interrupting metrics generation for all tenants sharing the same consumer group. {{< admonition type="note" >}} `leave_consumer_group_on_shutdown` requires Kafka 2.4 or later, which introduced the per-member `InstanceID` field in the `LeaveGroup` request (KIP-345). {{< /admonition >}} ## Multitenancy Tempo supports multitenancy in the metrics-generator through the use of environment variables and per-tenant overrides. Refer to the [Multitenant Support for Metrics-Generator](multitenancy/) documentation for more information. --- ### Sources Tempo Metrics From Traces Metrics Queries Index (`docs/sources/tempo/metrics-from-traces/metrics-queries/_index.md`) --- title: TraceQL metrics menuTitle: TraceQL metrics description: Learn about TraceQL metrics and queries weight: 200 keywords: - metrics query - TraceQL metrics aliases: - ../traceql/metrics-queries/ # /docs/tempo/next/traceql/metrics-queries/ refs: solve-problems-mq: - pattern: /docs/tempo/ destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/solutions-with-traces/solve-problems-metrics-queries/ - pattern: /docs/enterprise-traces/ destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/traceql/metrics-queries/solve-problems-metrics-queries/ mq-functions: - pattern: /docs/tempo/ destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/functions/ - pattern: /docs/enterprise-traces/ destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/traceql/metrics-queries/functions/ --- # TraceQL metrics TraceQL metrics is a feature in Grafana Tempo that creates metrics from traces. Metric queries extend trace queries by applying a function to trace query results. This powerful feature allows for ad hoc aggregation of any existing TraceQL query by any dimension available in your traces, much in the same way that LogQL metric queries create metrics from logs. Traces are a unique observability signal that contain causal relationships between the components in your system. TraceQL metrics can help answer questions like this: - How many database calls across all systems are downstream of your application? - What services beneath a given endpoint are failing? - What services beneath an endpoint are slow? TraceQL metrics can help you answer these questions by parsing your traces in aggregate. TraceQL metrics are powered by the [TraceQL metrics API](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#traceql-metrics). ## RED metrics, TraceQL, and PromQL RED is an acronym for three types of metrics: - Rate, the number of requests per second - Errors, the number of those requests that are failing - Duration, the amount of time those requests take For more information about the RED method, refer to [The RED Method: how to instrument your services](/blog/2018/08/02/the-red-method-how-to-instrument-your-services/). You can write TraceQL metrics queries to compute rate, errors, and durations over different groups of spans. For more information on how to use TraceQL metrics to investigate issues, refer to [Solve problems with metrics queries](ref:solve-problems-mq). ## Enable and use TraceQL metrics To use TraceQL metrics, you need to enable them on your Tempo database. Refer to [Configure TraceQL metrics](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/traceql-metrics/) for more information. From there, you can either query the TraceQL metrics API directly (for example, with `curl`) or using Grafana (recommended). To run TraceQL metrics queries in Grafana, you need Grafana Cloud or Grafana 10.4 or later. No extra configuration is needed. Use a Tempo data source that points to a Tempo database with TraceQL metrics enabled. Refer to [Solve problems using metrics queries](ref:solve-problems-mq) for some real-world examples. ### Functions TraceQL metrics queries currently include the following functions for aggregating over groups of spans: `rate`, `count_over_time`, `sum_over_time`, `max_over_time`, `min_over_time`, `avg_over_time`, `quantile_over_time`, `histogram_over_time`, and `compare`. These functions can be added as an operator at the end of any TraceQL query. The `topk` and `bottomk` functions and comparison operators (`>`, `>=`, `<`, `<=`, `=`, `!=`) are supported on TraceQL metrics results. You can also combine metrics queries with arithmetic operators (`+`, `-`, `*`, `/`) to compute derived metrics like error rate ratios. Each sub-query must be wrapped in parentheses: ```traceql 100 * ({status=error} | rate()) / ({} | rate()) ``` For detailed information and example queries for each function, refer to [TraceQL metrics functions](ref:mq-functions). ### Exemplars Exemplars are a powerful feature of TraceQL metrics. They allow you to see an exact trace that contributed to a given metric value. This is particularly useful when you want to understand why a given metric is high or low. Exemplars are available in TraceQL metrics for all range queries. To get exemplars, you need to configure it in the query-frontend with the parameter `query_frontend.metrics.max_exemplars`, or pass a query hint in your query. Example: ``` { span:name = "GET /:endpoint" } | quantile_over_time(duration, .99) by (span.http.target) with (exemplars=true) ``` ### Faster read path In vParquet5, TraceQL metrics queries use a span-only fetch layer by default, which significantly improves performance for most metrics queries. This optimized read path processes individual spans instead of full traces, reducing latency and memory usage. It applies to metrics queries that don't require knowledge of the full trace structure. Queries using structural operators like `>>`, `<<`, `~`, `!>>`, `!<<`, or `!~` still use the standard fetch layer. You can opt out of the faster read path using a query hint or a per-tenant override. #### Opt out with a query hint Add the `spanonly_fetch=false` hint to your query to disable the faster read path for that query. This hint requires [`unsafe_query_hints`](/docs/tempo/<TEMPO_VERSION>/configuration/#overrides) to be enabled for the tenant. ``` { resource.service.name = "frontend" } | rate() by (status) with (spanonly_fetch=false) ``` #### Opt out with a per-tenant override Operators can disable the faster read path by default for a tenant using the `metrics_spanonly_fetch` override. When set to `false`, all eligible metrics queries for that tenant use the standard fetch layer instead. This override doesn't require `unsafe_query_hints`. ```yaml overrides: 'tenant-id': read: metrics_spanonly_fetch: false ``` When `unsafe_query_hints` is also enabled for the tenant, the `spanonly_fetch` query hint takes precedence over the per-tenant override. You can set `spanonly_fetch=false` to opt out, or `spanonly_fetch=true` to opt back in even when the override disables it for the tenant. --- ### Sources Tempo Metrics From Traces Service Graphs Index (`docs/sources/tempo/metrics-from-traces/service_graphs/_index.md`) --- aliases: - ../server_side_metrics/service_graphs/ # /docs/tempo/next/server_side_metrics/service_graphs/ - ../metrics-generator/service_graphs/ # /docs/tempo/next/metrics-generator/service_graphs/ title: Service graphs description: Service graphs help you understand the structure of a distributed system and the connections and dependencies between its components. weight: 500 --- # Service graphs A service graph is a visual representation of the interrelationships between various services. Service graphs help you to understand the structure of a distributed system, and the connections and dependencies between its components: - **Infer the topology of a distributed system.** As distributed systems grow, they become more complex. Service graphs help you to understand the structure of the system. - **Provide a high-level overview of the health of your system.** Service graphs display error rates, latencies, as well as other relevant data. - **Provide an historic view of a system’s topology.** Distributed systems change very frequently, and service graphs offer a way of seeing how these systems have evolved over time. Service graphs can be generated from metrics created by the [metrics-generator](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/metrics-generator/) or Grafana Alloy. Refer to [Enable service graphs](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/enable-service-graphs/) for more information on how to enable service graphs in Tempo. To view service graphs in Grafana, refer to [Service graph view](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/service-graph-view/). ## How they work The metrics-generator and Grafana Alloy both process traces and generate service graphs in the form of Prometheus metrics. Service graphs work by inspecting traces and looking for spans with parent-children relationship that represent a request. The processor uses the [OpenTelemetry semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/trace.md) to detect a myriad of requests. It supports the following requests: - A direct request between two services where the outgoing and the incoming span must have [`span.kind`](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#spankind), `client`, and `server`, respectively. - A request across a messaging system where the outgoing and the incoming span must have `span.kind`, `producer`, and `consumer` respectively. - A database request; in this case the processor looks for spans containing attributes `span.kind`=`client` as well as one of `db.namespace`, `db.name`, `db.system`, or `db.system.name`. You can customize which attributes identify a database span using the `database_name_attributes` [configuration option](/docs/tempo/<TEMPO_VERSION>/configuration/#metrics-generator). See below for how the name of the node is determined for a database request. The processor keeps every span that can form a request pair in an in-memory store until the corresponding pair span arrives or the maximum waiting time passes. When either condition occurs, the processor records the request and removes it from the local store. Each emitted metrics series have the `client` and `server` label corresponding with the service doing the request and the service receiving the request. ``` traces_service_graph_request_total{client="app", server="db", connection_type="database"} 20 ``` ### Virtual nodes Virtual nodes are nodes that form part of the lifecycle of a trace, but spans for them aren't collected because they're outside the user's reach or aren't instrumented. For example, you might not collect spans for an external service for payment processing that's outside user interaction. The processor detects virtual nodes in two ways: - **Uninstrumented client (missing client span):** The root span has `span.kind` set to `server` or `consumer`, with no matching client span. This indicates that the request or message was initiated by an external system that isn't instrumented, like a scheduler, a frontend application, or an engineer using `curl`. - In the Tempo metrics-generator, the processor checks the configured `peer_attributes` on the server span first. If it finds a matching attribute, it uses that value as the client node name. Otherwise, the client node name defaults to `user`. - In Grafana Alloy and the OpenTelemetry Collector `servicegraph` connector, the connector doesn't evaluate peer attributes for this case. The client node name always defaults to `user` and you can't override it. An [upstream feature request](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/45397) exists to add this capability. - **Uninstrumented server (missing server span):** A `client` span doesn't have its matching `server` span, but has a peer attribute present. In this case, the client called an external service that doesn't send spans. The processor uses the peer attribute value as the virtual server node name. - The default peer attributes are `peer.service`, `db.name`, `db.system`, and `db.system.name`. - The processor searches the attributes in order and uses the first match as the virtual node name. The processor identifies a database node when the span has at least one `db.namespace`, `db.name`, `db.system`, or `db.system.name` attribute. The processor determines the database node name using the following span attributes in order of precedence: `peer.service`, `server.address`, `network.peer.address:network.peer.port`, and finally the first attribute from `database_name_attributes` that the span carries. ### Enabling specific metrics (subprocessors) Instead of enabling all service graph metrics, you can enable individual metric categories using subprocessors in the overrides configuration: - `service-graphs-request` — Enables only the `traces_service_graph_request_total` and `traces_service_graph_request_failed_total` counters - `service-graphs-latency` — Enables only the `traces_service_graph_request_server_seconds` and `traces_service_graph_request_client_seconds` histograms. The `traces_service_graph_request_messaging_system_seconds` histogram additionally requires `enable_messaging_system_latency_histogram: true` in the metrics-generator config. - `service-graphs-connection-info` — Enables only the `traces_service_graph_connection_info` gauge The bare `service-graphs` name enables request and latency metrics. The `service-graphs-connection-info` subprocessor is off by default and must be listed explicitly. Listing it alongside the bare name enables the subprocessor additively without disabling RED. Listing `service-graphs-request` or `service-graphs-latency` alongside the bare name is redundant and silently dropped. Example overrides configuration: ```yaml overrides: defaults: metrics_generator: processors: - service-graphs - service-graphs-connection-info ``` ### Connection information metric The `traces_service_graph_connection_info` metric is a presence-only gauge held at `1` while an edge between two services is being observed. The metric is intended for topology discovery rather than rate calculations. A single observed span keeps the gauge present, so service-to-service relationships stay visible on low-traffic endpoints even when `rate(traces_service_graph_request_total[...])` is noisy or zero under aggressive head sampling. Example query with a long range window: ```promql last_over_time(traces_service_graph_connection_info[1h]) > 0 ``` ### Metrics The following metrics are exported: | Metric | Type | Labels | Description | | ------------------------------------------------------- | --------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `traces_service_graph_request_total` | Counter | client, server, connection_type | Total count of requests between two nodes | | `traces_service_graph_request_failed_total` | Counter | client, server, connection_type | Total count of failed requests between two nodes | | `traces_service_graph_request_server_seconds` | Histogram | client, server, connection_type | Time for a request between two nodes as seen from the server | | `traces_service_graph_request_client_seconds` | Histogram | client, server, connection_type | Time for a request between two nodes as seen from the client | | `traces_service_graph_request_messaging_system_seconds` | Histogram | client, server, connection_type | (Off by default) Time between publisher and consumer for services communicating through a messaging system | | `traces_service_graph_connection_info` | Gauge | client, server, connection_type | (Off by default) Presence signal for service-to-service edges (value 1 per active edge) | | `traces_service_graph_unpaired_spans_total` | Counter | client, server, connection_type | Total count of unpaired spans | | `traces_service_graph_dropped_spans_total` | Counter | client, server, connection_type | Total count of dropped spans | The processor measures duration from both the client and server sides. Possible values for `connection_type`: unset, `virtual_node`, `messaging_system`, or `database`. You can include additional labels using the `dimensions` configuration option or the `enable_virtual_node_label` option. Duplicate dimensions are allowed after Prometheus label name conversion. This supports environments where different instrumentation libraries use different attribute naming conventions, for example, `deployment.environment` and `deployment_environment`. When a collision occurs, the last configured value wins. Since the service graph processor has to process both sides of an edge, it needs to process all spans of a trace to function properly. If spans of a trace spread across multiple instances, the processor can't pair them reliably. #### Activate `enable_virtual_node_label` Activating this feature adds the following label and corresponding values: | Label | Possible Values | Description | | -------------- | --------------------------- | -------------------------------------------- | | `virtual_node` | `unset`, `client`, `server` | Explicitly indicates the uninstrumented side | ## Configuration options The service graphs processor has several configuration options beyond `dimensions` and `enable_virtual_node_label`. For the full YAML schema and defaults, refer to the [configuration reference](/docs/tempo/<TEMPO_VERSION>/configuration/#metrics-generator). You don't need to set any of them to get started; use them when you want to adjust sampling, labels, database detection, or which spans are included. ### Span multiplier When traces are sampled, the raw request counts produced by the service graph processor underrepresent actual traffic. The `span_multiplier_key` option specifies a span or resource attribute that contains the sampling ratio. The processor computes the inverse of this value to scale the metrics accordingly. For example, if a span has attribute `X-SampleRatio=0.1` (10% sampling), setting `span_multiplier_key: "X-SampleRatio"` causes each sampled span to count as 10 requests. The `enable_tracestate_span_multiplier` option provides an alternative approach that extracts the multiplier from the W3C tracestate header using the [OpenTelemetry probability sampling threshold](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling/) (`ot=th:<hex>`). When enabled, the tracestate threshold takes priority over `span_multiplier_key`. ### Client and server dimension prefixes When you add extra `dimensions`, the same attribute can appear on both the client and server sides of an edge. By default, the processor writes one label per dimension, and if both sides carry the attribute the value is undetermined. Set `enable_client_server_prefix: true` to prefix those extra dimensions with `client_` and `server_`. The processor then emits two labels per additional dimension, for example `client_deployment_environment` and `server_deployment_environment`. That doubles the label count for each configured dimension, which increases cardinality. The default is `false`. ```yaml metrics_generator: processor: service_graphs: dimensions: - deployment.environment enable_client_server_prefix: true ``` ### Database name attributes The `database_name_attributes` option controls which span attributes the processor uses to identify a span as a database request. The defaults are `db.namespace`, `db.name`, `db.system`, and `db.system.name`. The processor searches the list in order and uses the first attribute the span carries. `db.namespace` and `db.name` identify the database itself, such as `mydb`, so they take precedence over `db.system` and `db.system.name`, which identify only the database product, such as `postgresql`. Within that second pair, `db.system.name` (which replaced `db.system` in semantic conventions v1.30.0) is listed last, so instrumentation that still emits `db.system` produces the same node name as before. You can override this list to match your instrumentation if it uses non-standard attribute names. ### Filter policies The `filter_policies` option lets you include or exclude spans from service graph generation. It uses the same policy format as [span metrics](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics/span-metrics-metrics-generator/#filtering). The service graph processor only evaluates spans that can form edges: `SPAN_KIND_CLIENT`, `SPAN_KIND_SERVER`, `SPAN_KIND_PRODUCER`, and `SPAN_KIND_CONSUMER`. When one side of an edge is filtered out, Tempo keeps a best-effort marker for the dropped side and drops the matching side if it arrives later or is already buffered. This reduces skewed edges and unwanted virtual nodes. The marker cache uses `wait` as its TTL and `max_items` as its maximum size. Policy behavior is: - `include`: all include policies must match. - `include_any`: any include_any policy can match. If only include_any policies are configured, non-matching spans are excluded. - `exclude`: matching spans are rejected, even if they match an include rule. Use scoped keys for resource and span attributes, such as `resource.service.name` or `span.http.route`. The supported intrinsic keys are `name`, `status`, and `kind`. Supported attribute value types are `bool`, `double`, `int`, and `string`. This example excludes service graph spans from `shop-backend`: ```yaml metrics_generator: processor: service_graphs: filter_policies: - exclude: match_type: strict attributes: - key: resource.service.name value: shop-backend ``` Monitor service graph filtering with: - `tempo_metrics_generator_spans_discarded_total{reason="service_graphs_filtered", processor="service-graphs"}` - `tempo_metrics_generator_processor_service_graphs_dropped_edges_total` - `tempo_metrics_generator_processor_service_graphs_dropped_span_side_cache_overflow_total` ## Next steps - [Enable service graphs](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/enable-service-graphs/) using the metrics-generator or Grafana Alloy. - [View service graphs](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/service-graph-view/) in Grafana. - [Create custom service graphs](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/custom-service-graphs/) with the Node graph visualization. - [Analyze service graph data](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/metrics-queries/) with PromQL queries. - Generate [span metrics](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics/) for RED metrics alongside service graphs. --- ### Sources Tempo Metrics From Traces Span Metrics Index (`docs/sources/tempo/metrics-from-traces/span-metrics/_index.md`) --- title: Span metrics description: Learn about span metrics in Grafana Tempo. weight: 400 --- # Span metrics Span metrics are generated from traces and can be used to create service graphs. You can create span metrics in Tempo's metrics-generator, Grafana Alloy, or the OpenTelemetry Collector. If you sample traces before they reach Tempo, choose where to generate those metrics first. Refer to [Choose where to generate metrics from traces](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/where-to-generate-metrics/). The pages in this section cover Tempo and Alloy settings. For OpenTelemetry Collector settings, including the pipeline diagram, refer to [Generate metrics from spans](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/otel-collector/#generate-metrics-from-spans). {{< section withDescriptions="true">}} --- ### Sources Tempo Operations Manage Advanced Systems Index (`docs/sources/tempo/operations/manage-advanced-systems/_index.md`) --- title: Manage advanced systems menuTitle: Manage advanced systems description: Learn how to manage and tune complex installations. weight: 700 --- # Manage advanced systems This section provides resources for managing and tuning advanced Tempo configurations. For an overview of Tempo components and how they interact, refer to the [Architecture](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/architecture/) topic. {{< section >}} --- ### Sources Tempo Operations Monitor Index (`docs/sources/tempo/operations/monitor/_index.md`) --- title: Monitor Tempo menuTitle: Monitor Tempo description: Use polling, alerts, and dashboards to monitor Tempo in production. weight: 200 --- # Monitor Tempo Tempo is instrumented to expose metrics, logs, and traces. Furthermore, the Tempo repository has a [mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) that includes a set of dashboards, rules, and alerts. Together, these can be used to monitor Tempo in production. ## Instrumentation Metrics, logs, and traces from Tempo can be collected to observe its services and functions. ### Metrics Tempo is instrumented with [Prometheus metrics](https://prometheus.io/) and emits RED metrics for most services and backends. RED metrics are a standardized format for monitoring microservices, where R stands for requests, E stands for errors, and D stands for duration. The [Tempo mixin](#dashboards) provides several dashboards using these metrics. #### Query frontend inspected bytes Use `tempo_query_frontend_bytes_inspected_total` to monitor how many bytes the query path behind the query frontend inspects. This counter is emitted per `tenant` and `op` (`traces`, `search`, `metadata`, `metrics`). Because cached responses from queriers are excluded, it reflects uncached query I/O. For recent-data queries, use `tempo_live_store_query_inspected_bytes_total` to isolate bytes inspected by live-stores. For PromQL examples and alerting guidance, refer to [Monitor query I/O and span timestamp distance](/docs/tempo/<TEMPO_VERSION>/operations/monitor/query-io-and-timestamp-distance/). ### Logs Tempo emits logs in the `key=value` ([logfmt](https://brandur.org/logfmt)) format. ### Traces You can configure the self-tracing for tempo using OpenTelemetry or Jaeger environment variables. Refer to [Configure Tempo self-tracing](self-tracing/) for the supported config, environment variables, sampling, propagation, and more details. ## Polling Tempo maintains knowledge of the state of the backend by polling it on regular intervals. Several components need this knowledge, including schedulers, workers, queriers, and query-frontends. Refer to [Use polling to monitor the backend status](polling/) for Tempo. ## Dashboards The [Tempo mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin) has ten Grafana dashboards in the `dashboards` folder that you can download and import into your Grafana UI. These dashboards work well when you run Tempo in a Kubernetes (k8s) environment and metrics scraped have the `cluster` and `namespace` labels. ### Tempo Reads dashboard > This is available as `tempo-reads.json`. The Reads dashboard gives information on Requests, Errors, and Duration (RED) on the query path of Tempo. Each query touches the Gateway, Query-Frontend, Queriers, Live-stores, Memcached, and the backend. Use this dashboard to monitor the performance of each of the mentioned components and to decide the number of replicas in each deployment. ### Tempo Writes dashboard > This is available as `tempo-writes.json`. The Writes dashboard gives information on RED on the write/ingest path of Tempo. A write query touches the Gateway, Distributors, and Kafka. The dashboard also shows compaction activity against Memcached and the backend. Use this dashboard to monitor the performance of each of the mentioned components and to decide the number of replicas in each deployment. ### Tempo Resources dashboard > This is available as `tempo-resources.json`. The Resources dashboard provides information on `CPU`, `Container Memory`, and `Go Heap Inuse`. This dashboard is useful for resource provisioning for the different Tempo components. Use this dashboard to see if any components are running close to their assigned limits. ### Tempo Operational dashboard > This is available as `tempo-operational.json`. The Tempo Operational dashboard deserves special mention because it is probably a stack of dashboard anti-patterns. It's big and complex, doesn't use `jsonnet`, and displays far too many metrics in one place. For just getting started, the RED dashboards are great places to learn how to monitor Tempo in an opaque way. This dashboard is included in the Tempo repository for two reasons: - The dashboard provides a stack of metrics for other operators to consider monitoring while running Tempo. - We want the dashboard in our internal infrastructure and we vendor the `tempo-mixin` to do this. ### Tempo Backend Work dashboard > This is available as `tempo-backendwork.json`. The Backend Work dashboard monitors blocklist maintenance, compaction jobs, and backend component resources. It tracks blocklist length and poll duration, active and completed compaction jobs, failure and retry rates, and objects written and combined during compaction. It also includes a **Redaction** row with per-tenant **Traces Redacted / h** and **Dry-run Blast Radius / h** panels for [`tempo-cli redact`](/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#redact-traces) jobs. The dashboard also shows CPU and memory usage for the backend-scheduler and backend-worker components. Use this dashboard to monitor compaction health, detect stalled or failing jobs, and right-size backend-scheduler and backend-worker resources. ### Tempo Block Builder dashboard > This is available as `tempo-block-builder.json`. The Block Builder dashboard monitors the Kafka-based ingest path introduced. It tracks Kafka fetch rates and read throughput, flushed blocks per second, and per-partition lag in both records and seconds. Heatmaps show partition section and cycle durations, and resource panels track CPU and memory usage. Use this dashboard to monitor block-builder throughput, detect partition lag, and ensure the ingest pipeline keeps up with incoming trace data. ### Tempo Rollout Progress dashboard > This is available as `tempo-rollout-progress.json`. The Rollout Progress dashboard tracks the health of a Tempo deployment during upgrades and rollouts. It breaks down write and read requests by status code (2xx, 4xx, 5xx), shows 99th percentile latency, and counts unhealthy Pods. A version panel shows the number of pods running each version, and a latency comparison panel shows current latency against the previous 24-hour baseline. Use this dashboard during upgrades to confirm that new versions aren't introducing errors or latency regressions. ### Tempo Tenants dashboard > This is available as `tempo-tenants.json`. The Tenants dashboard provides per-tenant visibility into ingestion, reads, storage, and metrics generation. It shows a limits table alongside distributor bytes and spans per second, live trace counts, query rates for ID lookups and searches, blocklist length, outstanding compactions, and metrics-generator bytes and active series. Use this dashboard in multitenant deployments to identify tenants with high ingestion rates, query volumes, or storage growth. ### Tempo Service Graph dashboard > This is available as `tempo-service-graph.json`. The Service Graph dashboard visualizes service-to-service topology discovered from the `traces_service_graph_connection_info` metric emitted by the metrics-generator's service-graphs connection-info subprocessor. Use this dashboard to see how services in your system call each other and to spot unexpected or missing connections in the traced topology. ### Tempo Livestore dashboard > This is available as `tempo-livestore.json`. The Livestore dashboard monitors the health and performance of the live-store component, which serves recent trace data directly from memory and the write-ahead log (WAL) before it's flushed to the backend. It's organized into a triage cockpit plus rows for read latency and query impact, ingest lag and throughput, backpressure and readiness, the WAL completion path, live trace and tenant pressure, resource usage, and autoscaling, with most rows broken down by live-store zone. Use this dashboard to diagnose read latency, ingest lag, and backpressure issues on the live-store path, and to monitor autoscaling behavior for live-store and its supporting Warpstream agents. ## Rules and alerts The Rules and Alerts are available as [YAML files in the compiled mixin](https://github.com/grafana/tempo/tree/main/operations/tempo-mixin-compiled) on the repository. To set up alerting, download the provided JSON files and configure them for use on your Prometheus monitoring server. Check the [runbook](https://github.com/grafana/tempo/blob/main/operations/tempo-mixin/runbook.md) to understand the various steps that can be taken to fix firing alerts. --- ### Sources Tempo Reference Tempo Architecture About Tempo Architecture Index (`docs/sources/tempo/reference-tempo-architecture/about-tempo-architecture/_index.md`) --- title: About the Tempo architecture menuTitle: About the architecture description: Understand the design philosophy and data flow in Tempo. weight: 100 topicType: concept versionDate: 2026-03-20 --- # About the Tempo architecture Grafana Tempo is a distributed tracing backend designed for high-volume trace ingestion and querying at scale. Tempo 3.0 introduces a new architecture that decouples the write and read paths. In [microservices mode](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/deployment-modes/), a Kafka-compatible message queue serves as a durable intermediary between the distributor and downstream consumers. In [monolithic mode](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/deployment-modes/), the distributor pushes data directly to the live-store in-process without Kafka. ## Design philosophy The Tempo architecture is built around several key principles. Separate components handle writing trace data to storage and serving queries. You can scale writes and reads independently, and a failure in one path doesn't affect the other. In microservices mode, Kafka serves as a durable write-ahead log (WAL) between distributors and downstream consumers. Once Kafka acknowledges a write, the data is safe. This replaces the previous in-process ingestion WAL that lived on local disks. Because Kafka provides durability on the write path, Tempo doesn't need to replicate data across multiple instances. This replication factor of 1 significantly reduces cost and query complexity. In monolithic mode, no Kafka is required. The distributor pushes trace data in-process directly to the live-store and metrics-generator. Live-stores still use a local WAL for quickly available search. Tempo uses Apache Parquet as the default columnar block format, storing trace data in a columnar layout that enables efficient querying of specific attributes without reading entire traces. Refer to [Apache Parquet block format configuration](/docs/tempo/<TEMPO_VERSION>/configuration/parquet/) and [Apache Parquet schema](/docs/tempo/<TEMPO_VERSION>/operations/schema/) to learn more. ## Write path The write path gets trace data from instrumented applications into long-term object storage. How data flows through the write path depends on the [deployment mode](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/deployment-modes/). ### Microservices write path <p align="center"><img src="tempo-write-path.svg" alt="Tempo write path: Application to Distributor to Kafka to Block-builder to Object storage"></p> 1. Distributors receive trace data over OTLP or other supported protocols, validate it against rate limits, shard traces by trace ID, and write records to Kafka partitions. 1. Kafka durably stores the records. The write is acknowledged to the client as soon as Kafka confirms receipt. 1. Block-builders consume records from Kafka, organize spans into blocks in Apache Parquet format, and flush those blocks to object storage. The block-builder operates on a consumption cycle: it reads a batch of records from Kafka, builds blocks from them, flushes the blocks to object storage, and commits the offset back to Kafka. Each cycle produces a clean cut of data. Traces that span multiple cycles have their spans split across blocks, which the query path handles at query time. ### Monolithic write path In monolithic mode, no Kafka or block-builder is involved. The distributor pushes trace data in-process directly to the live-store and metrics-generator. The live-store holds traces in memory, flushes them to a local WAL, cuts them into completed blocks, and flushes those completed blocks to the configured storage backend. ## Read path The read path serves queries by combining recent data from live-stores with historical data from object storage. <p align="center"><img src="tempo-read-path.svg" alt="Tempo read path: Client to Query frontend to Querier, branching to Live-store and Object storage"></p> 1. The query frontend receives a query, shards it into parallel jobs, and distributes them to queriers. 1. Queriers execute jobs by fetching data from two sources: live-stores for recent data, typically the last 30 minutes to 1 hour, and object storage for historical data, using bloom filters and indexes for efficient block lookups. 1. The query frontend merges results from all queriers and returns the response. ## Live-stores and the recent data window Live-stores are the read-path component responsible for serving recent trace data. They hold traces in memory and write them to temporary on-disk blocks, making data available for queries within seconds of ingestion. In microservices mode, live-stores consume from Kafka independently of block-builders. There's a gap between when trace data is written to Kafka and when the block-builder flushes it to object storage. During this window, the only way to query that data is through the live-store. In monolithic mode, live-stores receive trace data directly from the distributor in-process. There is no Kafka consumption or block-builder involved. In microservices mode, live-stores own the partition lifecycle within Tempo. They manage a partition ring that tracks which partitions are active and which live-stores own them. This is separate from Kafka's internal partition management. Refer to the [partition ring](../partition-ring/) documentation for details. ## How the paths connect The write and read paths connect through object storage. In microservices mode, block-builders write blocks there; in monolithic mode, live-stores write blocks there. Queriers read from object storage in both modes. In microservices mode, Kafka also connects the paths. Both block-builders and live-stores consume from the same Kafka partitions, but they track their own consumer offsets independently. Even if a block-builder is down or slow, live-stores continue serving recent data. If a live-store restarts, it replays from Kafka to rebuild its in-memory state. In monolithic mode, the connection is simpler. The distributor pushes data directly to the live-store, which flushes blocks to object storage. The querier reads from both the live-store and object storage within the same process. ## Component summary | Component | Path | Microservices mode | Monolithic mode | |---|---|---|---| | [Distributor](../components/distributor/) | Write | Receives traces, validates limits, writes to Kafka | Receives traces, validates limits, pushes in-process to live-store | | [Kafka](../components/kafka/) | Write | Durable message queue between distributor and consumers | Not used | | [Block-builder](../components/block-builder/) | Write | Consumes from Kafka, builds Parquet blocks, flushes to object storage | Not used | | [Live-store](../components/live-store/) | Read | Consumes from Kafka, serves recent data to queriers | Receives data from distributor, serves recent data to queriers | | [Query frontend](../components/query-frontend/) | Read | Shards queries into jobs, distributes to queriers, merges results | Same | | [Querier](../components/querier/) | Read | Executes query jobs against live-stores and object storage | Same | | [Backend scheduler/worker](../components/compaction/) | Maintenance | Compacts and deduplicates blocks, enforces retention | Same | | [Metrics-generator](../components/metrics-generator/) | Optional | Consumes from Kafka, derives metrics from traces | Receives data from distributor, derives metrics from traces | --- ### Sources Tempo Reference Tempo Architecture Components Index (`docs/sources/tempo/reference-tempo-architecture/components/_index.md`) --- title: Components menuTitle: Components description: Detailed reference for each Tempo component. weight: 200 topicType: introduction versionDate: 2026-03-20 --- # Components Tempo is composed of several components, each responsible for a specific part of the trace lifecycle. All components are compiled into the same binary, and the `-target` parameter controls which component runs. This section documents each component in detail, including its responsibilities, configuration, failure modes, and relevant metrics. {{< section menuTitle="true" >}} --- ### Sources Tempo Release Notes Version 1 Index (`docs/sources/tempo/release-notes/version-1/_index.md`) --- title: Version 1 release notes menuTitle: Version 1 description: Release notes for Grafana Tempo version 1.x weight: 900 --- # Version 1 release notes Release notes for Grafana Tempo version 1.x releases. {{< section >}} --- ### Sources Tempo Release Notes Version 2 Index (`docs/sources/tempo/release-notes/version-2/_index.md`) --- title: Version 2 release notes menuTitle: Version 2 description: Release notes for Grafana Tempo version 2.x weight: 875 --- # Version 2 release notes Release notes for Grafana Tempo version 2.x releases. For information about upgrading to Tempo 2.x, refer to [Upgrade](/docs/tempo/v2.10.x/set-up-for-tracing/setup-tempo/upgrade/) in the Tempo 2.10 documentation. To migrate from Tempo 2.x to 3.0, refer to [Migrate to Tempo 3.0](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/migrate-to-3/). {{< section >}} --- ### Sources Tempo Set Up For Tracing Instrument Send Index (`docs/sources/tempo/set-up-for-tracing/instrument-send/_index.md`) --- title: Instrument for distributed tracing menuTitle: Instrument for tracing description: Client instrumentation is the first building block to a functioning distributed tracing visualization pipeline. aliases: - ../guides/instrumentation/ # /docs/tempo/latest/guides/instrumentation/ - ../getting-started/instrumentation/ # /docs/tempo/latest/getting-started/instrumentation/ weight: 350 --- # Instrument for distributed tracing Client instrumentation is the first building block to a functioning distributed tracing visualization pipeline. Instrumentation handles how traces are generated. To learn more about instrumentation, refer to [About instrumentation](./about-instrumentation/). Instrumentation is the act of modifying the source code of a service to emit span information tied to a common trace ID. Traces themselves are a meta-object, comprised of nothing but spans that hold the same ID. To instrument your application or service, you need to: 1. [Set up instrumentation](./set-up-instrumentation/) to generate traces 1. [Set up a collector](./set-up-collector/) to receive traces from your application --- ### Sources Tempo Set Up For Tracing Setup Tempo Index (`docs/sources/tempo/set-up-for-tracing/setup-tempo/_index.md`) --- title: Set up Tempo menuTitle: Set up Tempo description: Learn how to set up a Tempo server or cluster and visualize data. aliases: - ../setup/ # /docs/tempo/next/setup/ weight: 300 --- # Set up Tempo To set up Tempo, you need to: 1. Plan your deployment 1. Deploy Tempo 1. Test your installation 1. (Optional) Configure Tempo services ## Plan your deployment How you choose to deploy Tempo depends upon your tracing needs. Tempo has two deployment modes: monolithic or microservices. Refer to [Plan your deployment](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/plan/) to determine the best method to deploy Tempo. {{< admonition type="note" >}} Tempo v3.0 microservices mode requires a Kafka-compatible system (such as Apache Kafka, Redpanda, or WarpStream). Monolithic mode (`target: all`) does not require Kafka. For setup guidance, refer to [Configure a Kafka-compatible backend](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/configure-kafka/). For architectural details, refer to the [Deployment modes](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/deployment-modes/) documentation. {{< /admonition >}} ## Deploy Tempo After you have decided the best method, you can deploy and set up Tempo. Grafana Tempo is available as a [pre-compiled binary, OS-specific packaging](https://github.com/grafana/tempo/releases), and [Docker image](https://github.com/grafana/tempo/tree/main/example/docker-compose). Refer to [Deploy Tempo](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/) for instructions for deploying using Kubernetes and deploying locally. ## Test your installation After Tempo is deployed, you can validate Tempo by visualizing traces data: - Using a [test application for a Tempo cluster](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/test/set-up-test-app/) for the Kubernetes with Tanka setup These visualizations test Kubernetes with Tanka and Linux procedures. They don't check optional configuration you have enabled. ## (Optional) Configure Tempo services Explore features by learning about [available features and configurations](/docs/tempo/<TEMPO_VERSION>/configuration/). If you would like to see a simplified, annotated example configuration for Tempo, the [Introduction to MLTP](https://github.com/grafana/intro-to-mltp) example repository contains a [configuration](https://github.com/grafana/intro-to-mltp/blob/main/tempo/tempo.yaml) for a monolithic instance. --- ### Sources Tempo Traceql Query Editor Index (`docs/sources/tempo/traceql/query-editor/_index.md`) --- title: TraceQL queries in Grafana description: Learn how to create TraceQL queries in Grafana using the query editor and search. aliases: - ./construct-query # /docs/tempo/<TEMPO_VERSION>/traceql/construct-query - ./query-editor/traceql-editor # /docs/tempo/<TEMPO_VERSION>/traceql/query-editor/traceql-editor - ./query-editor/traceql-search # /docs/tempo/<TEMPO_VERSION>/traceql/query-editor/traceql-search weight: 500 keywords: - Tempo query language - query editor - TraceQL refs: traceql: - pattern: /docs/tempo/ destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/ - pattern: /docs/enterprise-traces/ destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/traceql/ service-graph-view: - pattern: /docs/tempo/ destination: https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-generator/service-graph-view/ - pattern: /docs/enterprise-traces/ destination: https://grafana.com/docs/enterprise-traces/<ENTERPRISE_TRACES_VERSION>/metrics-generator/service-graph-view/ --- # TraceQL queries in Grafana You can compose TraceQL queries in Grafana and Grafana Cloud using **Explore** and a Tempo data source. The Tempo data source's query editor helps you query and display traces from Tempo in [Explore](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/explore/). The queries use [TraceQL](ref:traceql), the query language designed specifically for tracing. For general documentation on querying data sources in Grafana, refer to [Query and transform data](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/). Query results can stream to the client, which lets you look at traces matching your query before the entire query completes. To use streaming in Grafana, you must have `stream_over_http_enabled: true` enabled in Tempo. For information, refer to [Tempo GRPC API](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#tempo-grpc-api). {{< admonition type="note" >}} You can also use Traces Drilldown to explore your tracing data without writing queries. For more information, refer to the [Traces Drilldown documentation](/docs/grafana/<TEMPO_VERSION>/explore/simplified-exploration/traces/). {{< /admonition >}} ## Choose a query editing mode The query editor has three modes, or **Query types**, that you can use to explore your tracing data. You can use these modes by themselves or in combination to create building blocks to generate custom queries. The three **Query types** are: - The **Search** query builder provides a user interface for building a TraceQL query. - The **TraceQL** query editor lets you write your own TraceQL query with assistance from autocomplete. - The **Service Graph** view displays a visual relationship between services. Refer to the [Service graph view](ref:service-graph-view) documentation for more information. ### Stream query results Query results can stream to the client, which lets you look at traces matching your query before the entire query completes. The GRPC streaming API endpoint in the query frontend allows a client to stream search results from Tempo. The `tempo-cli` also uses this streaming endpoint. For more information, refer to the [Tempo CLI documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#query-api-command). To use streaming in Grafana, you must have `stream_over_http_enabled: true` enabled in Tempo. For information, refer to [Tempo GRPC API](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#tempo-grpc-api). ### Search query builder The Search query builder provides drop-down lists and text fields to help you write a query. The query builder is ideal for people who aren't familiar with or want to learn TraceQL. Refer to the [Search using the TraceQL query builder documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-search/) to learn more about creating queries using convenient drop-down menus. ### TraceQL query editor The TraceQL query editor lets you search by trace ID and write TraceQL queries using autocomplete. Refer to the [TraceQL query editor documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/traceql-editor/) to learn more about constructing queries using a code-editor-like experience. You can also search for a Trace ID by entering a trace ID into the query field. ### Service graph view Grafana’s service graph view uses metrics to display span request rates, error rates, and durations, as well as service graphs. Once the requirements are set up, this pre-configured view is immediately available. Using the service graph view, you can: - Discover spans which are consistently erroring and the rates at which they occur - Get an overview of the overall rate of span calls throughout your services - Determine how long the slowest queries in your service take to complete - Examine all traces that contain spans of particular interest based on rate, error, and duration values (RED signals) For more information about the service graph, refer to [Service graph view](ref:service-graph-view). ## Use TraceQL panels in dashboards To add TraceQL panels to your dashboard, refer to the [Traces panel documentation](/docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/traces/). To learn more about Grafana dashboards, refer to the [Use dashboards documentation](/docs/grafana/<GRAFANA_VERSION>/dashboards/use-dashboards/). --- ### Sources Tempo Troubleshooting Querying Index (`docs/sources/tempo/troubleshooting/querying/_index.md`) --- title: Issues with querying menuTitle: Querying description: Troubleshoot issues related to querying. weight: 300 --- # Issues with querying Learn about issues related to querying. {{< section withDescriptions="true">}} --- ### Sources Tempo Troubleshooting Send Traces Index (`docs/sources/tempo/troubleshooting/send-traces/_index.md`) --- title: Issues with sending traces menuTitle: Sending traces description: Troubleshoot issues related to sending traces. weight: 200 --- # Issues with sending traces Learn about issues related to sending traces. {{< section withDescriptions="true">}} --- ### Sources Tempo Set Up For Tracing Instrument Send Choose Instrumentation Method Index (`docs/sources/tempo/set-up-for-tracing/instrument-send/choose-instrumentation-method/_index.md`) --- title: Choose an instrumentation method description: Learn about the different instrumentation methods for sending traces to Tempo. weight: 300 draft: true --- # Choose an instrumentation method You need to instrument your app to enable it to emit tracing data. This data is then gathered by a collector and sent to Tempo. ## Instrumentation methods comparison You can instrument your code using one or more of the methods described in the table. | Instrumentation method | Description | Benefits | Drawbacks | | ------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------- | | Automatic instrumentation | Applies instrumentation automatically using agents or middleware, without code changes. |<ul><li>Quick setup: Enables tracing without code changes</li><li>Low overhead: Minimal performance impact</li></ul> | <ul><li>Limited customization: May not capture all use cases</li></ul> | | Zero-code instrumentation | Uses eBPF technology to instrument applications without code changes. | <ul><li>Non-intrusive: No code changes needed</li><li>High performance: Low overhead and efficient</li></ul> | <ul><li>Limited visibility: May not capture all behavior</li><li>Complexity: Requires eBPF knowledge</li></ul> | | Manual instrumentation | Involves adding code to create spans and traces, giving full control over collected data. | <ul><li>Full control: Define exactly what data is collected</li><li>Custom spans: Capture specific behavior</li></ul> | <ul><li>Higher effort: Requires code changes and maintenance</li><li>Potential for errors: Can introduce bugs</li></ul> | | Hybrid instrumentation | Combines automatic and manual methods, using automatic for most code and manual for custom tracing logic. | <ul><li>Flexibility: Leverage benefits of both methods</li><ul> | <ul><li> Complexity: May require managing both approaches</li><ul> | ## Next steps After you choose your instrumentation method, refer to [Set up instrumentation](/docs/tempo/<TEMPO_VERSION>/instrument-send/set-up-instrumentation/). --- ### Sources Tempo Set Up For Tracing Instrument Send Set Up Collector Index (`docs/sources/tempo/set-up-for-tracing/instrument-send/set-up-collector/_index.md`) --- title: Set up your collector description: Configure your collector to send traces to Tempo. weight: 550 --- # Set up your collector You can send data from your application using Grafana Alloy or OpenTelemetry Collector (OTel) collectors. [Grafana Alloy](https://grafana.com/docs/alloy/<ALLOY_VERSION>/) is a vendor-neutral distribution of the OpenTelemetry (OTel) Collector. Alloy uniquely combines the very best OSS observability signals in the community. The [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) is a vendor-agnostic way to receive, process, and export telemetry data. The collector is a component that runs alongside your application and periodically gathers tracing data from it. This method is suitable when you want to collect tracing from applications without modifying their source code. Here's how it works: 1. Install and configure the collector on the same machine or container where your application is running. 2. The collector periodically retrieves your application's performance tracing data, regardless of the language or technology stack your application is using. 3. The captured traces are then sent to the Tempo server for storage and analysis. Using a collector provides a hassle-free option, especially when dealing with multiple applications or microservices, allowing you to centralize the tracing process without changing your application's codebase. ## Use Alloy Grafana Labs maintains and supports Alloy, which packages various upstream OpenTelemetry Collector components. Alloy provides stability, support, and integration with Grafana Labs products. Refer to the [Alloy documentation](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/grafana-alloy/) for information about Alloy and it's tracing capabilities. Refer to [Collect and forward data with Alloy](https://grafana.com/docs/alloy/<ALLOY_VERSION>/collect/) for examples of collecting data. ## Use the OpenTelemetry Collector The OpenTelemetry project maintainers and the Cloud Native Computing Foundation maintain the upstream OpenTelemetry Collector. This is a community-supported project. Refer to the [OpenTelemetry Collector documentation](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/otel-collector/) for Tempo-specific configuration guidance, including examples for sending traces to Tempo using OTLP. Refer to the [Install the Collector documentation](https://opentelemetry.io/docs/collector/installation/) for instructions on installation. Refer to the [Grafana OpenTelemetry Collector documentation](https://grafana.com/docs/opentelemetry/collector/opentelemetry-collector/) to use the upstream OpenTelemetry Collector with Grafana Labs products. --- ### Sources Tempo Set Up For Tracing Instrument Send Set Up Instrumentation Index (`docs/sources/tempo/set-up-for-tracing/instrument-send/set-up-instrumentation/_index.md`) --- title: Set up instrumentation menuTitle: Set up instrumentation description: Learn how to set up instrumentation for distributed tracing. weight: 500 --- # Set up instrumentation Client instrumentation is the first building block to a functioning distributed tracing visualization pipeline. Client instrumentation is the process of adding instrumentation points in the application that create and offload spans. When you set up instrumentation, you: 1. Choose an instrumentation method to use with your application 1. Instrument your application to generate traces After you set up instrumentation, you can [set up a collector](../set-up-collector/) to receive traces from your application. Refer to [About instrumentation](../about-instrumentation/) for more information about instrumentation and how it works. ## Choose an instrumentation method When sending traces to Tempo, you can choose between four methods: - Auto-instrumentation applies instrumentation automatically using agents or middleware, without code changes. - Zero-code instrumentation, which uses eBPF technology to instrument applications without code changes. [Grafana Beyla](https://grafana.com/docs/beyla/latest/) is an example of a zero-code instrumentation tool. - Manual instrumentation involves adding code to create spans and traces, giving full control over collected data. - Hybrid instrumentation, which combines auto and manual instrumentation, using automatic for most code and manual for custom tracing logic. Refer to [About instrumentation](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/about-instrumentation/) for more information. If you are using OTel or Alloy, refer to [Instrument an application with OpenTelemetry](https://grafana.com/docs/opentelemetry/instrument/) for more information. These instructions are specific to Grafana Cloud, but can be adapted for self-hosted Tempo. ## Instrument your app Most of the popular client instrumentation frameworks have SDKs in the most commonly used programming languages. You should pick one according to your application needs. OpenTelemetry has the most active development in the community and may be a better long-term choice. Popular instrumentation frameworks include: - [OpenTelemetry](https://opentelemetry.io/docs/concepts/instrumenting/) - [Zipkin](https://zipkin.io/pages/tracers_instrumentation) - [Grafana Beyla](https://grafana.com/docs/beyla/) ### Instrument using OpenTelemetry A collection of tools, APIs, and SDKs, OpenTelemetry helps engineers instrument, generate, collect, and export telemetry data such as metrics, logs, and traces, to analyze software performance and behavior. For more information refer to [OpenTelemetry overview](https://grafana.com/oss/opentelemetry/). If you are using OTel with Grafana Cloud, refer to [Instrument an application with OpenTelemetry](https://grafana.com/docs/opentelemetry/instrument/) for more information. #### Use OpenTelemetry auto-instrumentation frameworks OpenTelemetry provides auto-instrumentation agents and libraries of Java, .NET, Python, Go, and JavaScript applications, among others. For more information, refer for the [OpenTelemetry Instrumentation documentation](https://opentelemetry.io/docs/instrumentation/). These libraries capture telemetry information from a client application with minimal manual instrumentation of the codebase. - [OpenTelemetry Java auto-instrumentation](https://github.com/open-telemetry/opentelemetry-java-instrumentation) - [OpenTelemetry .NET auto-instrumentation](https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation) - [How to configure OpenTelemetry .NET automatic instrumentation with Grafana Cloud](/blog/2023/10/31/how-to-configure-opentelemetry-.net-automatic-instrumentation-with-grafana-cloud) - [OpenTelemetry Python auto-instrumentation](https://github.com/open-telemetry/opentelemetry-python-contrib) - [OpenTelemetry Go auto-instrumentation](https://github.com/open-telemetry/opentelemetry-go-instrumentation) and [documentation](https://opentelemetry.io/docs/instrumentation/go/getting-started/) {{< admonition type="note" >}} Jaeger client libraries have been deprecated. For more information, refer to the [Deprecating Jaeger clients article](https://www.jaegertracing.io/docs/1.50/client-libraries/#deprecating-jaeger-clients). Jaeger recommends using OpenTelemetry SDKs. {{< /admonition >}} #### Additional OTel resources - [Grafana Application Observability](https://grafana.com/docs/grafana-cloud/monitor-applications/application-observability/) - [OpenTelemetry Go instrumentation examples](https://github.com/open-telemetry/opentelemetry-go-instrumentation/tree/main/examples) - [OpenTelemetry Language Specific Instrumentation](https://opentelemetry.io/docs/instrumentation/) ### Instrument with Zipkin auto-instrumentation Zipkin is a distributed tracing system that helps gather timing data needed to troubleshoot latency problems in microservice architectures. Refer to the [Zipkin Language Specific Instrumentation](https://zipkin.io/pages/tracers_instrumentation.html) documentation for more information. If you are using Zipkin with Alloy, refer to the Zipkin receiver, [otelcol.receiver.zipkin documentation](https://grafana.com/docs/alloy/<ALlOY_VERSION>/reference/components/otelcol/otelcol.receiver.zipkin/). In addition, you can use Zipkin to instrument a library, refer to [Instrumenting a library with Zipkin](https://zipkin.io/pages/instrumenting.html) Within Grafana, you can also use these Zipkin specific features: - [Zipkin data source](https://grafana.com/docs/grafana/latest/datasources/zipkin/) - [Monitor Zipkin with Prometheus and Grafana Cloud](https://grafana.com/docs/grafana-cloud/send-data/metrics/metrics-prometheus/prometheus-config-examples/the-zipkin-community-zipkin/) ### Instrument with Grafana Beyla Grafana Beyla is an eBPF-based application zero-code instrumentation tool to easily get started with Application Observability. Beyla uses eBPF to automatically inspect application executables and the OS networking layer, and capture trace spans related to web transactions and Rate Errors Duration (RED) metrics for Linux HTTP/S and gRPC services. All data capture occurs without any modifications to application code or configuration. Refer to [Set up Beyla](https://grafana.com/docs/beyla/<BEYLA_VERSION>/setup/) for information about how to instrument using Beyla. {{< admonition type="note" >}} While Beyla is a good choice for metrics, span metrics, and service graphs, if you choose to use it for tracing, check the [Beyla documentation](https://grafana.com/docs/beyla/<BEYLA_VERSION>/) to verify that it meets your tracing needs. {{< /admonition >}} ## Grafana Blog The Grafana blog periodically features instrumentation posts. - [Why OpenTelemetry instrumentation needs both eBPF and SDKs](https://grafana.com/blog/why-opentelemetry-instrumentation-needs-both-ebpf-and-sdks/) - [Database observability: How OpenTelemetry semantic conventions improve consistency across signals](https://grafana.com/blog/database-observability-how-opentelemetry-semantic-conventions-improve-consistency-across-signals/) - [Introducing OpenTelemetry eBPF Instrumentation: Why we donated Grafana Beyla to OpenTelemetry](https://grafana.com/blog/opentelemetry-ebpf-instrumentation-beyla-donation/) - [Demystifying the OpenTelemetry Operator: Observing Kubernetes applications without writing code](https://grafana.com/blog/demystifying-the-opentelemetry-operator-observing-kubernetes-applications-without-writing-code/) - [How to instrument your Python application using OpenTelemetry](https://grafana.com/blog/how-to-instrument-your-python-application-using-opentelemetry/) ### Community resources - [NodeJS](https://github.com/mnadeem/nodejs-opentelemetry-tempo) - [Java Spring Boot](https://github.com/mnadeem/boot-opentelemetry-tempo) - [Python](https://github.com/dgzlopes/foobar-demo) ## Next steps After you set up instrumentation, you can [set up a collector](../set-up-collector/) to receive traces from your application. --- ### Sources Tempo Set Up For Tracing Instrument Send Test Instrumentation Index (`docs/sources/tempo/set-up-for-tracing/instrument-send/test-instrumentation/_index.md`) --- title: Test your instrumentation menuTitle: Test your instrumentation description: Learn how to test your instrumentation by visualizing traces data. weight: 600 draft: true --- # Test your instrumentation ## Use OTel console exporter to test your instrumentation ## Use Alloy to test your instrumentation --- ### Sources Tempo Set Up For Tracing Setup Tempo Deploy Index (`docs/sources/tempo/set-up-for-tracing/setup-tempo/deploy/_index.md`) --- title: Deploy Tempo menuTitle: Deploy your Tempo instance description: Deploy Grafana Tempo for your tracing needs. weight: 300 aliases: - ../../deployment/ # /docs/tempo/next/deployment/ - ../../deployment/deployment/ # /docs/tempo/next/deployment/deployment/ - ../../setup/deployment/ # /docs/tempo/next/setup/deployment/ --- # Deploy Tempo Tempo can be easily deployed through a number of tools, including Helm, Tanka, Kubernetes, and Docker. The following procedures provide example Tempo deployments that you can use as a starting point. Tempo can be deployed in a number of ways, depending on your needs and environment. You can deploy Tempo in a monolithic mode or in a microservices mode. You can also use Docker to deploy Tempo using [the Docker examples](https://github.com/grafana/tempo/tree/main/example/docker-compose). {{< admonition type="note" >}} Grafana Tempo does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services to prevent unauthorized access to Tempo (for example, nginx). [Manage authentication](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/authentication/) for more details {{< /admonition >}} ## Kafka requirement Tempo 3.0 requires a Kafka-compatible message queue as part of its architecture. Distributors write incoming trace data to Kafka, and downstream components such as block-builders and live-stores consume from it. Any Kafka-compatible system works. ## Deploy locally Monolithic mode (single binary) is commonly used for a local installation, testing, or small-scale deployments. This mode can be deployed using a pre-compiled binary, OS-specific packaging, or Docker image. While it's possible to deploy monolithic mode in a Kubernetes cluster, it is not recommended for production use. Check out the [Docker examples](https://github.com/grafana/tempo/tree/main/example/docker-compose) for more information. ## Deploy using Kubernetes Kubernetes deployment examples: - [Deploy with Helm](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/kubernetes/helm-chart/) (microservices and monolithic) - [Deploy with Tempo Operator](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/kubernetes/operator/) (microservices) - [Deploy on Kubernetes using Tanka](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka/) (microservices) {{< admonition type="note" >}} The Tanka and Helm examples are equivalent. They are both provided for people who prefer different configuration mechanisms. {{< /admonition >}} For more information, refer to [Deploy Tempo on Kubernetes](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/kubernetes/). --- ### Sources Tempo Set Up For Tracing Setup Tempo Plan Index (`docs/sources/tempo/set-up-for-tracing/setup-tempo/plan/_index.md`) --- title: Plan your Tempo deployment menuTitle: Plan your deployment description: Plan your Grafana Tempo deployment weight: 200 --- # Plan your Tempo deployment Before you deploy Tempo, you should consider how you plan to use traces and Tempo as well as any special requirements for Tempo 3.0. ## Considerations for Tempo 3.0 Before planning your Tempo deployment, be aware of the following infrastructure requirements: - Object storage: Tempo stores trace data in object storage such as S3, GCS, or Azure Storage. Object storage is required for microservices deployments and recommended for production. Monolithic deployments can use local filesystem storage for development and test environments. - Kafka-compatible system: Microservices mode requires a Kafka-compatible system, such as Apache Kafka, Redpanda, or WarpStream, as the durable queue that decouples the write and read paths. Monolithic mode doesn't use Kafka. Refer to [Configure a Kafka-compatible backend](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/configure-kafka/) to plan the Kafka cluster and topic. Include these infrastructure components in your deployment planning and resource estimation. ## Develop a deployment plan To plan your Grafana Tempo deployment, you should: 1. Identify your use case Decide what you need from Tempo. For example, consider if you want a basic installation, horizontal scalability, multi-tenancy, or advanced data storage. Your use case determines the deployment architecture and resource requirements. 2. Estimate tracing data volume Assess the amount of tracing data your environment will generate. This helps you size your cluster and storage. Refer to the [Size your cluster](./size/) section for guidance on calculating storage and performance needs. 3. Choose a deployment mode. Tempo supports monolithic and microservices deployment modes. - Use monolithic mode for getting started, development, or smaller workloads. No Kafka is required. - Use microservices mode for production, high-volume, or highly available deployments. Requires a Kafka-compatible system. Review the [Deployment modes](./deployment-modes/) documentation to compare options and select the best fit for your use case. By following these recommendations, you can plan a Tempo deployment that matches your operational requirements and scales with your tracing data. ## Next steps Once you've planned your deployment, you can [Deploy your Tempo instance](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/) using the deployment mode you selected. --- ### Sources Tempo Set Up For Tracing Setup Tempo Test Index (`docs/sources/tempo/set-up-for-tracing/setup-tempo/test/_index.md`) --- title: Validate deployment menuTitle: Validate deployment description: Instructions for validating your Tempo deployment weight: 500 --- # Validate your Tempo deployment To test your Tempo deployment, select one of the procedures below to test your Tempo deployment: {{< section withDescriptions="true">}} --- ### Sources Tempo Set Up For Tracing Instrument Send Set Up Collector Grafana Alloy Index (`docs/sources/tempo/set-up-for-tracing/instrument-send/set-up-collector/grafana-alloy/_index.md`) --- title: Grafana Alloy description: Configure the Grafana Alloy to work with Tempo weight: 500 aliases: - ../../../configuration/grafana-alloy/ # /docs/tempo/latest/configuration/grafana-alloy/ - ../../../configuration/grafana-agent/ # /docs/tempo/latest/configuration/grafana-agent/ --- # Grafana Alloy Grafana Alloy offers native pipelines for Tempo, OpenTelemetry, Prometheus, Pyroscope, Loki, and many other metrics, logs, traces, and profile tools. In addition, you can use Alloy pipelines to do other tasks, such as configure alert rules in Loki and Mimir. Alloy is compatible with the OpenTelemetry Collector and Prometheus Agent. You can use Alloy to collect and forward traces to Tempo. Using Alloy provides a hassle-free option, especially when dealing with multiple applications or microservices, allowing you to centralize the tracing process without changing your application's codebase. You can use Alloy as an alternative to either of these solutions or combine it into a hybrid system of multiple collectors and agents. You can deploy Alloy anywhere within your IT infrastructure and pair it with your Grafana LGTM stack, a telemetry backend from Grafana Cloud, or any other compatible backend from any other vendor. You can configure Alloy to fit your needs for on-premise, cloud-only, or a mix of both. <p align="center"><img src="/media/docs/tempo/intro/tempo-auto-log.svg" alt="Automatic logging overview"></p> It's commonly used as a tracing pipeline, offloading traces from the application and forwarding them to a storage backend. Grafana Alloy configuration files are written in the [Alloy configuration syntax](https://grafana.com/docs/alloy/<ALLOY_VERSION>/get-started/configuration-syntax/). For more information, refer to the [Introduction to Grafana Alloy](https://grafana.com/docs/alloy/<ALLOY_VERSION>/introduction). ## Architecture Grafana Alloy can run a set of tracing pipelines to collect data from your applications and write it to Tempo. Pipelines are built using OpenTelemetry, and consist of `receivers`, `processors`, and `exporters`. The architecture mirrors that of the OpenTelemetry Collector's [design](https://opentelemetry.io/docs/collector/architecture/). Refer to the [components reference](https://grafana.com/docs/alloy/latest/reference/components/) for all available configuration options. <p align="center"><img src="alloy-pipeline-architecture.svg" alt="Alloy tracing pipeline architecture showing receivers, processors, and exporters"></p> This lets you configure multiple distinct tracing pipelines, each of which collects separate spans and sends them to different backends. ## Set up Alloy to receive traces Grafana Alloy supports multiple ingestion receivers: OTLP (OpenTelemetry), Jaeger, Zipkin, and [Kafka](https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/otelcol/otelcol.receiver.kafka/). Each tracing pipeline can be configured to receive traces in all these formats. Traces that arrive to a pipeline go through the receivers/processors/exporters defined in that pipeline. To use Alloy for tracing, you need to: 1. [Set up Grafana Alloy](https://grafana.com/docs/alloy/<ALLOY_VERSION>/set-up/) 2. [Configure Grafana Alloy](https://grafana.com/docs/alloy/<ALLOY_VERSION>/configure/) 3. Set up any additional features Refer to [Collect and forward data with Grafana Alloy](https://grafana.com/docs/alloy/<ALLOY_VERSION>/collect/) for examples of collecting data. ### Example configuration The following example configures Alloy to receive traces over both OTLP gRPC (port 4317) and OTLP HTTP (port 4318), then export them to Tempo: ```alloy otelcol.receiver.otlp "default" { grpc { endpoint = "0.0.0.0:4317" } http { endpoint = "0.0.0.0:4318" } output { traces = [otelcol.exporter.otlp.default.input] } } otelcol.exporter.otlp "default" { client { endpoint = env("TEMPO_ENDPOINT") } } ``` {{< admonition type="note" >}} When running Alloy in Docker or Kubernetes, set the receiver endpoint to `0.0.0.0:<port>` as shown above. The default binds to `localhost`, which prevents other containers or Pods from sending traces to Alloy. {{< /admonition >}} Refer to the `otelcol.receiver.otlp` block in the [components reference](https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/otelcol/otelcol.receiver.otlp/) for all available receiver options. ## Set up pipeline processing Grafana Alloy processes tracing data as it flows through the pipeline to make the distributed tracing system more reliable and leverage the data for other purposes such as trace discovery, tail-based sampling, and generating metrics. ### Batching Alloy supports batching of traces. Batching helps better compress the data, reduces the number of outgoing connections, and is a recommended best practice. To configure it, refer to the `otelcol.processor.batch` block in the [components reference](https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/otelcol/otelcol.processor.batch/). ### Attributes manipulation Grafana Alloy allows for general manipulation of attributes on spans that pass through it. A common use may be to add an environment or cluster variable. There are several processors that can manipulate attributes, some examples include: the `otelcol.processor.attributes` block in the [component reference](https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/otelcol/otelcol.processor.attributes/) and the `otelcol.processor.transform` block [component reference](https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/otelcol/otelcol.processor.transform/) ### Attach metadata with Prometheus Service Discovery Prometheus Service Discovery mechanisms enable you to attach the same metadata to your traces as your metrics. For example, for Kubernetes users this means that you can dynamically attach metadata for namespace, Pod, and name of the container sending spans. ```alloy otelcol.receiver.otlp "default" { http {} grpc {} output { traces = [otelcol.processor.k8sattributes.default.input] } } otelcol.processor.k8sattributes "default" { extract { metadata = [ "k8s.namespace.name", "k8s.pod.name", "k8s.container.name" ] } output { traces = [otelcol.exporter.otlp.default.input] } } otelcol.exporter.otlp "default" { client { endpoint = env("OTLP_ENDPOINT") } } ``` Refer to the `otelcol.processor.k8sattributes` block in the [components reference](https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/otelcol/otelcol.processor.k8sattributes/). ### Trace discovery through automatic logging Automatic logging writes well formatted log lines to help with trace discovery. For a closer look into the feature, visit [Automatic logging](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/grafana-alloy/automatic-logging/). ### Tail-based sampling Alloy implements tail-based sampling for distributed tracing systems and multi-instance Alloy deployments. With this feature, you can make sampling decisions based on data from a trace, rather than exclusively with probabilistic methods. For a detailed description, refer to [Tail sampling](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/tail-sampling). ### Generate metrics from spans Alloy can emit span metrics from traces in this pipeline. If you sample traces before they reach Tempo, choose where to generate those metrics first. Refer to [Choose where to generate metrics from traces](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/where-to-generate-metrics/). For Alloy settings, refer to [Use Alloy to generate metrics from spans](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics/span-metrics-alloy/). ### Service graph metrics Service graph metrics represent the relationships between services within a distributed system. This service graphs processor builds a map of services by analyzing traces, with the objective to find _edges_. Edges are spans with a parent-child relationship, that represent a jump, such as a request, between two services. The amount of requests and their duration are recorded as metrics, which are used to represent the graph. To read more about this processor, refer to [Service graphs](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/). ## Export spans Alloy can export traces to multiple different backends for every tracing pipeline. Exporting is built using the OpenTelemetry Collector's [OTLP exporter](https://opentelemetry.io/docs/collector/components/exporter/). Alloy supports exporting tracing in OTLP format. Aside from endpoint and authentication, the exporter also provides mechanisms for retrying on failure, and implements a queue buffering mechanism for transient failures, such as networking issues. To see all available options, refer to the `otelcol.exporter.otlp` block in the [Alloy configuration reference](https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/otelcol/otelcol.exporter.otlp/) and the `otelcol.exporter.otlphttp` block in the [Alloy configuration reference](https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/otelcol/otelcol.exporter.otlphttp/). --- ### Sources Tempo Set Up For Tracing Instrument Send Set Up Collector Otel Collector Index (`docs/sources/tempo/set-up-for-tracing/instrument-send/set-up-collector/otel-collector/_index.md`) --- title: OpenTelemetry Collector menuTitle: OpenTelemetry Collector description: Configure the upstream OpenTelemetry Collector to send traces to Tempo. weight: 550 --- # OpenTelemetry Collector The [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) is a vendor-agnostic component for receiving, processing, and exporting telemetry data. It's maintained by the OpenTelemetry project and the Cloud Native Computing Foundation (CNCF). You can use the OpenTelemetry Collector to collect and forward traces to Tempo. Using a collector provides a centralized tracing pipeline, especially when dealing with multiple applications or microservices. This lets you manage trace processing without changing your application's codebase. You can deploy the OpenTelemetry Collector anywhere within your infrastructure and pair it with a self-managed Tempo instance or Grafana Cloud Traces. For installation instructions, refer to the [Install the OpenTelemetry Collector](https://opentelemetry.io/docs/collector/installation/) documentation. For information about using the OpenTelemetry Collector with Grafana products, refer to the [Grafana OpenTelemetry Collector documentation](https://grafana.com/docs/opentelemetry/collector/opentelemetry-collector/). ## Architecture The OpenTelemetry Collector runs tracing pipelines that collect data from your applications and write it to Tempo. Pipelines consist of `receivers`, `processors`, and `exporters`, all configured in a single YAML file. This architecture lets you configure multiple distinct tracing pipelines, each of which collects separate spans and sends them to different backends. For more information, refer to the [OpenTelemetry Collector configuration documentation](https://opentelemetry.io/docs/collector/configuration/). ## Before you begin Before you set up the OpenTelemetry Collector to send traces to Tempo, you need: - The OpenTelemetry Collector [installed](https://opentelemetry.io/docs/collector/installation/). The [contrib distribution](https://github.com/open-telemetry/opentelemetry-collector-contrib) is recommended because it includes additional processors and exporters such as `k8sattributes`, `tail_sampling`, and the `spanmetrics` connector. - A running Tempo instance with the distributor OTLP endpoints accessible. By default, Tempo accepts OTLP traces over gRPC on port `4317`. To also accept HTTP, configure the OTLP HTTP receiver on port `4318`. - For Grafana Cloud, use the `otlphttp` exporter. Refer to [Send data to the Grafana Cloud OTLP endpoint](https://grafana.com/docs/grafana-cloud/send-data/otlp/send-data-otlp/) for connection and authentication details. ## Set up the OpenTelemetry Collector to receive traces The OpenTelemetry Collector supports multiple ingestion receivers, including OTLP (OpenTelemetry Protocol), Jaeger, Zipkin, and Kafka. OTLP is the recommended protocol for forwarding traces to Tempo. Each tracing pipeline can be configured to receive traces in any of these formats. Traces that arrive to a pipeline go through the receivers, processors, and exporters defined in that pipeline. ### Minimal configuration The following example configures the OpenTelemetry Collector to receive OTLP traces over both gRPC and HTTP, batch them, and export to Tempo using the OTLP gRPC exporter: ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: exporters: otlp: endpoint: <TEMPO_HOST>:4317 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp] ``` Replace `<TEMPO_HOST>` with the hostname or address of your Tempo distributor. Set `tls.insecure: true` only for development or non-production environments. For production TLS configuration, refer to [TLS configuration](#tls-configuration). ### Use OTLP/HTTP instead of gRPC To export traces over HTTP instead of gRPC, use the `otlphttp` exporter. The following example is a complete configuration: ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: exporters: otlphttp: endpoint: http://<TEMPO_HOST>:4318 service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp] ``` ## Set up pipeline processing The OpenTelemetry Collector processes tracing data as it flows through the pipeline to make the distributed tracing system more reliable and to leverage the data for other purposes such as tail-based sampling and generating metrics. ### Batching The OpenTelemetry Collector supports batching of traces. Batching helps better compress the data, reduces the number of outgoing connections, and is a recommended best practice. To configure batching, add the `batch` processor to your pipeline. Refer to the [`batch` processor documentation](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor) for all available configuration options. ### Attribute manipulation The OpenTelemetry Collector allows for general manipulation of attributes on spans that pass through it. A common use case is to add environment or cluster labels. Several processors can manipulate attributes: - The [`attributes` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor) inserts, updates, deletes, or hashes span attributes. - The [`transform` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) applies transformations using the OpenTelemetry Transformation Language (OTTL). - The [`resourcedetection` processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor) automatically detects resource information from the host environment, including Docker, cloud providers (AWS, GCP, Azure), and Kubernetes. ### Attach metadata with Kubernetes attributes The `k8sattributes` processor enriches traces with Kubernetes metadata, allowing you to attach the same metadata to your traces as your metrics. For Kubernetes users, this means you can dynamically attach metadata for namespace, Pod, and container name. ```yaml receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: k8sattributes: extract: metadata: - k8s.namespace.name - k8s.pod.name - k8s.container.name batch: exporters: otlp: endpoint: <TEMPO_HOST>:4317 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [k8sattributes, batch] exporters: [otlp] ``` Refer to the [`k8sattributes` processor documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor) for all available configuration options. {{< admonition type="note" >}} The `k8sattributes` processor is only available in the [contrib distribution](https://github.com/open-telemetry/opentelemetry-collector-contrib) of the OpenTelemetry Collector. {{< /admonition >}} ### Tail-based sampling The OpenTelemetry Collector supports tail-based sampling, where sampling decisions are made after considering all or most of the spans in a trace. This lets you make sampling decisions based on trace data, rather than exclusively with probabilistic methods. You can, for example, sample only traces that contain errors or traces with high request durations. Refer to the [`tail_sampling` processor documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor) for configuration details. For a detailed description of sampling strategies, refer to the [Sampling](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/tail-sampling) documentation. ### Generate metrics from spans The `spanmetrics` connector generates RED (Rate, Error, Duration) metrics from trace spans flowing through the pipeline. Tempo can also generate these metrics after ingest. To decide between the collector and Tempo, refer to [Choose where to generate metrics from traces](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/where-to-generate-metrics/). Configure the `spanmetrics` connector before the `tail_sampling` processor so span metrics see every span. You can export the generated metrics using Prometheus remote write, OTLP, or any other supported metrics exporter. Refer to the [`spanmetrics` connector documentation](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/spanmetricsconnector) for configuration details. ### Service graph metrics Service graph metrics represent the relationships between services within a distributed system. The OpenTelemetry Collector contrib distribution includes a [service graph connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/servicegraphconnector) that builds a map of services by analyzing traces, with the objective of finding edges, which are spans with a parent-child relationship that represent a request between two services. Refer to [Service graphs](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/) for more information. ## Export traces to Tempo The OpenTelemetry Collector can export traces to Tempo using either the OTLP gRPC or OTLP HTTP exporter. Both exporters provide mechanisms for retrying on failure and implement a queue buffering mechanism for transient failures, such as networking issues. To see all available options, refer to: - [`otlp` exporter documentation](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlpexporter) for gRPC export - [`otlphttp` exporter documentation](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/otlphttpexporter) for HTTP export ## Multi-tenancy Tempo supports multi-tenancy through the `X-Scope-OrgID` HTTP header. To send traces for a specific tenant from the OpenTelemetry Collector, set the `x-scope-orgid` header in the exporter configuration: ```yaml exporters: otlp: endpoint: <TEMPO_HOST>:4317 tls: insecure: true headers: x-scope-orgid: <TENANT_ID> ``` Replace _TENANT_ID_ with the identifier for your tenant. Refer to [Enable multi-tenancy](/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/multitenancy/) for more information about configuring multi-tenancy in Tempo. ## TLS configuration For production deployments, configure TLS on the exporter to encrypt communication between the OpenTelemetry Collector and Tempo: ```yaml exporters: otlp: endpoint: <TEMPO_HOST>:4317 tls: cert_file: /certs/client.crt key_file: /certs/client.key ca_file: /certs/ca.crt ``` Refer to [Configure TLS communication](/docs/tempo/<TEMPO_VERSION>/configuration/network/tls/) for details on configuring TLS on the Tempo receiver side. ## Tempo 3.0 considerations When using the OpenTelemetry Collector with Tempo 3.0, be aware of the following changes. ### OpenCensus removal Tempo 3.0 removes the OpenCensus receiver. If your OpenTelemetry Collector previously used the `opencensus` exporter to send traces to Tempo, switch to the `otlp` or `otlphttp` exporter. ### No collector configuration changes required Tempo 3.0 introduces a new internal architecture (Kafka-based ingestion, live store), but the distributor OTLP endpoints on ports `4317` and `4318` are unchanged. Your existing OpenTelemetry Collector configuration continues to work without modification. ### Verify the Tempo receiver binding address Since Tempo 2.7, the OTLP receiver defaults to `localhost` instead of `0.0.0.0`. When running Tempo in Docker or Kubernetes, ensure the Tempo configuration explicitly binds to an accessible address, for example, `endpoint: "0.0.0.0:4317"`. Refer to [OpenTelemetry Collector receiver listens on `localhost` by default](/docs/tempo/<TEMPO_VERSION>/release-notes/version-2/v2-7/#opentelemetry-collector-receiver-listens-on-localhost-by-default) for details. --- ### Sources Tempo Set Up For Tracing Instrument Send Set Up Collector Tail Sampling Index (`docs/sources/tempo/set-up-for-tracing/instrument-send/set-up-collector/tail-sampling/_index.md`) --- title: Sampling menuTitle: Sampling description: Learn how to streamline tracing data by using sampling to determine which data to keep and which to drop. aliases: - ../../../configuration/grafana-alloy/tail-sampling/ # /docs/tempo/next/configuration/grafana-alloy/tail-sampling/ - ../../../configuration/grafana-agent/tail-sampling/ # /docs/tempo/latest/configuration/grafana-agent/tail-sampling/ --- # Sampling Grafana Tempo is a cost-effective solution that ingests and stores traces that provide maximum observability across your application estate. However, sometimes constraints mean that storing all of your traces is not desirable, for example runtime or egress traffic related costs. There are a number of ways to lower trace volume, including varying sampling strategies. Sampling is the process of determining which traces to store (in Tempo or Grafana Cloud Traces) and which to discard. Sampling comes in two different strategy types: head and tail sampling. Sampling functionality exists in both [Grafana Alloy](https://grafana.com/docs/alloy/) and the OpenTelemetry Collector. Alloy can collect, process, and export telemetry signals, with configuration files written in [Alloy configuration syntax](https://grafana.com/docs/alloy/<ALLOY_VERSION>/get-started/configuration-syntax/). ## Head and tail sampling When sampling, you can use a head or tail sampling strategy. With a head sampling strategy, the decision to sample the trace is usually made as early as possible and doesn’t need to take into account the whole trace. It’s a simple but effective sampling strategy. {{< youtube id="8JDgAEM5nHs" >}} With a tail sampling strategy, the decision to sample a trace is made after considering all or most of the spans. For example, tail sampling is a good option to sample only traces that have errors or traces with long request duration. Tail sampling is more complex to configure, implement, and maintain but is the recommended sampling strategy for large systems with a high telemetry volume. You can use sampling with Tempo using Grafana or Grafana Cloud. {{< admonition type="tip" >}} If you send traces to Grafana Cloud Traces, you can use [Adaptive Traces](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/adaptive-telemetry/adaptive-traces/) instead of building and operating your own tail sampling pipeline. Adaptive Traces is a managed tail sampling capability that analyzes your trace data, recommends sampling policies, for example, keeping traces with errors or high latency, and applies them for you. Downsampling from Adaptive Traces doesn't affect generation of metrics. Metrics are generated before sampling. {{< /admonition >}} ### Resources - [OpenTelemetry Sampling documentation](https://opentelemetry.io/docs/concepts/sampling/) - Sampling in Grafana Cloud Traces with a collector: [Head sampling](https://grafana.com/docs/opentelemetry/collector/sampling/head/) and [Tail sampling](https://grafana.com/docs/opentelemetry/collector/sampling/tail/) - [Enable tail sampling in Tempo](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/grafana-alloy/tail-sampling/enable-tail-sampling/) - [Sampling policies and strategies](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/grafana-alloy/tail-sampling/policies-strategies/) - [Adaptive Traces](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/adaptive-telemetry/adaptive-traces/): managed tail sampling for Grafana Cloud Traces ## Sampling and telemetry correlation Sampling is a decision on whether or not to keep (and then store) a trace, or whether to discard it. These decisions have implications when it comes to correlating trace data with other signals. For example, many services that are instrumented also produce logs, metrics, or profiles. These signals can reference each other. In the case of a trace, this reference can be via a trace ID embedded into a [log line](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/traces-in-grafana/link-trace-id/), an [exemplar](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/fundamentals/exemplars/) embedded into a metric value, or a profile ID [embedded into a trace](https://grafana.com/docs/grafana-cloud/monitor-applications/profiles/traces-to-profiles/). By choosing to not sample a trace, a particular signal references the dropped trace's ID in some cases. This drop can lead to a situation in Grafana where following a link to a trace ID from a log line or an exemplar from a metric value, results in a query for that trace ID failing because the trace has not been sampled. Profiles may not show up without specifically querying for them, because a trace that would have included the profile's flame graph hasn't been stored. This isn't usually a huge issue, because sampling policies tend to be chosen that show non-normative behavior, for example, errors being thrown or long latencies on requests. An observer is more likely to be choosing traces that show these issues rather than the required behavior. Understand how signals correlate between each other helps determine how to choose these policies. ## How tail sampling works in the OpenTelemetry Tail Sampling Processor In tail sampling, sampling decisions are made at the end of the workflow allowing for a more accurate sampling decision. Alloy uses the [OpenTelemetry Tail Sampling Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md). Alloy organizes spans by trace ID and evaluates its data to see if it meets one of the defined policy types (for example, `latency` or `status_code`). For instance, a policy can check if a trace contains an error or the trace duration was longer than a specified threshold. A trace is sampled if it meets the conditions of at least one policy. ### Decision periods To group spans by trace ID, Alloy buffers spans for a configurable amount of time, after which it considers the trace complete. This configurable amount of time is known as the decision period. Longer running traces are split into more than one. In situations where a specific trace is longer in duration than the decision period, multiple decisions might be made for any future spans that fall outside of the decision period window. This can result in some spans for a trace being sampled, while others are not. For example, consider a situation where the tail sampler decision period is 10 seconds, and a single policy exists to sample traces where an error is set on at least one span. One of the traces is 20 seconds in duration and a single span at time offset 15 seconds exhibits an error status. When the first span for the trace is observed, the decision period time of 10 seconds is initiated. After the decision period has expired, the tail sampler won't have observed any spans with an error status, and will therefore discard the trace spans. When the next span for the trace arrives, a new decision period of 10 seconds begins. In this period, one of the observed spans has an error set on it. When the decision period expires, all of the spans for the trace in that period will be sampled. This leads to a fragmented trace being stored in Tempo, where only the spans for the last 10 seconds of the trace will be available to query. While this is still a potentially useful trace, careful determination of how to set the decision period is key to ensuring that trace spans are sampled correctly. However, using longer decision periods increases the memory overhead of buffering the spans required to make a decision for each trace. For this reason, enabling a decision cache can ensure that previous sampling decisions for a specific trace ID are honored even after the expiration of the decision period. For more details, refer to the Caches section. ### Caches The OpenTelemetry tail sampling processor includes two separate caches, the sampled and non-sampled caches. The sampled cache keeps a list of all trace IDs where a prior decision to keep spans has been made. The non-sampled cache keeps a list of all trace IDs where a prior decision to drop spans has been made. Both caches are configured by the maximum number of traces that should be stored in the cache, and can be enabled either independently or jointly. In the above diagram, should both caches be enabled, then a decision to drop samples for the trace is made after 10 seconds and the trace ID stored in the non-sampled cache. This means that even spans that have an error status for that trace are dropped after the initial decision period, as the non-sampled cache matches the trace ID and preemptively drops the span. However, the same is true should a sampled decision have been made, where any future spans do not match any policies but whose trace ID is found in the sampled cache. Understanding how these caches work ensures that you still keep decisions that have previously been made. For example, you could use the sampled cache to short-circuit future decisions for a trace, immediately sampling the incoming span. This allows a decision to be made without having to buffer any other spans. Here are some general guidelines for using caches. Every installation is different. Using the caches can impact the amount of data generated. | Cache type | Use case | Benefits/Considerations | | ------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Sample caches | Keep any future spans from traces that have been sampled. | <ul><li>Cuts down span storage per trace to only those matching policies.</li><li> Can cause fragmented tracing.</li></ul> | | Non-sampled | Drop any future spans from traces where a decision to not sample those traces has explicitly occurred. | <ul><li>Lowers chance of storing traces after the initial decision period. </li><li>Misses any trace whose spans exhibit future policy criteria matching.</li></ul> | | Both | Use an initial decision period that makes a decision once and uses that decision going forward. | <ul><li>Guarantees capture of full traces.</li><li>Lower chance of capturing useful traces with a long duration.</li><li>Can lose spans if they are longer than the decision period.</li><ul> | {{< admonition type="note" >}} Enabling both sampled and non-sampled caches exhibits functionality similar to that of not enabling caches. However, it short-circuits any future decision making once an initial decision period has expired. Enabling both caches lowers memory requirements for buffering spans. {{< /admonition >}} ### Tail sampling load balancing Situations may arise for multi-instance Alloy deployments where emitted spans belonging to the same trace may arrive at different instances. For most cases, sampling decisions rely on all the spans for a specific trace ID being received by a single instance. You can configure Alloy to load balance traces across instances by exporting spans belonging to a specific trace ID to the same instance. For example, if 10 traces are coming in and there are four Alloy instances, then each instance will receive three traces and one instance will receive four traces. The load balancing maintains consistent hashing across all instances. Tail sampling load balancing is usually carried out by running two layers of collectors. The first layer receives the telemetry data (in this case trace spans), and then distributes these to the second layer that carries out the sampling policies. Alloy includes a [load-balancing exporter](https://grafana.com/docs/alloy/<ALLOY_VERSION>/reference/components/otelcol/otelcol.exporter.loadbalancing/) that can carry out routing to further collector targets based on a set number of keys (in the case of trace sampling, usually the `traceID` key). Alloy uses the [OpenTelemetry load balancing exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/loadbalancingexporter/README.md). The routing key ensures that a specific collector in the second layer always handles spans from the same trace ID, guaranteeing that sampling decisions are made correctly. You can configure the exporter with targets using static IP addresses, multi-IP DNS A record entries, and a Kubernetes headless service resolver. Using this configuration lets you scale up or down the number of layer two collectors. There are some important points to note with the load balancer exporter around scaling and resilience, mostly around its eventual consistency model. For more information, refer to [Resilience and scaling considerations](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/loadbalancingexporter/README.md#resilience-and-scaling-considerations). The most important in terms of tail sampling is that routing occurs based on an algorithm taking into account the number of backends available to the load balancer. This can affect the target for trace ID spans before eventual consistency occurs. For an example manifest for a two layer OpenTelemetry Collector deployment based around Kubernetes services, refer to the [Kubernetes service resolver README](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/loadbalancingexporter/example/k8s-resolver/README.md). ## Pipeline workflows When implementing tail sampling into your telemetry collection pipeline, there are some considerations that should be applied. The act of sampling reduces the amount of tracing telemetry data that's sent to Tempo. This can have an effect on observation of data inside Grafana. The following is a suggested pipeline that can be applied to both [Grafana Alloy](https://grafana.com/docs/alloy/latest/) and the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/), to carry out tail sampling, but also ensure that other telemetry signals are still captured for observation from within Grafana and Grafana Cloud. This section describes how to build and operate this pipeline yourself. If you send traces to Grafana Cloud Traces, [Adaptive Traces](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/adaptive-telemetry/adaptive-traces/) provides equivalent tail sampling as a managed feature, without requiring you to deploy and maintain a two-layer collector pipeline. Downsampling from Adaptive Traces doesn't affect generation of metrics. Metrics are generated before sampling. To decide whether to generate those metrics in the collector or in Tempo, refer to [Choose where to generate metrics from traces](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/where-to-generate-metrics/). This pipeline exists in the second layer of collectors, sent data by the load balancing layer, and is commonly deployed as a Kubernetes `StatefulSet` to ensure that each instance has a consistent identity. A realistic example pipeline could be made of up the following components: - The **OTLP Receiver** is the [OpenTelemetry Protocol](https://opentelemetry.io/docs/specs/otel/protocol/) (OTLP) receiver in this pipeline, and receives traces from the load balancing exporter. This receiver is responsible for initiating the processing pipeline within this collector layer. - The **Transform Processor** is used to modify any incoming trace spans before they are exported to other components in the pipeline. This allows the mutation of attributes (for example, deletion, mutation, insertion, etc.), as well as any other required [OpenTelemetry Transform Language](https://opentelemetry.io/docs/collector/transforming-telemetry/) (OTTL)-based operations. This component must come before metric generation Connectors or the tail sampling Processor so that required changes can be used for label names (for metrics) or policy matching (for tail sampling). - The **SpanMetrics Connector** is responsible for extracting metrics from the incoming traces, and can be used as a fork in the pipeline. These metrics include crucial information such as trace latency, error rates, and other performance indicators, which are essential for understanding the health and performance of your services. It's important to ensure that this Connector is configured to receive span data before any tail sampling occurs. - The **ServiceGraph Connector** generates service dependency graphs from the traces, and can be used as a fork in the pipeline or chained together with the span metrics connector. These graphs visually represent the interactions between various services in your system, helping to identify bottlenecks and understand the flow of requests. It's important to ensure that this Connector is configured to receive span data before any tail sampling occurs. - The **Tail Sampling Processor** is the core of the secondary collector layer. It applies the sampling policies you’ve configured to decide which traces should be retained and further processed. The sampling decision is made after the entire trace has been observed, or the decision wait time has elapsed, allowing the processor to make more informed choices based on the full context of the trace. - The **OTLP Exporter** exports the sampled traces (or generated span and service metrics) to Grafana Cloud via OTLP. - The **Prometheus Exporter** is optional. If metrics aren't sent via OTLP, then you can use this component to send Prometheus compatible metrics to [Mimir](https://grafana.com/oss/mimir/) or [Grafana Cloud Metrics](https://grafana.com/products/cloud/metrics/). --- ### Sources Tempo Set Up For Tracing Setup Tempo Deploy Kubernetes Index (`docs/sources/tempo/set-up-for-tracing/setup-tempo/deploy/kubernetes/_index.md`) --- title: Deploy Tempo on Kubernetes description: Instructions for deploying Tempo on Kubernetes weight: 400 --- # Deploy Tempo on Kubernetes Kubernetes manages distributed deployments and provides orchestration for containerized applications. Kubernetes offers several deployment options for Tempo: - **Helm charts**: Package manager approach for installation and configuration - **Tempo Operator**: Kubernetes-native operator for managing Tempo deployments - **Tanka/Jsonnet**: Configuration management using Jsonnet templating - **Kubernetes manifests**: Direct YAML configuration files Choose the method that best fits your Kubernetes workflow and operational requirements. {{< admonition type="note" >}} Grafana Tempo doesn't come with any included authentication layer. You must run an authenticating reverse proxy in front of your services to prevent unauthorized access to Tempo (for example, nginx). [Manage authentication](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/authentication/) for more details {{< /admonition >}} ## Helm Helm charts are available in the `grafana-community/helm-charts` repository: - [monolithic mode](https://github.com/grafana/helm-charts/tree/main/charts/tempo) - [microservices mode](https://github.com/grafana-community/helm-charts/tree/main/charts/tempo-distributed) and [`tempo-distributed` chart documentation](/docs/helm-charts/tempo-distributed/next/) In addition, several Helm chart examples are available in the Tempo repository. ### Kubernetes Tempo Operator The operator is available in [grafana/tempo-operator](https://github.com/grafana/tempo-operator) repository. The operator reconciles `TempoStack` resource to deploy and manage Tempo microservices installation. Refer to the [operator documentation](../kubernetes/operator/) for more details. ### Tanka/Jsonnet The Jsonnet files that you need to deploy Tempo with Tanka are available here: - [monolithic mode](https://github.com/grafana/tempo/tree/main/operations/jsonnet/single-binary) - [microservices mode](https://github.com/grafana/tempo/tree/main/operations/jsonnet/microservices) Here are a few [examples](https://github.com/grafana/tempo/tree/main/example/tk) that use official Jsonnet files. They display the full range of configurations available to Tempo. Refer to [Deploy on Kubernetes with Tanka](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka/) for an example installation using Tanka. ### Kubernetes manifests You can find a collection of Kubernetes manifests to deploy Tempo in the [operations/jsonnet-compiled](https://github.com/grafana/tempo/tree/main/operations/jsonnet-compiled) folder. These are generated using the Tanka/Jsonnet. --- ### Sources Tempo Set Up For Tracing Setup Tempo Deploy Locally Index (`docs/sources/tempo/set-up-for-tracing/setup-tempo/deploy/locally/_index.md`) --- title: Deploy Tempo locally description: Instructions for deploying Tempo locally using Docker Compose or on Linux weight: 400 --- # Deploy Tempo locally This section provides instructions for deploying Tempo locally using Docker Compose or directly on a Linux host. You can download the latest version from the [Tempo Releases page](https://github.com/grafana/tempo/releases). {{< section withDescriptions="true">}} {{< admonition type="note" >}} Grafana Tempo doesn't come with any included authentication layer. You must run an authenticating reverse proxy in front of your services to prevent unauthorized access to Tempo (for example, nginx). [Manage authentication](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/authentication/) for more details {{< /admonition >}} --- ### Sources Tempo Set Up For Tracing Setup Tempo Deploy Kubernetes Helm Chart Index (`docs/sources/tempo/set-up-for-tracing/setup-tempo/deploy/kubernetes/helm-chart/_index.md`) --- title: Deploy Tempo with Helm menuTitle: Deploy with Helm description: Learn how to deploy Tempo with Helm aliases: - ../../../../setup/helm-chart/ # /docs/tempo/next/setup/helm-chart/ weight: 350 --- # Deploy Tempo with Helm The Helm charts for Grafana Tempo allow you to configure, install, and upgrade Grafana Tempo within a Kubernetes cluster. To deploy Tempo using the `tempo-distributed` Helm chart, read the [Get started with Grafana Tempo using Helm](/docs/helm-charts/tempo-distributed/next/get-started-helm-charts). ## Example Helm chart The Tempo repository has an [example Helm chart](https://github.com/grafana/tempo/tree/main/example/helm) that shows a complete microservice-based deployment. ## Helm charts for deployment Tempo has two primary charts used for deployment: - [`tempo-distributed` Helm chart](https://github.com/grafana-community/helm-charts/tree/main/charts/tempo-distributed) deploys Tempo in microservices mode ([read the documentation](/docs/helm-charts/tempo-distributed/next/get-started-helm-charts)) - [`tempo` Helm chart](https://github.com/grafana-community/helm-charts/tree/main/charts/tempo) deploys Tempo in monolithic (single binary) mode --- ### Sources Tempo Set Up For Tracing Setup Tempo Deploy Kubernetes Operator Index (`docs/sources/tempo/set-up-for-tracing/setup-tempo/deploy/kubernetes/operator/_index.md`) --- title: Deploy Tempo with Tempo Operator menuTitle: Deploy with operator description: Learn how to deploy Tempo with Tempo Operator weight: 375 aliases: - ../../../../operator/operator/ # /docs/tempo/next/operator/operator/ - ../../../../setup/operator/ # /docs/tempo/next/setup/operator/ --- # Deploy Tempo with Tempo Operator The Tempo Operator allows you to configure, install, upgrade, and operate Grafana Tempo on Kubernetes and OpenShift clusters. Some of the operator features are: - **Resource Limits** - Specify overall resource requests and limits in the `TempoStack` CR; the operator assigns fractions of it to each component - **AuthN and AuthZ** - Supports OpenID Control (OIDC) and role-based access control (RBAC) - **Managed upgrades** - Updating the operator will automatically update all managed Tempo clusters - **Multitenancy** - Multiple tenants can send traces to the same Tempo cluster - **mTLS** - Communication between the Tempo components can be secured via mTLS - **Jaeger UI** - Traces can be visualized in Jaeger UI and exposed via Ingress or OpenShift Route - **Observability** - The operator and `TempoStack` operands expose telemetry (metrics, traces) and integrate with Prometheus `ServiceMonitor` and `PrometheusRule` The source of the Tempo Operator can be found at [grafana/tempo-operator](https://github.com/grafana/tempo-operator). ## Installation The operator can be installed from: - [Kubernetes manifest](https://github.com/grafana/tempo-operator/releases/latest/download/tempo-operator.yaml) file on a Kubernetes cluster - [operatorhub.io](https://operatorhub.io/operator/tempo-operator) on a Kubernetes cluster - OperatorHub on an OpenShift cluster ## Compatibility ### Tempo The supported Tempo version by the operator can be found in the [changelog](https://github.com/grafana/tempo-operator/blob/main/CHANGELOG.md) or on the [release page](https://github.com/grafana/tempo-operator/releases). ### Kubernetes The supported Kubernetes versions can be found in the [changelog](https://github.com/grafana/tempo-operator/blob/main/CHANGELOG.md) or on the [release page](https://github.com/grafana/tempo-operator/releases). ### cert-manager The operator Kubernetes manifest installation files use cert-manger `v1` custom resources to provision certificates for admission webhooks. ## Community - Reach out to us on [#tempo-operator](https://grafana.slack.com/archives/C0414EUU39A) Grafana Slack channel. - Participate on [Tempo community call](https://grafana.com/docs/tempo/<TEMPO_VERSION>/community/). --- ### Sources Tempo Set Up For Tracing Setup Tempo Deploy Kubernetes Operator Quickstart (`docs/sources/tempo/set-up-for-tracing/setup-tempo/deploy/kubernetes/operator/quickstart.md`) --- title: 'Quickstart' description: Quickstart to deploy Tempo with the Tempo Operator menuTitle: Quickstart weight: 100 aliases: - ../../../../../operator/quickstart/ # /docs/tempo/next/operator/quickstart/ - ../../../../../setup/operator/quickstart/ # /docs/tempo/next/setup/operator/quickstart/ --- # Quickstart One page summary on how to start with Tempo Operator and `TempoStack`. ## Requirements The easiest way to start with the Tempo Operator is to use Kubernetes [kind](https://kind.sigs.k8s.io/). ## Deploy To install the operator in an existing cluster, make sure you have [cert-manager](https://cert-manager.io/docs/installation/) installed and run: ```shell kubectl apply -f https://github.com/grafana/tempo-operator/releases/latest/download/tempo-operator.yaml ``` Once you have the operator deployed you need to install a storage backend. For this quick start guide, we will install [`MinIO`](https://min.io/) as follows: ```shell kubectl apply -f https://raw.githubusercontent.com/grafana/tempo-operator/main/minio.yaml ``` After minio was deployed, create a secret for MinIO in the namespace you are using: ```yaml kubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: minio-test stringData: endpoint: http://minio.minio.svc:9000 bucket: tempo access_key_id: tempo access_key_secret: supersecret type: Opaque EOF ``` Then create Tempo CR: ```yaml kubectl apply -f - <<EOF apiVersion: tempo.grafana.com/v1alpha1 kind: TempoStack metadata: name: simplest spec: storage: secret: name: minio-test type: s3 storageSize: 1Gi resources: total: limits: memory: 2Gi cpu: 2000m template: queryFrontend: jaegerQuery: enabled: true EOF ``` After create the `TempoStack` CR, you should see a some pods on the namespace. Wait for the stack to stabilize. The stack deployed above is configured to receive Jaeger, Zipkin, and OpenTelemetry (OTLP) protocols. Because the Jaeger Query is enabled, you can also use the Jaeger UI to inspect the data. To do a quick test, deploy a Job that generates some traces. ```yaml kubectl apply -f - <<EOF apiVersion: batch/v1 kind: Job metadata: name: tracegen spec: template: spec: containers: - name: tracegen image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/tracegen:latest command: - "./tracegen" args: - -otlp-endpoint=tempo-simplest-distributor:4317 - -otlp-insecure - -duration=30s - -workers=1 restartPolicy: Never backoffLimit: 4 EOF ``` Forward the Jaeger Query port to see the traces: ``` kubectl port-forward svc/tempo-simplest-query-frontend 16686:16686 ``` Visit http://localhost:16686 to view the results. --- ### Project Context (`docs/project-context.md`) # Project context (local) ## Identity - **Product name (short):** Tempo - **Product name (first mention in prose):** Grafana Tempo - **GitHub org/repo:** grafana/tempo ## Branches and releases - **Default development branch:** `main` - **Release branch pattern:** `release-vX.Y` (for example, `release-v2.10`) - **Docs version mapping:** each `release-vX.Y` branch maps to the `vX.Y.x` version on grafana.com/docs/tempo/. Docs sync from `main` to the website repo via `docs/sources/tempo/sync.json`; the root `_index.md` sets `GRAFANA_VERSION: next` for the current development version ## Documentation paths - **Documentation root (filesystem):** `docs/sources/tempo/` - **Helm chart docs:** `docs/sources/helm-charts/tempo-distributed/` - **Shared partials (headless):** `docs/sources/tempo/shared/` (reusable snippets included by other pages) - **Generated pages (do not hand-edit):** `docs/sources/tempo/configuration/manifest.md` — refer to `docs/sources/tempo/configuration/AGENTS.md` for details. Only edit `_index.md` for config reference changes. - **Configuration reference:** `docs/sources/tempo/configuration/` - **Changelog:** `CHANGELOG.md` at repo root - **Architecture / "start here" pages:** `docs/sources/tempo/introduction/` (overview, glossary, trace structure) and `docs/sources/tempo/reference-tempo-architecture/` (detailed component reference) - **Design proposals (internal, not published):** `docs/design-proposals/` - **Doc build tooling:** `docs/Makefile`, `docs/docs.mk`, `docs/variables.mk`, `docs/make-docs` ## Documentation tree > **Maintenance note:** The section table below lists the major sections as of the last update. If a new top-level directory is added under `docs/sources/tempo/`, add a row to the table. The table is not auto-generated — update it manually when the section structure changes. The published docs live under `docs/sources/tempo/`. Major sections: | Section path | Title | Covers | |-------------|-------|--------| | `introduction/` | Introduction | Architecture, glossary, telemetry, Tempo in Grafana, trace structure | | `solutions-with-traces/` | Use traces to find solutions | Use-case pages: metrics queries, app insights, diagnose errors | | `set-up-for-tracing/` | Set up for tracing | Instrument → collectors → deploy Tempo → validate | | `set-up-for-tracing/instrument-send/` | Instrument for distributed tracing | Choose method, set up instrumentation, set up collector (Alloy, OTel, sampling), test | | `set-up-for-tracing/setup-tempo/` | Set up Tempo | Plan, deploy (locally, K8s/Helm/Operator/Tanka), validate, upgrade, CLI flags | | `traceql/` | TraceQL | Query language: constructing/tuning queries, architecture, query editor in Grafana | | `metrics-from-traces/` | Metrics from traces | Metrics-generator, span metrics, service graphs, TraceQL metrics | | `configuration/` | Configure Tempo | Core config, Parquet, tenant IDs, usage, network options, hosted storage (S3, GCS, Azure) | | `operations/` | Manage Tempo | Auth, caching, backend search, schema, CLI, vulture, dedicated columns, ingestion, monitoring, advanced systems | | `reference-tempo-architecture/` | Tempo architecture reference | Architecture overview, deployment modes, object storage, block format, partition ring, components | | `api_docs/` | Tempo HTTP API | HTTP ingest endpoints, MCP server doc | | `release-notes/` | Release notes | Current version (v3-0) at top level; `version-2/` (v2-0 through v2-10) and `version-1/` (v1-2 through v1-5) in subdirectories | | `troubleshooting/` | Troubleshoot Tempo | Sending traces, querying issues | | `community/` | Community | Community content | Helm chart docs under `docs/sources/helm-charts/tempo-distributed/`: > **Note:** The `tempo-distributed` Helm chart is now community-maintained at [grafana-community/helm-charts](https://github.com/grafana-community/helm-charts/tree/main/charts/tempo-distributed). The docs below reflect the last Grafana-maintained state and may have drifted. Verify against the community chart before updating or referencing these pages. | Section path | Title | |-------------|-------| | (root) | Grafana tempo-distributed Helm chart documentation | | `get-started-helm-charts/` | Get started with Grafana Tempo using the Helm chart | | `release-notes/` | Grafana Tempo Helm chart release notes | ## Helm chart - **Helm chart repository:** `https://github.com/grafana-community/helm-charts` (community-maintained; source of truth for chart values). Local checkout is typically a sibling directory: `../helm-charts` - **Tempo-related charts in that repo:** `charts/tempo`, `charts/tempo-distributed`, and `charts/tempo-vulture`. **Only `tempo-distributed` is documented** in the Tempo docs; `tempo` (single-binary) and `tempo-vulture` are not - **Documented chart path in that repo:** `charts/tempo-distributed` - **Helm docs location (filesystem):** `docs/sources/helm-charts/tempo-distributed/` (in this repo — prose install/get-started/release-notes pages) - **Generated Helm values reference (do not hand-edit):** `charts/tempo-distributed/README.md` in the helm-charts repo (generated by helm-docs from `values.yaml`). There is no generated values reference inside the Tempo docs tree ## Code ↔ documentation mapping | Code area | Documentation area | |-----------|---------------------| | `pkg/traceql/` | `docs/sources/tempo/traceql/` | | `pkg/api/` | `docs/sources/tempo/api_docs/` | | `modules/generator/` | `docs/sources/tempo/metrics-from-traces/metrics-generator/` and `docs/sources/tempo/reference-tempo-architecture/components/metrics-generator.md` | | `modules/distributor/` | `docs/sources/tempo/reference-tempo-architecture/components/distributor.md` | | `modules/querier/` | `docs/sources/tempo/reference-tempo-architecture/components/querier.md` | | `modules/frontend/` | `docs/sources/tempo/reference-tempo-architecture/components/query-frontend.md` | | `modules/blockbuilder/` | `docs/sources/tempo/reference-tempo-architecture/components/block-builder.md` | | `modules/livestore/` | `docs/sources/tempo/reference-tempo-architecture/components/live-store.md` | | `modules/cache/` | `docs/sources/tempo/operations/caching.md` | | `modules/overrides/` | `docs/sources/tempo/operations/manage-advanced-systems/` (user-configurable overrides) | | `modules/storage/` | `docs/sources/tempo/reference-tempo-architecture/object-storage.md` | | `tempodb/` | `docs/sources/tempo/configuration/parquet.md` and `docs/sources/tempo/reference-tempo-architecture/block-format.md` | | `cmd/tempo-cli/` | `docs/sources/tempo/operations/tempo_cli.md` | | `cmd/tempo-vulture/` | `docs/sources/tempo/operations/tempo-vulture.md` | | `cmd/tempo/` | `docs/sources/tempo/configuration/` | | `operations/` (repo root) | `docs/sources/helm-charts/tempo-distributed/` (Helm/Jsonnet ops tooling) | The legacy ingester component (replaced in 3.0 by live-store and block-builder) has no `modules/ingester/` directory or component doc page. Refer to the Gotchas section. ## Code validation paths Paths the agent should check when validating documentation claims against code. | What to validate | Where to look | |-----------------|---------------| | TraceQL syntax / grammar | `pkg/traceql/lexer.go`, `pkg/traceql/parse.go`, `pkg/traceql/expr.y.go`, `pkg/traceql/ast_validate.go` | | TraceQL types / intrinsics | `pkg/traceql/ast.go`, `pkg/traceql/enum_attributes.go`, `pkg/traceql/enum_statics.go` | | TraceQL operators | `pkg/traceql/enum_operators.go` | | TraceQL aggregates / hints | `pkg/traceql/enum_aggregates.go`, `pkg/traceql/enum_hints.go` | | TraceQL metrics engine | `pkg/traceql/engine_metrics.go`, `pkg/traceql/engine_metrics_functions.go` | | Configuration structs / defaults | `modules/*/config.go`, `cmd/tempo/app/*.go`, `cmd/tempo/app/modules.go` | | Manifest / config reference generation | `pkg/docsgen/generate_manifest.go` | | Metrics-generator config | `modules/generator/` | | Metrics-generator processors | `modules/generator/processor/spanmetrics/`, `modules/generator/processor/servicegraphs/` | | Valid processor names | `modules/generator/validation/fields.go` | | API endpoints | `pkg/api/http.go`, `cmd/tempo/`, `modules/frontend/` | | Distributor / ingestion behavior | `modules/distributor/`, `modules/livestore/`, `modules/blockbuilder/` | | Storage / compaction | `tempodb/`, `modules/storage/` | | Block format versions & default | `tempodb/encoding/versioned.go` (`DefaultEncoding`, `LatestEncoding`, `AllEncodings`, `WritesSupported`); per-version code in `tempodb/encoding/vparquet{3,4,5}/` | ## Frontmatter and site conventions - **Frontmatter fields:** `title`, `menuTitle`, `description`, `keywords`, `weight`, `aliases`, `topicType`, `versionDate` (optional; ISO date string indicating when the page was last reviewed or updated, for example `2026-03-20`) - **`topicType` values:** `concept` (reference/explanatory pages), `task` (procedural pages), `introduction` (section landing pages) - **Weight / ordering:** lower `weight` values appear first in navigation - **Internal link style:** relative paths ending in `/` (not `.md`) - **Section pages:** named `_index.md`; leaf pages are named `index.md` or `<slug>.md` - **Shared partials:** `docs/sources/tempo/shared/` contains headless snippets reused via Hugo `readfile` shortcode - **Admonitions:** `{{</* admonition type="note|caution|warning" */>}}` - **Public preview:** `{{</* docs/public-preview product="<PRODUCT>" */>}}` ## Conventions for agents - **Query language:** TraceQL (always capitalized, one word) - **Storage format versions:** don't hardcode the supported versions or the default — they change per release and the code is the source of truth. Check `tempodb/encoding/versioned.go`: `DefaultEncoding()` / `LatestEncoding()` (current default), `AllEncodings()` (readable versions), and `WritesSupported()` / `AllEncodingsForWrites()` (writeable versions). For the user-facing summary, refer to `docs/sources/tempo/configuration/parquet.md`. Style: lowercase "v", camelCase "Parquet", version digit (for example, `vParquet4`) - **Block format:** always "block format" or "Parquet block format" (not "block type") - **Product references:** "Grafana Tempo" on first mention, "Tempo" thereafter; always "Grafana Cloud" (never just "Cloud"); always "Grafana Alloy" (not "Alloy" alone on first mention) - **Vale / linter config:** prose linting uses the Grafana Writers' Toolkit `Grafana` style (with the Google, Hugo, and Readability packages). Refer to [Lint prose with the Vale linter](https://grafana.com/docs/writers-toolkit/review/lint-prose/). A local `.vale.ini` at the repo root configures these packages, but it's currently untracked (local only). Repositories with the `make docs` target also expose `make vale` (run from `docs/`) to lint `docs/sources/` ## Agent and skill resources | Location | Purpose | |----------|---------| | `.agents/guidance/coding.md` | Go coding standards | | `.agents/guidance/code-review.md` | Code review standards | | `.agents/guidance/precommit.md` | Pre-commit checklist | | `.claude/skills/shared/style-guide.md` | Docs style guide | | `.claude/skills/shared/release-notes-workflow.md` | Release notes process | | `.claude/skills/shared/release-notes-placement.md` | Changelog-to-release-notes placement | | `.claude/skills/shared/verification-checklist.md` | Docs verification | | `.claude/skills/shared/best-practices.md` | Docs best practices | | `.claude/skills/shared/docs-context-guide.md` | Docs context guide | | `.claude/skills/shared/load-context.md` | Context loading instructions | | `.claude/skills/shared/personas.md` | Persona and intent model | | `.claude/skills/shared/agent_personas.yaml` | Structured persona data | | `.claude/skills/` | Agent skills: docs-pr-check, docs-pr-write, docs-review, docs-workflow, persona-check, fix-vendor-conflicts, update-go-version | | `.github/instructions/docs/` | DOCS.md, release-notes.instructions.md, toolkit.instructions.md | ## Subsystem knowledge - **Metrics-generator:** code in `modules/generator/`; docs in `docs/sources/tempo/metrics-from-traces/metrics-generator/` and `docs/sources/tempo/reference-tempo-architecture/components/metrics-generator.md`; config validation in `modules/generator/processor/spanmetrics/`, `modules/generator/processor/servicegraphs/`, and `modules/generator/validation/fields.go` - **TraceQL engine:** `pkg/traceql/` — AST, parser, lexer, metrics engine - **Storage / tempodb:** `tempodb/` — block format, compaction, encoding; docs in `reference-tempo-architecture/` - **Components:** `modules/` subdirectories map to `reference-tempo-architecture/components/` doc pages - **Code-adjacent `AGENTS.md` files:** check for an `AGENTS.md` in the relevant directory before docs work that touches that area — repo root `AGENTS.md`, `modules/generator/AGENTS.md` (metrics-generator), and `docs/sources/tempo/configuration/AGENTS.md` (notes on the auto-generated config `manifest.md`) ## Gotchas Non-obvious facts that cause errors if you assume the obvious: 1. **The ingester is legacy.** The ingester is replaced in 3.0 by live-store and block-builder. The `modules/ingester/` directory was removed — do not reference or document it. Use `reference-tempo-architecture/` for current components. 2. **API parameters keep old names.** Some query parameters (for example, `mode=ingesters`) retain 2.x names while routing to new components. Check the code (for example, `modules/frontend/` query handlers) for the current mapping — code is the source of truth. 3. **Block format versions change between releases — verify against code, don't assume.** `tempodb/encoding/versioned.go` is the source of truth: which versions exist (`AllEncodings`), which are writeable (`WritesSupported`), and the default (`DefaultEncoding` / `LatestEncoding`). For the user-facing summary, refer to `docs/sources/tempo/configuration/parquet.md`. 4. **Helm chart docs live in three places:** - `docs/sources/helm-charts/tempo-distributed/` — primary procedures - `docs/sources/tempo/set-up-for-tracing/setup-tempo/deploy/kubernetes/helm-chart/` — landing page that links to the primary procedures - `docs/sources/tempo/configuration/` — storage, TLS, and network config that mentions Helm values ## Shared features across doc trees Features that require coordinated updates across the Tempo docs and Helm chart docs: - **Deployment configuration** — changes to Tempo config may affect Helm chart values and vice versa - **Release notes** — both `docs/sources/tempo/release-notes/` and `docs/sources/helm-charts/tempo-distributed/release-notes/` must be updated for major releases - **Upgrade instructions** — `set-up-for-tracing/setup-tempo/upgrade.md` may reference Helm-specific upgrade steps --- ### Design Proposals 2022 01 Metrics Generator (`docs/design-proposals/2022-01 Metrics-generator.md`) --- Authors: Koenraad Verheyden (@kvrhdn), Mario Rodriguez (@mapno) Created: December 2021 - January 2022 Last updated: 2022-02-07 --- # Metrics-generator ## Summary This design document describes adding a mechanism to Tempo that can generate metrics from ingested spans. To generate metrics we propose adding a new optional component: the _metrics-generator_. If present, the distributor will write received spans to both the ingester and the metrics-generator. The metrics-generator processes spans and writes metrics to a Prometheus datasource using the Prometheus remote write protocol. _Note: this feature is sometimes referred to as "server-side metrics". The Grafana Agent already supports these capabilities (to generate metrics from traces), in that context moving these processors from the Agent into Tempo moves them server-side._ _Note: this proposal describes an initial implementation of the metrics-generator. As such, some features will be marked as out-of-scope (for now). This implementation should not be deemed fully production-ready yet._ ### Table of Contents - [Architecture](#architecture) - [Comparison with the cortex and loki ruler](#comparison-with-the-cortex-and-loki-ruler) - [Components](#components) - [Distributor](#distributor) - [Metrics-generator](#metrics-generator) - [Metrics processors](#metrics-processors) - [Service graph](#service-graph) - [Span metrics](#span-metrics) - [Configuration](#configuration) ## Architecture Generating and writing metrics introduces a whole new domain to Tempo unlike any other functionality thus far. Instead of integrating this into an existing component, we propose adding a new component dedicated to working with metrics. This results in a clean division of responsibility and limits the blast radius from a metrics processors or the Prometheus remote write exporter blowing up. Alternatives considered: - integrate into the distributor: as some processors (i.e. service graph processor) have to process all spans of a trace, this would either require trace-aware load balancing to the distributor or an external store shared by all instances. This would complicate the deployment of the distributor and distract from its main responsibility. - integrate into the ingester: this is mostly rejected because the ingester is already a very complicated and critical component, adding additional responsibilities will further complicate this. Diagram of the ingress path with the new metrics-generator: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Comparison with the Cortex and Loki ruler The metrics-generator looks similar to the ruler in Cortex and Loki: both the ruler and the metrics-generator are optional components that can generate metrics and remote write them. The main difference with the metrics-generator is that the ruler uses a query engine to query the ingesters and backend. The metrics-generator does not query any other component but instead consumes the ingress stream directly. The Cortex and Loki ruler have a query engine powered by PromQL and LogQL respectively. Tempo does not have a query engine yet, so it's not possible yet to build a Tempo ruler. If at some point Tempo gets a query engine with similar capabilities, we can introduce a Tempo ruler and integrate it with the metrics-generator. A couple of notable differences between the Tempo metrics-generator and the Cortex/Loki ruler: - The metrics-generator has to consume the ingress stream. Because of this, the metrics-generator can only generate metrics about data that is being ingested. I.e. it's not possible to generate metrics from previously ingested data or to backfill metrics. - The metrics-generator uses fixed processors, these are less flexible than a rule which can contain a user-defined custom query. On the other hand, these processors can perform calculations which can't be expressed in a query language. The processing done by the service graph processor for instance will be difficult to express in a query. ## Components This section takes a more detailed look at the components involved in the path between ingesting traces and writing metrics. ### Distributor The distributor is the entrypoint for Tempo writes: it will receives batches of spans and forwards them to the ingesters (using replication if enabled). With the metrics-generator in the system, the distributor will now also have to write data to the metrics-generator. The distributor will first write data to ingesters and if this was successful it will push the same data to the metrics-generator. Writing to the metrics-generator is on a _best effort basis_: even if writing to the metrics-generator fails the Tempo write is still considered successful. This is a trade-off to keep request handling simple: if writing to the ingester succeeds but writing to the metrics-generator fails, the distributor should also revert the ingester write. The logic to discard already ingested data is deemed too complex. This tradeoff will result in missing or incomplete metrics whenever the metrics-generator is not able to ingest some data. As the client will not be aware of this, it will not resend the request. Failed writes should be reported with a metric on the distributor which can alert an operator (e.g. `distributor_metrics_generator_pushes_failures_total`). #### Metrics-generator ring The distributor has to find metrics-generator instances present in the cluster. When multiple instances of the metrics-generator are running, the distributor should load balance writes across these instances. Load balancing has to be trace-aware: all spans with the same trace ID should consistently be sent to the same instance. To achieve this we propose using the [dskit ring](https://github.com/grafana/dskit/tree/main/ring) backed by memberlist. This will be the same mechanism as used by the ingesters. The distributor will shard requests across metrics-generator instances based upon the tokens they own. _Out-of-scope_: in a later revision we can look into running the metrics-generators with a replication factor of 2 or higher. This is already supported by the ring, but will require extra logic to deduplicate metrics when exporting them (otherwise they are counted multiple times). This is out-of-scope for this design document. Initially the metrics-generator will run with RF=1 only. Note this will result in data loss when an instance crashes. #### gRPC protocol Similar to other Tempo components, inter-component requests are sent over gRPC. The existing APIs are defined in [`tempopb/tempo.proto`](../../pkg/tempopb/tempo.proto). Proposed service definition: ```protobuf service MetricsGenerator { rpc PushSpans(PushSpansRequest) returns (PushResponse) {}; } // Note: a PushSpansRequest should only contain spans that are relevant to the configured tenants // and processors. We can reduce the amount of data sent to the metrics-generator by trimming spans // and their metadata in the distributor. message PushSpansRequest { // Note: these are full traces. For further optimisation we should consider using a slimmer span // format containing the minimal amount of data. repeated tempopb.trace.v1.ResourceSpans batches = 1; } message PushResponse { } ``` Since the metrics-generator is directly in the write path, an increase in ingress will directly impact the metrics-generator. To reduce the amount of data sent from the distributor to the metrics-generator, the distributor should only send spans that are relevant for the configured metrics processors and tenants. If, for example, a processor only requires a subset of spans the distributor should drop not relevant spans before sending them. This should allow the metrics-generator to scale at a slower rate than the ingesters and saves bandwidth/processing time. This will require that the distributor is aware of the tenants and processors configured in the metrics-generator. This configuration will thus have to be shared with both components. ### Metrics-generator Diagram of what the metrics-generator could look like internally: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` #### Metrics processor Processors run inside the metrics-generator, they ingest span batches and keep track of metrics. To ensure isolation between tenants, the metrics processors are run per tenant and each tenant has their own configuration. It should be possible to (re)configure the processors without restarting the metrics-generator. Processes might build up some state as parts of a trace are received. This state will be kept in-memory and will be lost if the metrics-generator crashes. The implementation of the processors is described in more detail in [Metrics processors](#metrics-processors) _Out-of-scope_: persist the state of the processors to minimize data loss during a crash. #### Metrics processor configuration Configuration has to be reloadable at run-time. Tempo already uses the overrides to configure limits dynamically. The metrics-generator can piggyback on this system and read per-tenant configuration from the overrides. _Out-of-scope_: add a management API to configure the processors for a tenant. This should allow tenants to configure their own processors using a command line tool (similar to [cortextool](https://github.com/grafana/cortex-tools#cortextool)). Configuration would be written to and read from a bucket. Before this can be implemented, limits should be in place to protect both the Tempo cluster and the metrics database against excessive metrics or high cardinality. The flow of such a request would look like: ``` ┌────────────┐ ┌──────────────────────┐ ┌────────────┐ │ Client │─────Manage ────▶│ Metrics-generator │────Store/fetch ────▶│ Bucket │ └────────────┘ processors └──────────────────────┘ config └────────────┘ ``` #### Metrics collector & Prometheus remote write The metrics collector is a little process within the metrics-generator that on regular intervals collects metric samples from the processors. The samples are then written to a time series database using the Prometheus remote write protocol. The collector should work similar to a Prometheus instance scraping a host. So it should be possible to configure the collection interval and add external labels. When Tempo is run in multi-tenant mode, the `X-Scope-OrgID` header used to identify a tenant will be forwarded to the Prometheus-compatible backend. _Potential future feature_: also support writing OTLP metrics. ## Metrics processors The metrics processors are at the core of the metrics-generator, they are responsible for converting trace data into metrics. This initial proposal describes two processors that already exist in the Grafana Agent: the service graph processor and the span metrics processor. The implementation of a processor should be flexible enough, so it's easy to add additional processors at a later stage. ### Service graph _Note: this processor also exist in the Grafana Agent. Ideally the metrics exported by Tempo match exactly with the metrics from the Agent so a frontend (e.g. Grafana) can work with both._ A service graph is a visual representation of the interrelationships between various services. The service graph processor will analyse trace data and generate metrics describing the relationship between the services. These metrics can be used by e.g. Grafana to draw a service graph. The goal is to mirror the implementation from the Grafana Agent. Service graphs are described [here](https://grafana.com/docs/tempo/next/grafana-agent/service-graphs/). The [Agent code lives here](https://github.com/grafana/agent/tree/release-v0.21.2/pkg/traces/servicegraphprocessor). The service graph processor builds its metadata by analysing edges in the trace: an edge is two spans with a parent-child relationship of which the parent span has [SpanKind](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#spankind) `client` and the child span has SpanKind `server`. Each edge represents a request from one service to another. The amount of requests and their duration are recorded in metrics. The following metrics should be exported: | Metric | Type | Labels | Description | |---------------------------------------------|-----------|----------------|--------------------------------------------------------------| | traces_service_graph_request_total | Counter | client, server | Total count of requests between two nodes | | traces_service_graph_request_failed_total | Counter | client, server | Total count of failed requests between two nodes | | traces_service_graph_request_server_seconds | Histogram | client, server | Time for a request between two nodes as seen from the server | | traces_service_graph_request_client_seconds | Histogram | client, server | Time for a request between two nodes as seen from the client | | traces_service_graph_unpaired_spans_total | Counter | client, server | Total count of unpaired spans | | traces_service_graph_dropped_spans_total | Counter | client, server | Total count of dropped spans | Since the service graph processor has to process both sides of an edge, it needs to process all spans of a trace to function properly. If spans of a trace are spread out over multiple instances it will not be possible to pair up spans reliably. The following aspects should be configurable: - `success_codes`: the status code considered a successful request, this is used to distinguish between successful and failed requests. - `buckets`: the buckets to use for the histogram. ### Span metrics The span metrics processor aggregates request, error and duration metrics (RED) from span data. The goal is to mirror the implementation from the OpenTelemetry Collector. Span metrics are described [here](https://grafana.com/docs/tempo/next/grafana-agent/span-metrics/). The [code lives here](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/spanmetricsprocessor). The span metrics processor will compute the total count and the duration of spans for every unique combination of dimensions. Dimensions can be the service name, the operation, the span kind, the status code and any tag or attribute present in the span. The more dimensions are enabled, the higher the cardinality of the generated metrics. The following metrics should be exported: | Metric | Type | Labels | Description | |--------------------------------------|-----------|------------|-------------------------| | traces_spanmetrics_duration_seconds | Histogram | Dimensions | Duration of the span | | traces_spanmetrics_calls_total | Counter | Dimensions | Total count of the span | The following aspects should be configurable: - `dimensions`: the labels to include in the generated metrics, each dimension must match with an attribute of the span. - `buckets`: the buckets to use for the histogram. - `include`/`exclude`: filter which spans to include when generating span metrics. It should be possible to only generate span metrics for a subet of services for instance. ## Configuration Example of what the configuration of the distributor and the metrics-generator could look like: _Note: this is just a proposal, the final configuration can be found in the documentation._ ```yaml distributor: # Toggle to enable or disable the metrics-generator ring. If disabled, the distributor should # not initialize the metrics-generator ring and does not send data to the metrics-generator. enable_metrics_generator_ring: true # Similar to the ingester_client, configure the client used by the distributor metrics_generator_client: # Same settings as ingester_client metrics_generator: collection_interval: 15s external_labels: some_static_label: foo # Global settings for the metrics processors processor: service_graphs: histogram_buckets: [0.1, 0.2, 0.5, 1, 2, 5, 10] span_metrics: dimensions: - http.method - http.target # Configure remote write target remote_write: enabled: true client: # prometheus.RemoteWriteConfig # https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write url: http://prometheus:9090/prometheus/api/v1/write ``` Example of what the overrides could look like: ```yaml overrides: 1: metrics_generator_processors: - service-graphs - span-metrics 2: metrics_generator_processors: - service-graphs ``` --- ### Design Proposals 2022 04 Parquet (`docs/design-proposals/2022-04 Parquet.md`) --- Authors: Martin Disibio (@mdisibio), Annanay Agarwal (@annanay25) Created: April 2022 Last updated: 2022-04-22 --- # Parquet ## Summary This design document describes adding a new columnar block format to Tempo based on Apache Parquet. A columnar block format has many advantages within Tempo such as enabling faster search but also downstream by enabling a large ecosystem of tools access to the underlying trace data. ## Context Over the last few months, the Tempo Team has invested a lot of effort in implementing efficient search queries over trace data spanning long time ranges. We mainly worked on two major initiatives to enable search and summarise our findings below: 1. **Scaling up the existing Protobuf-based blocks**. Tempo currently leverages massively-parallel serverless functions to perform full backend-search. However even with thousands of functions, Tempo search is not able to achieve the search speeds that we would like on larger datasets. Each trace is stored as an individual proto message and must be unmarshaled to look for things like service names and tags. Even with scanning speeds of 60 gb/s from object storage, the large I/O makes it infeasible to query >24h periods over larger datasets. 2. **Implementing a Flatbuffer-based storage format**. Tempo currently uses flatbuffer-based data to power ingester search. Flatbuffers improve search speeds by incurring no deserialization cost as the on-disk format matches the in-memory format. However, the current ingester data is very limited to the current application and stores only a subset of searchable data. Storing a 100% roundtrippable version of traces proved to be ~50% larger than the existing block size and is not feasible. This is due to the design of flatbuffers for alignment of various values, padding, and inability to use space-efficient data encodings such as variable-width integers. ## Design Goals and Requirements The main design goals in choosing a new block format are to increase the speed and efficiency of Tempo search, and power the upcoming TraceQL language for querying and extracting metrics from traces. We also had the following requirements of the new format: * Roundtrippable with OTLP - A new block format must support full trace read/write, so it must be able to be converted from OTLP and back again. * More efficient search - Reduce i/o * Faster search - There's no point in having a new format that is slower * Similar or better block size - No significant increase in storage requirements for the same data * Object storage - Works with object storage as the only dependency * Shardable - Ability to scan a block by multiple queriers concurrently, with no upper limit (i.e. 1000's of workers) ### Why parquet Parquet fits all of the requirements: * Roundtrippable - Support for hiearchical data means regenerating OTLP is straightformat (more on the nested schema below) * More efficient - only read the columns needed for the search * Faster - only read the columns needed * Similar or better block size - Current prototype has ~5% smaller block size for parquet * Object storage - Parquet is a file-based approach which translates well to Tempo's block-based design * Shardable - Parquet naturally includes sub-block structures such as row groups and column chunks which can be processed independently. ## Schema ### Fully Nested vs. Span-oriented There are two overall approaches to a columnar schema: fully nested or span-oriented. Span-oriented means a flattened schema where traces are destructured into rows of spans. A fully nested schema means the current trace structures such as Resource/InstrumentationLibrary/Spans/Events are preserved (nested data is natively supported in Parquet). In both cases individual leaf values such as span name and duration are individual columns. We chose the nested schema for several reasons. (a) The block size is much smaller for the nested schema. This is due to the high data duplication incurred when flattening resource-level attributes such as service.name to each individual span. (b) A flat schema is not truly "flat" because each span still contains nested data such as attributes and events. (c) Nested schema is much faster to search for resource-level attributes because the resource-level columns are very small (1 row for each batch) (d) Translation to/from OTLP is very straightforward (e) Easily add computed columns (ex: Trace duration) at multiple levels such as per-trace, per-batch, etc. ### Static vs Dynamic columns Additionally there is another layer to the schema which is dynamic vs static columns. A dynamic schema means storing each attribute such as "service.name" and "http.status_code" as its own column, and the columns in each parquet file can be different. A static schema means it is irresponsive to the shape of the data, and all attributes are stored in generic key/value containers. The dynamic schema is the ultimate "dream" for a columnar format but it is too complex for a first release. However the benefits of that approach are also too good to pass up, so we propose a hybrid approach. It is primarily a static schema but with some dynamic columns extracted from trace data based on some heuristics of frequently queried attributes. We plan to continue investing in this direction to implement a fully dynamic schema where trace attributes are blown out into independent parquet columns at runtime. ### Proposed Schema Here is the proposed parquet schema. It is mainly a directly transation of OTLP but with some key differences. We will discuss details and rationale of several areas below: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### Trace-level attributes For speed and ease-of-use, we are projecting several values to columns at the trace-level: * Trace ID - Don't store on each span. It is a UTF-8 hexadecimal string for user-friendliness. We could save a few bytes by storing the underlying 16 bytes, but this makes it easier to work with for downstream applications. * Root service/span names/StartTimeUnixNano - These are selected properties of the root span in each trace (if there is one). These are used for display of results in the Grafana UI. Computed at ingest time and stored once for efficiency, so we don't have to find the root span. * DurationNanos - The total trace duration, computed at ingest time. This powers the min/max duration filtering in the current Tempo search and is more efficient than scanning the spans duration column. However, it may go away with TraceQL or we could decide to change it to span-level duration filtering too. ### Dedicated columns Projecting attributes to their own columns has massive benefits for speed and size, and these are too good to pass up on. Therefore we are taking an opinionated approach and projecting some common attributes to their own columns. All other attributes are stored in the generic key/value maps and are still searchable, but not as quickly. We chose these attributes based on what we commonly use ourselves (scratching our own itch), but we think they will be useful to most workloads. There is an associated cost for unused attributes as it will still store a column of nulls, however the cost should be minimal. We can be more generous with unused resource-level attributes because their overhead is the smallest of all, we are finding it to be ~.05% (0.0005) of the total block size. Resource-level: * `service.name` * `cluster` and `k8s.cluster.name` * `namespace` and `k8s.namespace.name` * `pod` and `k8s.pod.name` * `container` and `k8s.container.name` Span-level * `http.method` * `http.url` * `http.status_code` (int) ### "Any"-type Attributes OTLP attributes have variable data types, which is easy to accomplish in formats like protocol-buffers, but does not translate directly to Parquet. Each column must have a concrete type. There are several possibilities here but we chose to have an optional values for each concrete type. We aren't 100% sold on this approach and open to feedback. Array and KeyValueList types are stored as JSON-encoded strings. The data is portable and still searchable at extremely basic levels. ``` repeated group Attrs { required binary Key (STRING); # Only one of these will be set optional binary Value (STRING); optional boolean ValueBool; optional double ValueDouble; optional int64 ValueInt (INT(64,true)); optional binary ValueArray (STRING); optional binary ValueKVList (STRING); } ``` Pros: * Preserves the functionality of parquet column min/max statistics which can be leveraged during search * Native data types should be more efficient to store and process Cons: * Many nulls - as each attribute only populates 1 column, the other 5 are guaranteed to be null. This has a non-trivial impact on storage size. ### Event Attributes Span event attributes are stored as JSON-encoded strings in a generic key/value map. This is by far the most space-efficient encoding and the trade-off of decreased searchability seems worthwhile. Storing event attributes this way reduces the block size ~16% for our dataset, which is huge. There are currently no use cases to search event attributes, but we can revisit this in the future if needed. ``` repeated group Attrs { required binary Key (STRING); required binary Value (STRING); } ``` ### Compression and Encoding Parquet has robust support for many compression algorithms and data encodings. We have found excellent combinations of storage size and performance with the following: 1. LZ4 Compression - Enable on all columns 2. Dictionary encoding - Enable on all string columns (including byte array ParentSpanID). Most strings are very repetitive so this works well to optimize storage size. However we can greatly speed up search by inspecting the dictionary first and eliminating pages with no matches. 3. Time and duration unix nanos - Delta encoding 4. Rarely used columns such as DroppedAttributesCount - These columns are usually all zeroes, RLE works well. ### Bloom Filters Parquet has native support for bloom filters however we are not using them at this time. Tempo already has sophisticated support for sharding and caching bloom filters, and we will continue to leverage that for now. ## Results from Local Testing Here are some interesting column sizes from a block of size ~600MB containing ~150K traces: ``` column.Trace.DurationNanos 154414 values 0.64 MB column.Trace.ResourceSpans.Resource.ServiceName 2672245 values 2.01 MB column.Trace.ResourceSpans.InstrumentationLibrarySpans.Spans.HttpStatusCode 10428415 values 6.35 MB column.Trace.ResourceSpans.InstrumentationLibrarySpans.Spans.ID 10428415 values 82.37 MB # this is never used! ``` Some super early benchmarks on local SSDs: * Searching protobuf data for a simple query `cluster=ops and minDuration=1s` ``` === RUN TestSearchProto Traces : 55 Traces inspected: 154414 --- PASS: TestSearchProto (21.11s) ``` * Same query on Parquet blocks ``` === RUN TestSearchPipelineStatic Traces : 55 Traces inspected: 154414 Reads : 290 6.65 MB --- PASS: TestSearchPipelineStatic (0.18s) ``` ## Implementation We have been refactoring Tempo recently in anticipation of this, therefore Parquet-formatted blocks should be as easy as creating a new folder `/tempodb/encoding/vparquet` and implementing the `VersionedEncoding` interface. Tempo's parquet support will be based on the new library https://github.com/segmentio/parquet-go which we have used for all prototyping and have had excellent success with. ### Write path * Creating blocks: The ingester will write and flush parquet-formatted blocks. The compactor will only compact like-encoded-blocks, i.e. parquet to parquet. This seems better than having the ingester continue to flush proto and the compactor to convert them. * One tunable is the row group size. Our index pages target something like 256K, but row groups are typically much larger, 50-100MB. Not sure about the best value here, will need to experiment. ### Read path * Trace Search: We will implement a query engine to search over the columns involved in the search and join results in memory to output valid traces matching all query parameters. * Trace Lookup: Our current index pages will be replaced by scanning over the `TraceID` column in the Parquet block. This will provide us with the corresponding row number that matches the queried ID. * Concurrent search: We will shard searches based on RowGroups and they will work similarly to how we shard on index pages. The metadata for each block will know how many there are and the query-frontend can used them to divvy up tasks, and then the queriers can jump directly to them. --- ### Design Proposals 2022 04 TraceQL Concepts (`docs/design-proposals/2022-04 TraceQL Concepts.md`) --- Authors: Yuri Shkuro (@yurishkuro), Tom Wilkie (@tomwilkie), Cyril Tovena (@cyriltovena), Martin Disibio (@mdisibio), Joe Elliott (@joe-elliott) Created: 2021 October - 2022 March Last updated: 2022-04-12 --- # TraceQL Concepts ## Summary This design document describes a language for selecting traces that Tempo will implement. This language is currently only focused on trace selection. It will be extended to derive metrics from traces in the future. This document is **not** meant to be a complete language specification. Rather it is meant to invite the community to comment on the basic structure and concepts of the language. ### Table of Contents - [Capabilities](#capabilities) - [Structure](#structure) - [Selecting spans](#selecting-spans) - [Intrinsic fields](#intrinsic-fields) - [Attribute fields](#attribute-fields) - [Field expressions](#field-expressions) - [Combining Spansets](#combining-spansets) - [Structural Operators](#structural-operators) - [Aggregators](#aggregators) - [Pipelining](#pipelining) - [Grouping](#grouping) - [Examples](#examples) - [Shorthand](#shorthand) ## Capabilities A TraceQL query can select traces based on - span attributes, timing and duration - structural relationships between spans - aggregated data from the spans in a trace The TraceQL language is designed to use the same syntax and semantics as PromQL and LogQL where possible. The syntax and semantics differ due to the specialized needs of querying traces. ## Structure A query is an expression that is evaluated on one trace at a time. The query is structured as a set of chained expressions (a pipeline). Each expression in the pipeline selects or discards spansets from being included in the results set. E.g. `{ .http.status = 200 } | by(.namespace) | count() > 3` Queries select sets of spans and filter them through a pipeline of aggregators and conditions. If a spanset is produced after evaluation on a trace then this spanset (and by extension the trace) is included in the resultset of the query. ## Selecting spans In TraceQL the curly brackets `{}` always select a set of spans from the current trace. They will commonly be paired with a condition to reduce the spans being passed in. `{ .http.status = 200 }` This simple query will be evaluated on every span of every trace, one at a time. If the trace being evaluated contains no spans with an attribute `http.status` with the value `200` then no spans will be selected and this trace will not appear in our resultset. If the trace does contain spans with an attribute `http.status` with the value `200` then only those spans will be returned. The trace is reduced to only the set of spans that match the condition inside the `{}`. The resultset will contain only this subset of spans matching the condition. ### Intrinsic fields Every span has certain intrinsic fields that can be referenced when selecting spans | field name | description | | ---------- | ------------------------------------------- | | `duration` | end - start time of the span | | `name` | operation or span name | | `status` | status values are error, ok, or unset | | `parent` | the parent of this span | **Examples** Find traces that contain spans whose duration is greater than 2 seconds: `{ duration > 2s }` Find traces that contain a span named "HTTP POST": `{ name = "HTTP POST" }` ### Attribute fields We can refer to dynamic attributes (also known as tags) on the span or the span's resource. **Examples** Find traces with the GET http method: `{ .http.method = "GET" }` Find traces that passed through the prod namespace: `{ .namespace = "prod" }` Find traces that cross service boundaries: `{ parent.service.name != .service.name }` ### Scoped attribute fields Attributes can be specifically scoped to either "span" or "resource". Specifying "span" or "resource" can result in significant performance benefits. `{ span.http.status = 200 }` `{ resource.namespace = "prod" }` ### Field expressions Fields can also be combined in various expected ways. **Examples** Find traces where the difference of the duration of a set of parent/child spans exceeds 500ms: `{ parent.duration - duration > 500ms }` Find traces with "success" http status codes: `{ .http.status >= 200 && .http.status < 300 }` Note that the second expression requires both conditions to be true on the same span. The entire expression inside of `{}` must be evaluated as true on a single span for it to be included in the resultset. ## Combining Spansets Logical operators combine sets of spans. For example, if we wanted to find a trace that went through two specific regions: `{ .region = "eu-west-0" } && { .region = "eu-west-1" } ` Note the difference between the above and the following: `{ .region = "eu-west-0" && .region = "eu-west-1" } ` The second expression will return no traces because it's impossible for both conditions to be simultaneously true on the same span. ### Structural operators Structural operators evaluate traces based on the span relationships within the span tree. `{ } >> { }` This is the descendant operator. The spans returned from this operator will match the right hand side conditions while also being descendants of spans that match the left hand side conditions. `{ } > { }` This is the child operator. The spans returned from this operator will match the right hand side conditions while also being children of spans that match the left hand side conditions. `{ } ~ { }` This is the sibling operator. The spans returned from this operator will match the right hand side conditions while also being siblings of spans that match the left hand side conditions. ## Aggregators All of the above expressions involve asking questions about individual spans. However, sometimes we want to ask questions about a set of spans. For that we have aggregate functions. **Examples** Find traces where the total number of spans is greater than 10: `count() > 10` Find traces where the average duration is greater than 1s: `avg(duration) > 1s` ## Expression Pipelining Pipelining allows us to "pipe" a set of spans from one expression to the next. This is particularly useful if you want to perform an aggregate over a subset of a trace. **Examples** Find traces that have more than 3 spans with an attribute `http.status` with a value of `200`: `{ .http.status = 200 } | count() > 3` ## Grouping Grouping allows us to take a trace and break it down into sets of spans that are evaluated by future pipeline entries. Each set of spans created by the group is _individually_ evaluated by downstream expressions. **Examples** Find traces that have more than 5 spans in any region: `by(.region) | count() > 5 ` ## Examples Any span matches an attribute: `{ .namespace = "prod" }` Two attributes appear on the same span: `{ .namespace = "prod" && .http.status = 200 }` Two attributes appear anywhere within a trace: `{ .namespace = "prod" } && { .http.status = 200 }` Any span has a duration over one second: `{ duration > 1s }` The trace as a whole has a duration of over one second: `max(end) - min(start) > 1s` The average duration of spans exceeds 1s and any span has a specific attribute: `avg(duration) > 1s && { .namespace = "prod" }` A trace has over 5 spans with http.status = 200 in any given namespace: `{ .http.status = 200 } | by(.namespace) | count() > 5` A trace passed through two regions in a specific order: `{ .region = "eu-west-0" } >> { .region = "eu-west-1" }` A trace sees network latency > 1s when passing between any two services: `{ parent.service.name != .service.name } | max(parent.duration - duration) > 1s` --- ### Design Proposals 2023 05 VParquet3 (`docs/design-proposals/2023-05 vParquet3.md`) --- Authors: Adrian Stoewer (@stoewer), Mario Rodriguez (@mapno) Created: May 2023 Last updated: 2023-05-31 --- # vParquet3 ## Summary This document describes how to extend Tempo’s Parquet based block format such that a configurable subset of span attributes can be stored in dedicated columns. The goal is to improve the search performance of TraceQL in Tempo. ### Table of Contents * [Context](#context) * [Goals and requirements](#goals-and-requirements) * [Implementation](#implementation) * [Schema with spare columns](#schema-with-spare-columns) * [Configuration and overrides](#configuration-and-overrides) * [Compaction](#compaction) * [Tooling](#tooling) ## Context The current Parquet schema stores span attributes in the repeated group `Attrs`. The group contains a single column for all attribute keys and separate columns for the corresponding values (depending on the value type). The size of the span attribute columns combined contribute to 30-70% of the overall block size. To search for a specific attribute, it is necessary to access this large chunk of data. The analysis of Tempo blocks from different use cases has further revealed that the 10 most frequently used span attributes are responsible for over 50% of the overall attribute size. A benchmark with a schema that stores the 5 most frequently occurring attributes in dedicated columns showed significant performance improvements: * The block size decreased by ~10% * Compressed size of the generic attribute columns was reduced by ~40% * The search speed for queries without matches improved by ~80% for attributes with dedicated columns and ~70% for generic attributes columns * The search speed for queries returning matches improved by ~50% for attributes with dedicated columns and ~30% for generic attribute columns The amount and kind of span attributes varies a lot between use cases and the above benchmark was conducted with tracing data that captured a lot of span attributes. Therefore, the search speed improvements that can be achieved by storing span attributes in dedicated columns, can deviate from the above results. However, the data still shows that dedicated columns for span attributes that can be tuned to a specific use case can improve Tempo’s search performance significantly. This document describes how dedicated columns for span attributes will be implemented in Tempo. ## Goals and requirements The main goal for the next iteration of the block format `vParquet3` is to be able to manually configure dedicated attribute columns for very heavy workloads. To keep the scope small only the following features are implemented for now: * Only attributes with string values can be promoted to dedicated columns on resource and span level * Attributes promoted to dedicated columns can be manually configured per tenant * Support for changing dedicated column configurations * Compactor treats different configs as different encodings * Queriers read blocks with different configured columns * Provide tooling and documentation that enables tenants to find good attributes to promote to dedicated columns * Find attributes that contribute the most to block size * (Stretch) Find the most common used attributes in search Future iterations may support additional features: * Workload analysis to determine suitable dedicated column configurations which can be used to make suggestions or to automate the configuration * Fully dynamic schema that allows to add dedicated columns for different data types and encodings (e.g. delta or run length encoding) * Apply config dynamically to old blocks during compaction ## Implementation ### Schema with spare columns There are two approaches for storing span attributes in dedicated columns in Tempo. One possibility is to extend the Parquet schema dynamically and add the required extra columns at runtime. This approach would allow to configure the number, type, and encoding of dedicated attribute columns and therefore allow Tempo to optimize the block layout specifically for every use case. Ultimately this is what Tempo should be capable of in the future. However, handling dynamically changing Parquet schemata correctly adds a lot of complexity. The alternative approach is to keep a set of spare columns in the schema and dynamically assign them to the respective span attributes. The downside here is that the number, type, and encoding for dedicated span attributes is very restricted. To keep the added complexity manageable, dedicated columns for span attributes are implemented using the spare columns approach for now. In the analyzed use cases, the 20 span attributes that contributed most to the overall attribute size, were almost exclusively attributes with string values. Given that the benchmark achieved good results with 5 dedicated columns, it can be expected that extending the schema by 10 spare columns for string values is sufficient to achieve good performance improvements for most use cases. The following code outlines the schema changes: ```go type DedicatedAttrs struct { String01 *string `parquet:",snappy,optional,dict"` String02 *string `parquet:",snappy,optional,dict"` ... String10 *string `parquet:",snappy,optional,dict"` } type Span struct { ... DedicatedAttrs DedicatedAttrs `parquet:""` } ``` To execute TraceQL queries it is necessary to know which attribute is assigned to which spare column. Since the assignment of spare columns to specific attributes changes over time, it is necessary to store this information for each block. Tempo stores metadata about each Block in the `meta.json` file. The schema of this file will be extended such that it contains the assignment information: ```go { ... "dedicatedColumns": [ {"scope": "span", "name": "attrA", "type": "string"}, {"scope": "span", "name": "attrB", "type": "string"}, {"scope": "span", "name": "attrC", "type": "string"}, ] } ``` ## Configuration and overrides Dedicated columns have been part of the Parquet-based formats since the first version. As we gained operational knowledge, it became clear that the hardcoded set of attributes selected for dedicated columns didn’t work for all workloads. The biggest improvement of vParquet3 compared to older block formats is that the attributes that can be stored in dedicated columns are partially configurable. And, it only makes sense that they’re configured per tenant. Another important consideration is that dedicated columns need to be configured at runtime. It is expected that selected attributes will change over time, as more data is ingested, new query patterns appear,new attributes are used, etc. This would also allow for smoother operations, as a change in configuration doesn’t require a restart of Tempo. For these reasons, the configuration will be added to the storage / block section of Tempo’s yaml configuration file as well as to the override’s configuration: ```go type Limits struct { ... DedicatedColumns []sharedconfig.DedicatedColumn `yaml:"dedicated_columns"` } ``` Dedicated columns will be configured as an array of definitions of columns, where basic information about the selected attribute that is defined. Required configurations are the scope of the attribute, its name and its type. ```go type DedicatedColumn struct { Scope string `yaml:"scope"` // can be 'span' or 'resource' Attribute string `yaml:"attribute"` Type string `yaml:"type"` // for now only 'string' is supported } ``` Additional config options can be added later to configure extra parameters such as encodings (e.g. dictionaries, lists, delta encoding), compression format (e.g. snappy). ## Compaction When blocks are compacted the spare column assignment has to be taken into consideration: In cases where blocks have different column assignments, values need to be transferred from one column into another. Although it is possible to implement compaction across different column assignments, this is not strictly necessary: The assignment of span attributes to spare columns is not expected to change frequently therefore not compacting blocks with different assignments does not have a large impact on the compaction result. To handle compaction correctly the compaction block selector is adjusted such that blocks with different spare column assignments are not selected for compaction. <img src="./assets/vparquet3-compaction.jpg"> ## Tooling A key factor in the performance of dedicated columns is the attributes that are used for them. We want to include tools that help operators find good candidates for dedicated columns. What makes an attribute a good candidate will most likely vary from user to user, but there are some basic rules that can help us find them. From our analysis of Tempo blocks, we’ve found that attributes that occupy the most space in a block usually have the biggest impact on performance when promoted to dedicated columns. Even when queries don’t search by those attributes. A lot of work to help analyze parquet blocks has already been done in [parquet-cli](https://github.com/stoewer/parquet-cli). With this tool, we can inspect individual blocks and get aggregated data for the attributes present in a block, and the size that each represents in the generic `attr` columns. We can combine the capabilities of this tool with tempo-cli, which has better integration with Tempo backends. This way, we could get aggregated data for all the blocks of a tenant present in a Tempo backend storage. (Stretch) Another method of finding good candidates for dedicated columns is by analyzing the read path. An attribute that is queried by significantly more than the rest is a potential good candidate. _Hot_ attributes can be found by extending metrics in the read path to record query patterns and extract frequently queried attributes. --- ### Design Proposals 2023 11 TraceQL Extensions (`docs/design-proposals/2023-11 TraceQL Extensions.md`) --- Authors: Martin Disibio (@mdisibio), Joe Elliott (@joe-elliott), Jennie Pham (@ie-pham), Adrian Stoewer (@stoewer) Created: 2023 July - 2023 November Last updated: 2023-11-28 --- # TraceQL Extensions ## Summary This design document describes extensions to the existing TraceQL language that allow for searching new fields and data types. This document is **not** meant to be a complete language specification. Rather it is meant to invite the community to comment on the basic structure and concepts of the language extensions. ### Table of Contents - [Escaping Attribute Names](#escaping-attribute-names) - [Additional Scopes](#additional-scopes) - [Trace](#trace) - [Instrumentation Scope](#instrumentation-scope) - [Event](#event) - [Link](#link) - [Scoped Intrinsics](#scoped-intrinsics) - [Additional Data Types](#additional-data-types) - [Arrays](#arrays) - [IDs](#ids) - [Future Considerations](#future-considerations) ## Escaping Attribute Names OpenTelemetry supports [any valid Unicode sequence](https://opentelemetry.io/docs/specs/otel/common/attribute-naming/) for attribute names. This includes characters problematic for a query language such as whitespace, mathematical symbols, various braces, etc. Previously TraceQL did not support [these problematic characters](https://github.com/grafana/tempo/blob/c00e7ef1103a8e7804b6f71fe8e90d0a2d9a9b02/pkg/traceql/lexer.go#L344) in attributes names. TraceQL will be extended to support these characters by allowing attribute names wrapped in `"` to be unparsed. `{ span."attribute with spaces" = "foo"}` TraceQL will also support the escape sequences `\\` and `\"` in an attribute name. `{ span."this is \"bad\"" = "foo"}` ## Additional Scopes We have added four additional scopes to the TraceQL language: [trace](#trace), [scope](#instrumentation-scope), [event](#event), and [link](#link). See below for details. ### Trace The trace scope allows a user to access intrinsic fields that exist at the trace level. There are no attributes at the trace scope. See [intrinsics](#scoped-intrinsics) for a complete list of allowed trace level intrinsics. ``` { trace:duration > 100ms } ``` ### Instrumentation Scope The [Instrumentation Scope](https://github.com/open-telemetry/opentelemetry-proto/blob/ea449ae0e9b282f96ec12a09e796dbb3d390ed4f/opentelemetry/proto/common/v1/common.proto#L71) object contains information about the instrumentation used to generate telemetry data. This scope contains new intrinsics as well as attributes. It will be referenced using the `scope` keyword. ``` { scope.foo = "bar" } { scope:name ~= ".*Java.*" } ``` ### Event Span [events](https://github.com/open-telemetry/opentelemetry-proto/blob/ea449ae0e9b282f96ec12a09e796dbb3d390ed4f/opentelemetry/proto/trace/v1/trace.proto#L213) contain attributes as well as intrinsic properties that describe discrete events that occur during a span's lifetime. Note that both events and [links](#link) are different than existing scopes because there can be many events to one span. Due to this we are being purposefully conservative when defining event scopes and intend to extend functionality as we better understand use cases. See [future considerations](#future-considerations) for additional discussion. ``` { event.exception.message =~ ".*Division by zero.*" } { event:name = "exception" } ``` In the above cases the operators will return true if there are any events on the span that meets the specified condition. ### Link Span [links](https://github.com/open-telemetry/opentelemetry-proto/blob/ea449ae0e9b282f96ec12a09e796dbb3d390ed4f/opentelemetry/proto/trace/v1/trace.proto#L242) contain attributes as well as intrinsic properties that can create connections to other spans. Note that both [events](#event) and links are different than existing scopes because there can be many events to one span. Due to this we are being purposefully conservative when defining event scopes and intend to extend functionality as we better understand use cases. See [future considerations](#future-considerations) for additional discussion. ``` { link.foo = "bar" } { link:traceID = "<hex string>" } ``` In the above cases the operators will return true if there are any links on the span that meets the specified condition. ## Scoped Intrinsics TraceQL supports a number of intrinsics that will continue to exist in the language. These will be referred to as "Legacy Intrinsics" and there are currently no plans to remove them. However, due to the increasing number of [scopes](#additional-scopes) we will provide scoped names for all legacy intrinsics and all future intrinsics will be scoped only. Scoped intrinsics will always use a `:` to separate the scope from the intrinsic to differentiate between an attribute of the same name. i.e. `span.name` accesses an attribute "name" on the span vs `span:name` which accesses the intrinsic span name. Legacy intrinsics will be kept for the foreseeable future to preserve backwards compatibility, but we will not be adding any new unscoped intrinsics. | Scope | Intrinsic | Legacy | Description | | ------ | -------------- | ------ | ----------- | | trace | :duration | traceDuration | Max end time of any span minus the min start time of any span in the trace. | | | :id | | Trace id | | | :rootName | rootName | `span:name` of the root span | | | :rootService | rootServiceName | `resource.service.name` of the root span | | scope | :name | | Instrumentation scope name | | | :version | | Instrumentation scope version | | span | :id | | Span id | | | :name | name | Span name | | | :duration | duration | Span duration | | | :status | status | Span status | | | :statusMessage | statusMessage | Span status | | | :kind | kind | Span kind | | event | :name | | Event name | | link | :spanID | | Link's span id | | | :traceID | | Link's trace id | | parent | :id | | Parent's `span:id` | ## Additional Data Types ### Arrays TraceQL will support [array](https://github.com/open-telemetry/opentelemetry-proto/blob/ea449ae0e9b282f96ec12a09e796dbb3d390ed4f/opentelemetry/proto/common/v1/common.proto#L44) attribute types with the `[]` syntax. To access a specific element of an array use a 0-based array syntax. ``` { span.http.response.header.content-type[0] = "application/json" } ``` To test all elements of an array use empty square brackets. This will evaluate to true if any of the elements in the array `http.response.header.content-type` are equal to the value `"application/json"`. ``` { span.http.response.header.content-type[] = "application/json" } ``` Note that currently there is no array literal or operations defined on the array data type itself. For now we will limit ourselves to existing operations on the elements of the array using the above syntax. ### IDs The new [scoped intrinsics](#scoped-intrinsics) section above defines a few fields with an "id" data type such as `span:id`. The id type is only comparable to hex strings and only supports the operations `=` and `!=`. ``` { span.id = "8bf5306cb6a28" } ``` Span IDs and trace IDs are always 64 bit or 128 bit. However, for the purposes of comparison, leading 0s will be ignored. The following are equivalent. ``` { trace.id = "0007f2b8d1c69375e0d46a9cf8072bc4" } { trace.id = "7f2b8d1c69375e0d46a9cf8072bc4" } ``` ## Future Considerations [Links](#link) and [events](#event) have been introduced with purposefully limited syntax. We think it wise to not commit to language features that we will regret later and want the community to experiment with basic access to these fields and communicate their needs so we can continue to move the language forward in a way that makes sense for everyone. It would be very easy to unnecessarily over complicate the language and we'd prefer taking a slower approach. Note that currently you can only assert whether there are any links or events with specific field values on a span. We believe this covers a huge range of use cases, but also leaves some obvious gaps. As you are exploring links and events consider how you would like to express more complex relationships such as asserting two conditions on the same event or comparing fields across links. Thanks for your patience! Hopefully this functionality will jump start links and events in TraceQL and we can quickly follow up with more advanced functionality. --- ### Design Proposals 2023 11 TraceQL Metrics (`docs/design-proposals/2023-11 TraceQL Metrics.md`) --- Authors: Martin Disibio (@mdisibio) Created: 2023 November Last updated: 2023-11-27 --- # TraceQL Metrics ## Summary This design document describes TraceQL language additions for extracting time-series metrics like error rates and latencies from traces. This document is **not** meant to be a complete language specification. Rather it is meant to invite the community to comment on the direction and capabilities. ### Table of Contents - [Intro](#intro) - [First Stage](#first-stage) - [Interval](#interval) - [Grouping](#grouping) - [Filtering](#filtering) - [Additional Stages](#additional-stages) - [Arithmetic](#arithmetic) - [Future](#future) - [Joins](#joins-between-time-series) - [Batching](#multiple-metrics-per-query) - [Notes](#notes) - [Step Interval](#step-interval) - [Departure from PromQL/LogQL](#departure-from-promqllogql) ## Intro Let's look at an example straight away. This query plots the error rate by service over time: `{status = error } | rate() by (resource.service.name)` A normal TraceQL query finds matching spans and returns data points like time, duration, service name, status code, etc. A metrics query at the core does the same thing: _find matching spans_, but instead of seeing individual results, they are aggregated over time to create time-series. A metrics query is built by extending a regular query with metrics pipeline stages. The structure looks like this: `<span query...> | <metrics query...>` An important understanding is that _any_ valid TraceQL query can be turned into a metrics query. This finds the call rates to service B but only when initiated upstream by service A: `{resource.service.name="A"} >> {resouce.service.name="B"} | rate()` ## First Stage The first stage in a metrics query is responsible for turning spans into time series. This is a unique intersection between the two data types and has specialized aggregation functions. We use the `..._over_time()` suffix to distinguish them from their counterparts. | function | description | ---------------------------------------------| ---------------------------- |`rate()` | The rate of spans per second |`count_over_time()` | The count of spans |`avg_over_time(<field>)` | The average of a numeric value like `duration` or [http.request.body.size](https://opentelemetry.io/docs/specs/semconv/attributes-registry/http/) |`max_over_time(<field>)` | The maximum of a numeric value. |`min_over_time(<field>)` | The minimum of a numeric value. |`quantile_over_time(<field>, q1, q2, ...)` | The quantile of a numeric value (e.g. p95). Multiple quantiles can be requested and a time-series is created for each. | ### Interval All aggregations take an optional interval like `30s`, `5m`, `1h`. If not specified then the interval will automatically match the step interval of the query. The interval is the last parameter of each function. `rate(5m)` `quantile_over_time(duration, 0.95, 5m)` ### Grouping All aggregations can group by one or more attributes and generate a time-series for each combination of values. Plot the request rate of `/myapi` by user ID and HTTP status code: `{ span.http.path = "/myapi" } | rate() by (span.user_id, span.http.status_code)` Plot the p95 duration of requests to service by HTTP endpoint: `{ resource.service.name = "myservice" } | quantile_over_time(duration, 0.95) by (span.http.path)` ### Filtering Aggregations can be combined with comparison operations to filter the output. Find the request rate per service but only if more than 1000 req/s: `{ } | rate() by (resource.service.name) > 1000` ## Additional Stages Additional metrics stages can be added to further refine and transform the time-series. These stages receive time-series and generate new time-series so they are separate from the first stage. Most functions support the same ability to regroup using `by()` and comparison operators to refilter. These can be thought of as instant functions that work across inputs at each point in time. Therefore they do not have an interval and do not alter the frequency of data points. If the original aggregation creates a data point every 60s, these will also output a data point every 60s. | function | description | ----------------------------------| ---------------------------- | `... \| max() [by(...)]` | Maximum value for each point in time | `... \| min() [by(...)]` | Minimum value for each point in time | `... \| avg() [by(...)]` | Average across values for each point in time | `... \| stddev() [by(...)]` | Standard deviation across values for each point in time | `... \| quantile(q) [by(...)]` | Quantile across values for each point in time | `... \| topk(N)` | Return the top series for each point in time (grouping TBD) Find the highest per-pod failure rate in each cluster: `{ status = error } | rate() by (cluster, pod) | max() by (cluster)` ## Arithmetic Arithmetic operations between time-series can be used to calculate things like error rates. Find the 5xx error rate for an endpoint: ``` ({ span.http.path = "/myapi" && span.http.status_code >= 500 } | rate()) / ({ span.http.path = "/myapi" | rate()) ``` Operations: * `*` * `/` * `+` * `-` ## Future These are language features and ideas that will be iterated on in the future. ### Joins between time-series Arithmetic operations bring up the need to define the join semantics between time-series. For context see [PromQL join semantics (vector-matching)](https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching) This area is TBD however the current direction is to provide the same functionality with a useful default behavior and specializations to support one-to-many and many-to-one. ### Multiple metrics per query It would be great to extend the query language to support calculating multiple metrics in a single query. It is common to plot related values like error rate and total rate together. This is more efficient because Tempo could compute all the metrics in a single pass over the data. `quantile_over_time` is already a step in this direction and supports calculating multiple quantiles (e.g. p50 and p95 in one query). However the full solution needs a higher-level construct like pipeline-splitting to support embedding the full range of metrics calculations in one query. ## Notes ### Step Interval The step interval is the explicit resolution of a query. It is identical to the [Prometheus step interval](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries). A step interval of `1m` means that a data point will be returned once every 60 seconds. It is separate from the `interval` of an aggregation like `rate(1h)`. The query `rate(1h)` with `step=1m` will still return a data point every 60 seconds but each data point is the rate smoothed over the previous hour. ### Departure from PromQL/LogQL The proposed pipeline-oriented syntax and aggregation functions are a departure from PromQL and LogQL. This is not done lightly but after much consideration deemed a better fit for Tempo architecture and goals. #### Labels Tempo's fundamental architecture is different than Prometheus/Loki which would require awkward and misleading translations of even the most basic PromQL queries to the tracing data model. Let's take a look at an example. Suppose we want to plot the rate by service. The PromQL-like query would look like this: `sum(rate({}[5m])) by (resource.service.name)` The outer sum expects the resource.service.name label to be provided by the rate, but it is not. Let's unwind one level. `rate({}[5m])` In PromQL/LogQL this returns a time-series per stream and they have inherent labels, defined from a combination of the application instrumentation and the metrics pipeline. But in Tempo there is no stream definition and we are left with two possibilities. 1. It returns an unlabeled total rate unless it has knowledge of the grouping labels. This is the proposed syntax for `rate() by(...)` 2. It simulates streams using the available data such as resource/span attributes. Unfortunately this is not feasible because there is no rule to choose which attributes should be included, and including everything leads to impossibly high cardinality. I.e. imagine a standard http request: attributes like path, method, status code are desirable for a stream definition, but others like request/response content length, headers, or full url with query params leads to uncontrollable cardinality. To summarize: what does `{} | rate()` return? Option 1 is a single time-series, Option 2 would be billions. #### Rate PromQL and LogQL rates are calculated using a [range vector selector](https://prometheus.io/docs/prometheus/latest/querying/basics/#range-vector-selectors) like `[5m]` combined with a metric. Both of these examples are valid constructs: `metric{}` `metric{}[5m]` This does not translate to Tempo because there are no streams, only a sea of spans. `{}` is valid and finds spans `{}[5m]` But this doesn't mean anything without aggregation. From there, the interval can be simplified as an parameter to the aggregation function instead of specialized syntax. #### Pipeline The pipeline syntax was chosen over functional syntax for a few reasons. * Easier to understand as computation flows in a single direction. * Easier to write as modifications require simply adding or deleting a stage instead of the delicate removal/addition of a function, its parameters and closing parenthesees. An existing query to find spans is converted to a metrics query by adding a stage in one place. --- ### Sources Tempo Docker Example (`docs/sources/tempo/docker-example.md`) --- title: Quick start for Tempo menuTitle: Quick start for Tempo description: Use Docker to quickly view traces using K-6 and Tempo weight: 200 aliases: - ./getting-started/docker-example/ # /docs/tempo/next/getting-started/docker-example/ killercoda: title: Quick start for Tempo description: Use Docker to quickly view traces using K-6 and Tempo details: intro: foreground: docker-compose-update.sh backend: backend: imageid: ubuntu --- # Quick start for Tempo Using this quickstart, you can set up a Tempo instance using Docker Compose so you can play and understand how Tempo works. If you want to use Tempo in production, refer to the [Set up for tracing](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/) documentation. The Tempo repository provides [multiple examples](https://github.com/grafana/tempo/tree/main/example/docker-compose) to help you quickly get started using Tempo and distributed tracing data. Every example has a `docker-compose.yaml` manifest that includes all of the options needed to explore trace data in Grafana, including resource configuration and trace data generation. The Tempo examples running with Docker using docker-compose include a version of Tempo and a storage configuration suitable for testing or development. ## Before you begin To follow this guide, you need: - Git - [Docker](https://docs.docker.com/compose/install/) - The Docker Compose plugin (included with Docker Desktop) {{< admonition type="tip" >}} Alternatively, you can try out this example in our interactive learning environment: [Quick start for Tempo](https://killercoda.com/grafana-labs/course/tempo/quick-start). It's a fully configured environment with all the dependencies installed. Provide feedback, report bugs, and raise issues in the [Grafana Killercoda repository](https://github.com/grafana/killercoda). {{< /admonition >}} {{<docs/ignore>}} {{< admonition type="tip" >}} A setup script is running in the background to install the necessary dependencies. This should take no longer than 30 seconds. Your instance will be ready to use once you `Setup complete. You may now begin the tutorial`. {{< /admonition >}} {{</docs/ignore>}} ## Clone the Tempo repository and start Docker This quick start guide uses the `single-binary` docker-compose example. Check out our [collection of examples](https://github.com/grafana/tempo/blob/main/example/docker-compose) to explore various operating modes and configurations. 1. Clone the Tempo repository: ```bash git clone https://github.com/grafana/tempo.git ``` 1. Go into the examples directory: ```bash cd tempo/example/docker-compose/single-binary ``` 1. Start the services defined in the docker-compose file: ```bash docker compose up -d ``` 1. Verify that the services are running: ```bash docker compose ps ``` You should see something like: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ## Explore the traces in Grafana As part of the Docker Compose manifest, Grafana is now accessible on port 3000. You can use Grafana to explore the traces generated by the k6-tracing service. 1. Open a browser and navigate to [http://localhost:3000](http://localhost:3000). 1. Once logged in, navigate to the **Explore** page, select the **Tempo** data source and select the **Search** tab. Select **Run query** to list the recent traces stored in Tempo. Select one to view the trace diagram: {{< figure align="center" src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-builder-span-details-v11.png" alt="Use the query builder to explore tracing data in Grafana" >}} 1. A couple of minutes after Tempo starts, select the **Service graph** tab for the Tempo data source in the **Explore** page. Select **Run query** to view a service graph, generated by Tempo’s metrics-generator. {{< figure align="center" src="/media/docs/grafana/data-sources/tempo/query-editor/tempo-ds-query-service-graph.png" alt="Service graph sample" >}} 1. To stop the services: ```bash docker compose down -v ``` ## Traces Drilldown plugin The [Traces Drilldown](https://grafana.com/docs/grafana/latest/explore/simplified-exploration/traces/) plugin offers an opinionated non query-based approach to exploring traces. Let's take a look at some of its key features and panels. 1. Open a browser and navigate to [http://localhost:3000/a/grafana-exploretraces-app](http://localhost:3000/a/grafana-exploretraces-app). 2. Within the filter bar, there is a dropdown menu set to **Rate** of **Full traces**. Change this to **Duration** and **All spans**. This updated panel view looks like this: {{< figure align="center" src="/media/docs/tempo/explore-spans-error-view.png" alt="Traces Drilldown panel" >}} Breakdown of the view: - The histogram at the top shows the distribution of span durations. The lighter the color, the more spans in that duration bucket. In this example, most spans fall within `537ms`, which is considered the average duration for the system. - The high peaks in the histogram indicate spans that are taking longer than the average (As high as `2.15s`). These are likely to be the spans that are causing performance issues. You can investigate further to identify the root cause. Select `Slow traces` tab in the navigation bar to view the slowest traces in the system. {{< figure align="center" src="/media/docs/tempo/slow-trace-view.png" alt="Slow traces panel" >}} `shop-backend` appears to be the primary culprit for the slow traces. This happens when a user initiates the `article-to-cart` operation. From here, you can select the **Trace Name** to open the **Trace View** panel. {{< figure align="center" src="/media/docs/tempo/slow-trace-trace-view.png" alt="Trace View panel" >}} The **Trace View** panel provides a detailed view of the trace. The panel is divided into three sections: - The top section shows the trace ID, duration, and the service that generated the trace. - The middle section shows the trace timeline. Each span is represented as a horizontal bar. The color of the bar represents the span's status. The width of the bar represents the duration of the span. - The bottom section shows the details of the selected span. This includes the span name, duration, and tags. Drilling into the `shop-backend` span, you can see that the `place-articles` operation has an exception event tied to it. This is likely the root cause of the slow trace. {{< figure align="center" src="/media/docs/tempo/slow-trace-root-cause-2.png" alt="Span View panel" >}} If you would like to dive deeper into the Traces Drilldown plugin and its panel concepts, refer to the [Traces Drilldown Concepts](https://grafana.com/docs/grafana/latest/explore/simplified-exploration/traces/concepts/). ## Next steps You have successfully set up Tempo and Grafana to explore traces generated by the k6-tracing service. Now that you've explored the basics of Tempo, you can: - Set up Tempo by following the [Set up for tracing documentation](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/). - Learn how to [instrument your application](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-instrumentation/) to emit traces. ### Alternative: Complete MLTP example If you would like to use a demo with multiple telemetry signals, then try the [Introduction to Metrics, Logs, Traces, and Profiling in Grafana](https://github.com/grafana/intro-to-mltp). `Intro-to-mltp` provides a self-contained environment for learning about Mimir, Loki, Tempo, Pyroscope, and Grafana. The project includes detailed explanations of each component and annotated configurations for a single-instance deployment. Data from `intro-to-mltp` can also be pushed to Grafana Cloud. ### Further reading Here are some resources to help you learn more about Tempo: - [Instrumenting your application](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/) - [Set up for tracing](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/) --- ### Modules Frontend Docs Aggregates (`modules/frontend/docs/aggregates.md`) # TraceQL Aggregates Documentation ## Overview TraceQL aggregates allow you to perform calculations on sets of spans within traces. These functions help select individual traces based on aggregate data in the spans. ## Available Aggregate Functions - `count` - Count of spans in the spanset - `avg` - Average of a numeric attribute or intrinsic - `max` - Maximum value of a numeric attribute or intrinsic - `min` - Minimum value of a numeric attribute or intrinsic - `sum` - Sum of a numeric attribute or intrinsic ## Basic Usage Aggregates are applied to spansets using the pipe operator `|`: ``` { <span_selection> } | <aggregate_function> ``` ## Examples ### Count Aggregates Count total spans in a trace: ``` { } | count() ``` Count spans with specific conditions: ``` { span.http.status_code = 200 } | count() ``` Find traces with more than 10 spans: ``` { } | count() > 10 ``` Find traces with more than 3 error spans: ``` { span:status = error } | count() > 3 ``` ### Duration Aggregates Find traces with average span duration > 20ms: ``` { } | avg(span:duration) > 20ms ``` Find traces with maximum span duration > 1s: ``` { } | max(span:duration) > 1s ``` Find traces with minimum span duration < 1ms: ``` { } | min(span:duration) < 1ms ``` ### Custom Attribute Aggregates Sum up custom numeric attributes: ``` { } | sum(span.bytes_processed) > 1000000 ``` Average response size: ``` { span.http.response_size != nil } | avg(span.http.response_size) > 1024 ``` ### Combining with Conditions Find slow frontend operations: ``` { resource.service.name = "frontend" } | avg(span:duration) > 100ms ``` Find services with high error rates: ``` { span:status = error } | count() > 5 ``` ## Grouping with Aggregates You can group spans before applying aggregates using the `by()` operator: ``` { span:status = error } | by(resource.service.name) | count() > 1 ``` This finds services with more than 1 error span per trace. ## Performance Notes - Aggregates are computed per trace --- ### Modules Frontend Docs Basic (`modules/frontend/docs/basic.md`) # TraceQL Basic Query Documentation ## Overview TraceQL is a query language for searching and filtering traces. Queries are expressions evaluated on one trace at a time, structured as a set of chained expressions called a pipeline. TraceQL selection can not answer broad questions about trace data. It can only find specific traces. Use metrics for broader questions. ## Query Structure TraceQL queries use curly brackets `{}` to select spans from traces. Each query expression filters spans based on conditions. ### Basic Syntax ``` { <conditions> } ``` ### Pipeline Structure ``` { <span_selection> } | <aggregator> | <condition> ``` ## Scopes and Attributes TraceQL differentiates between intrinsics (fundamental to spans) and attributes (customizable key-value pairs). ### Intrinsic Fields Intrinsic fields use colon notation `<scope>:`: - `span:status` - Status: error, ok, or unset - `span:statusMessage` - Optional text accompanying span status - `span:duration` - End - start time of the span - `span:name` - Operation or span name - `span:kind` - Kind: server, client, producer, consumer, internal, unspecified - `span:id` - Span ID using hex string - `span:parentID` - Parent span ID using hex string - `span:childCount` - Number of direct children of the span (requires vParquet5 or later) - `trace:duration` - Max(end) - min(start) time of spans in trace - `trace:rootName` - Name of the root span (if it exists) - `trace:rootService` - Service name of the root span (if it exists) - `trace:id` - Trace ID using hex string - `event:name` - Name of event - `event:timeSinceStart` - Time of event relative to span start - `link:spanID` - Link span ID using hex string - `link:traceID` - Link trace ID using hex string - `instrumentation:name` - Instrumentation scope name - `instrumentation:version` - Instrumentation scope version ### Attribute Fields Attributes use dot notation `<scope>.`: - `span.` - Span attributes - `resource.` - Resource attributes - `event.` - Event attributes - `link.` - Link attributes - `instrumentation.` - Instrumentation scope attributes ## Value Types and Literals TraceQL supports several literal types for expressing values in queries. - Integers: positive or negative whole numbers, for example `200` or `-1`. The constants `minInt` and `maxInt` represent the minimum and maximum 64-bit integer values. - Floats: decimal notation, for example `1.5`. - Durations: a number with a unit. Supported units are `ns`, `us`, `ms`, `s`, `m`, and `h`, for example `100ms` or `5s`. Durations can be signed. - Strings: enclosed in double quotes or backticks, for example `"GET"` or `` `GET` ``. - Nil: use `nil` to check for missing or null attributes, and `!= nil` to require an attribute to be present. Examples: ``` { span.http.status_code = 200 } { span:duration > 100ms } { span.value > 1.5 } { span.optional_field = nil } { span.required_field != nil } ``` ## Comparison Operators - `=` - Equality - `!=` - Inequality - `>` - Greater than - `>=` - Greater than or equal - `<` - Less than - `<=` - Less than or equal - `=~` - Regular expression match (fully anchored) - `!~` - Negated regular expression match ### Regular Expression Notes - All regex patterns are fully anchored (treated as `^pattern$`) - Use `.*pattern.*` for unanchored matching - TraceQL uses Golang regular expressions ## Logical Operators - `&&` - AND (all conditions must be true on the same span) - `||` - OR (either condition can be true) ## Spanset Operators Spanset operators allow conditions to be asserted on different spans within a trace. Also see structural operators. - `{condA} && {condB}` - Both conditions found matches - `{condA} || {condB}` - Either condition found matches ## Common Examples ### Find traces of a specific operation ``` { resource.service.name = "frontend" && span:name = "POST /api/orders" } ``` ### Find traces with environment and service specification ``` { resource.service.namespace = "ecommerce" && resource.service.name = "frontend" && resource.deployment.environment = "production" && span:name = "POST /api/orders" } ``` ### Find traces with errors ``` { resource.service.name = "frontend" && span:name = "POST /api/orders" && span:status = error } ``` ### Find traces with HTTP 5xx errors ``` { resource.service.name = "frontend" && span:name = "POST /api/orders" && span.http.status_code >= 500 } ``` ### Find traces with specific behavior patterns ``` { resource.service.name = "frontend" && span:name = "GET /api/products/{id}" } && { span.db.system = "postgresql" } ``` ### Find traces going through multiple environments ``` { resource.deployment.environment = "production" } && { resource.deployment.environment = "staging" } ``` ### Find traces with arrays ``` { span.foo = "bar" } { span.http.request.header.Accept =~ "application.*" } ``` ### Field expressions with combined conditions ``` { span.http.status_code >= 200 && span.http.status_code < 300 } { span.http.method = "DELETE" && span:status != ok } ``` ### Check attribute existence ``` { span.any_attribute != nil } ``` ### Using quoted attribute names ``` { span."attribute name with space" = "value" } { span.attribute."attribute name with space" = "value" } ``` ## Selecting Fields Use the `select()` pipeline operator to return specific attributes from matching spans. Selected fields aren't retrieved until all other criteria are met, which makes this performant. ``` { span.http.status_code = 200 } | select(resource.service.name) { span:status = error } | select(span.http.status_code, span.http.url) ``` ## Arithmetic in Expressions TraceQL supports arithmetic within field expressions, which can make queries more readable. ``` { span.http.request_content_length > 10 * 1024 * 1024 } ``` ## Retrieve Most Recent Results By default, Tempo returns the first `N` matching traces, which might not be the newest. Use the `most_recent=true` query hint with any selection query to force Tempo to return the most recent results ordered by time. ``` {} with (most_recent=true) { span.foo = "bar" } >> { span:status = error } with (most_recent=true) ``` ## Performance Tips - Use trace-level intrinsics (`trace:duration`, `trace:rootName`, `trace:rootService`) when possible for better performance - Use appropriate scopes to limit data scanning - Use specific span selections to reduce data processing --- ### Modules Frontend Docs Config Reference (`modules/frontend/docs/config-reference.md`) # Tempo Configuration Reference This is the complete reference of every Tempo configuration option and its default value, rendered from the default configuration. Most installations only set 10 to 20 of these options. For a guided overview of which block does what, request the "config-overview" doc. For exact key names, nesting, and defaults, use the reference below. ## Complete configuration ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Modules Frontend Docs Metrics (`modules/frontend/docs/metrics.md`) # TraceQL Metrics Documentation ## Overview TraceQL metrics functions convert trace queries into time-series metrics. These functions aggregate span data over time intervals to provide insights into system performance and behavior. ## Available Metrics Functions ### rate() Calculates the number of matching spans per second. ``` { <span_selection> } | rate() ``` Examples: ``` { span:status = error } | rate() { resource.service.name = "frontend" } | rate() by (span.http.route) ``` ### count_over_time() Counts matching spans per time interval (controlled by `step` parameter). ``` { <span_selection> } | count_over_time() ``` Examples: ``` { span:name = "GET /api/users" } | count_over_time() { span.http.status_code >= 500 } | count_over_time() by (resource.service.name) ``` ### sum_over_time() Sums numeric values per time interval. ``` { <span_selection> } | sum_over_time(<numeric_field>) ``` Examples: ``` { } | sum_over_time(span.bytes_processed) { resource.service.name = "api" } | sum_over_time(span:duration) ``` ### avg_over_time() Calculates average values per time interval. ``` { <span_selection> } | avg_over_time(<numeric_field>) ``` Examples: ``` { } | avg_over_time(span:duration) { span.http.method = "POST" } | avg_over_time(span.http.response_size) ``` ### min_over_time() and max_over_time() Find minimum and maximum values per time interval. ``` { <span_selection> } | min_over_time(<numeric_field>) { <span_selection> } | max_over_time(<numeric_field>) ``` Examples: ``` { } | min_over_time(span:duration) { span.http.status_code >= 400 } | max_over_time(span:duration) ``` ### quantile_over_time() Calculate percentiles of numeric values. ``` { <span_selection> } | quantile_over_time(<numeric_field>, <quantile1>, <quantile2>, ...) ``` Examples: ``` { } | quantile_over_time(span:duration, .50, .95, .99) { resource.service.name = "api" } | quantile_over_time(span:duration, .99) by (span.http.route) ``` ### histogram_over_time() Evaluate frequency distribution over time. ``` { <span_selection> } | histogram_over_time(<numeric_field>) by (<grouping_field>) ``` ## Grouping with by() Use `by()` to group metrics by specific attributes: ``` { } | rate() by (resource.service.name) { span:status = error } | count_over_time() by (resource.service.name, span.http.route) ``` ## Filtering Results with topk() and bottomk() Limit results to top or bottom N series: ``` { } | rate() by (resource.service.name) | topk(10) { } | rate() by (span.http.route) | bottomk(5) ``` ## Comparison Operators on Results Apply comparison operators to a metrics result to keep only the data points that meet a threshold. Supported operators are `>`, `>=`, `<`, `<=`, `=`, and `!=`. You can compare against integers, floats, and durations (for example `1s` or `500ms`). ``` { } | rate() by (resource.service.name) > 10 { span:name = "GET /:endpoint" } | avg_over_time(span:duration) > 1s { } | count_over_time() by (span:name) != 0 ``` Data points that don't match are removed. If all data points in a series are removed, the entire series is dropped. Comparison operators can be chained with `topk` and `bottomk` in any order, applied left to right: ``` { } | rate() by (span.http.url) | topk(5) > 10 { } | rate() by (span.http.url) > 0 | topk(5) ``` ## Data Sampling TraceQL metrics queries support sampling hints to optimize performance. Sampling hints only work with metrics queries (functions like `rate()`, `count_over_time()`, and so on). - Dynamic sampling: `with(sample=true)` automatically determines the sampling strategy and amount based on the query. - Fixed sampling: `with(sample=0.xx)`. - Fixed span sampling: `with(span_sample=0.xx)` selects the given percentage of spans. - Fixed trace sampling: `with(trace_sample=0.xx)` selects complete traces. Examples: ``` { resource.service.name = "frontend" } | rate() with(sample=true) { } | count_over_time() by (resource.service.name) with(sample=0.1) { span:status = error } | count_over_time() with(span_sample=0.1) { } | count_over_time() by (resource.service.name) with(trace_sample=0.05) ``` ## Compare Function Split spans into a selection group and a baseline group to highlight the differences between them. The function returns time-series for all attributes found on the spans. ``` { <base_condition> } | compare({<selection_condition>}, <topN>, <start_timestamp>, <end_timestamp>) ``` Parameters: - Required. A spanset filter that chooses the subset of spans. Matching spans are the selection; the rest are the baseline. Common filters are `{span:status = error}` (what is different about errors?) or `{span:duration > 1s}` (what is different about slow spans?). - Optional. The top `N` values to return per attribute. Defaults to `10`. - Optional. Start and end timestamps in Unix nanoseconds to constrain the selection window by time. Both must be given, or neither. Example: ``` { resource.service.name = "api" && span.http.path = "/users" } | compare({span:status = error}) ``` The output is flat time-series for each attribute/value found in the spans. Each series carries a `__meta_type` label indicating the group. Possible values are `baseline`, `selection`, `baseline_total`, and `selection_total`: ``` { __meta_type="baseline", resource.cluster="prod" } 123 { __meta_type="selection", resource.cluster="prod" } 456 { __meta_type="baseline_total", resource.cluster="prod" } 1000 { __meta_type="selection_total", resource.cluster="prod" } 800 ``` The `baseline_total` and `selection_total` series report the overall count across all values for each attribute, which helps calculate relative proportions. When an attribute exceeds the top `N` limit, an error indicator series is included: ``` { __meta_error="__too_many_values__", resource.cluster=<nil> } ``` `compare()` can't be combined with second-stage functions such as `topk`, `bottomk`, comparison operators, or arithmetic expressions. It's generally run as an instant query. ## Practical Examples ### Error Rate Monitoring ``` { span:status = error } | rate() by (resource.service.name) ``` ### Response Time Analysis ``` { span.http.method = "GET" } | quantile_over_time(span:duration, .50, .95, .99) by (span.http.route) ``` ### Service Performance Comparison ``` { resource.service.name = "api" } | avg_over_time(span:duration) by (resource.deployment.environment) ``` ### Top Slowest Endpoints ``` { } | avg_over_time(span:duration) by (span.http.route) | topk(10) ``` ### Error Distribution ``` { span:status = error } | count_over_time() by (resource.service.name, span.http.status_code) ``` ## Step Parameter The `step` parameter controls the granularity of time-series data: - Default: Automatically chosen based on query time range - Custom: Use values like `30s`, `1m`, `5m` - Configured via Grafana Explore or API ## Performance Tips - Group by high-cardinality fields carefully - Use topk/bottomk to limit result sets - Consider using instant queries for single-point-in-time analysis --- ### Modules Frontend Docs Structural (`modules/frontend/docs/structural.md`) # TraceQL Structural Operators Documentation ## Overview Structural operators in TraceQL examine the hierarchical relationships between spans within a trace. These operators help you find traces based on parent-child relationships and span ordering. **Important** For all structural operators (except union) only the spans matching the right hand side conditions are returned. ## Structural Operators ### Descendant Operator (`>>`) Finds spans matching the right condition that are descendants of spans matching the left condition. ``` {condA} >> {condB} ``` Example - Find database calls downstream from frontend service: ``` { resource.service.name = "frontend" } >> { span.db.system = "postgresql" } ``` ### Ancestor Operator (`<<`) Finds spans matching the right condition that are ancestors of spans matching the left condition. ``` {condA} << {condB} ``` Example - Find root spans that have error descendants: ``` { span:status = error } << { span:kind = server } ``` ### Child Operator (`>`) Finds spans matching the right condition that are direct children of spans matching the left condition. ``` {condA} > {condB} ``` Example - Find direct child spans of HTTP requests: ``` { span.http.method = "GET" } > { span.db.system != nil } ``` ### Parent Operator (`<`) Finds spans matching the right condition that are direct parents of spans matching the left condition. ``` {condA} < {condB} ``` Example - Find parent spans of database operations: ``` { span.db.system = "postgresql" } < { span.http.method != nil } ``` ### Sibling Operator (`~`) Finds spans matching the right condition that have siblings matching the left condition. ``` {condA} ~ {condB} ``` Example - Find spans that are siblings of error spans: ``` { span:status = error } ~ { span:status = ok } ``` ## Union Structural Operators These operators return spans that match on both sides of the operator: ### Union Descendant (`&>>`) Returns both the ancestor and descendant spans. ``` { span.http.url = "/api/orders" && span:status = error } &>> { span:status = error } ``` ### Union Ancestor (`&<<`) Returns both the descendant and ancestor spans. ### Union Child (`&>`) Returns both the parent and child spans. ### Union Parent (`&<`) Returns both the child and parent spans. ### Union Sibling (`&~`) Returns both sibling spans. ## Experimental Structural Operators (Not Operators) These operators can have false positives but are useful for specific queries: ### Not-Descendant (`!>>`) Finds spans that are NOT descendants of the left condition. ``` { } !>> { resource.service.name = "database" } ``` ### Not-Child (`!>`) Finds spans that are NOT direct children of the left condition. ### Not-Parent (`!<`) Finds spans that are NOT direct parents of the left condition. Example - Find leaf spans in a service: ``` { } !< { resource.service.name = "productcatalog" } ``` ### Not-Sibling (`!~`) Finds spans that are NOT siblings of the left condition. ## Practical Examples ### Find cascading errors Find the last error in a series of cascading errors: ``` { span:status = error } !< { span:status = error } ``` ### Find service interaction patterns Find frontend calls that lead to database operations: ``` { resource.service.name = "frontend" && span.http.method = "POST" } >> { span.db.system = "postgresql" } ``` ### Find cross-environment issues Find traces that go through both production and staging: ``` { resource.deployment.environment = "production" } && { resource.deployment.environment = "staging" } ``` ### Find service dependencies Find which services call the user service: ``` { resource.service.name = "user" } << { resource.service.name != "user" } ``` ## Performance Considerations - Structural operators can be expensive on large traces - Consider using logical operators (`&&`, `||`) instead when span relationships aren't required - Combine with other filters to improve performance --- ### Sources Tempo Api Docs Mcp Server (`docs/sources/tempo/api_docs/mcp-server.md`) --- title: Tempo MCP server description: Grafana Tempo exposes an MCP server to allow LLMs and AI assistants to interact with your trace data. menuTitle: MCP server weight: 800 --- # Model Context Protocol (MCP) Server Tempo includes an MCP (Model Context Protocol) server that provides AI assistants and large language models (LLMs) with direct access to distributed tracing data through TraceQL queries and other endpoints. For examples on how you can use the MCP server, refer to [LLM-powered insights into your tracing data: introducing MCP support in Grafana Cloud Traces](https://grafana.com/blog/2025/08/13/llm-powered-insights-into-your-tracing-data-introducing-mcp-support-in-grafana-cloud-traces/). For more information on MCP, refer to the [MCP documentation](https://modelcontextprotocol.io/docs/getting-started/intro). ## Configuration You can enable the MCP server in your Tempo configuration via YAML: ```yaml query_frontend: mcp_server: enabled: true ``` Or via a command-line flag: `--query-frontend.mcp-server.enabled=true`. {{< admonition type="warning" >}} Be aware that using this feature may cause tracing data to be passed to an LLM or LLM provider. Consider the content of your tracing data and organizational policies when enabling this feature. {{< /admonition >}} The MCP server uses the same authentication and [multi-tenancy](../../operations/manage-advanced-systems/multitenancy/) behavior as other Tempo API endpoints. ## Available tools The MCP server exposes the following tools that AI assistants can use to interact with your tracing data: | Tool | Description | | ------------------------- | ----------------------------------------------------------------------- | | `traceql-search` | Search for traces using TraceQL queries | | `traceql-metrics-instant` | Retrieve a single metric value given a TraceQL metrics query | | `traceql-metrics-range` | Retrieve a metric series given a TraceQL metrics query | | `get-trace` | Retrieve a specific trace by ID | | `traces-diff` | Compare two complete traces and return their differences | | `get-attribute-names` | Get available attribute names for use in TraceQL queries | | `get-attribute-values` | Get values for a specific scoped attribute name | | `docs-traceql` | Retrieve TraceQL documentation (basic, aggregates, structural, metrics) | | `docs-config` | Retrieve Tempo configuration documentation (overview, reference) | ### Compare traces The experimental `traces-diff` tool compares a baseline trace with another complete trace. Provide `base_trace_id` and `compare_trace_id`. By default, the tool returns `trace-summary-v0-composed`, which always contains a compact summary and includes the full span-level patch when the patch is no larger than 64 KiB. The 64 KiB limit applies only to the attached patch, not to the complete composed response. Omit `format` to use the composed response. Set it to `trace-summary-v0-native` to return only the compact summary, or to `trace-patch-v0` when you need the complete span-level patch. An explicitly empty `format` is invalid. If a composed response contains `patchOmitted`, use the summary for initial analysis. Request `trace-patch-v0` only when span-level evidence is required and the client can accept a patch of `patchOmitted.bytes` bytes. Full patch responses have no output-size guarantee. The optional `base_start` and `base_end` arguments limit the baseline lookup to an RFC3339 time range; both must be provided together. The same rule applies to `compare_start` and `compare_end` for the comparison trace. Tempo searches all blocks for a trace when its range is omitted. Partial traces are rejected because their comparison could be inaccurate. ## Available resources The MCP server also provides the following resources containing TraceQL and configuration documentation: | Resource URI | Description | | --------------------------- | -------------------------------------------------------------------- | | `docs://traceql/basic` | Basic TraceQL syntax, intrinsics, operators, and attributes | | `docs://traceql/aggregates` | TraceQL aggregate functions (count, sum, etc.) | | `docs://traceql/structural` | Advanced structural query patterns | | `docs://traceql/metrics` | Generating metrics from tracing data with TraceQL | | `docs://config/overview` | Orientation map of what each top-level configuration block controls | | `docs://config/reference` | Complete reference of all configuration options and their defaults | ## Quick start To experiment with the MCP server using dummy data and Claude Code: 1. Run the local docker-compose example in `/example/docker-compose/single-binary`. This exposes the MCP server at `http://localhost:3200/api/mcp` 1. Run `claude mcp add --transport=http tempo http://localhost:3200/api/mcp` to add a reference to Claude Code. 1. Run `claude` and ask some questions. The Tempo MCP server uses the Streamable HTTP transport. Any MCP client that supports this transport can connect directly using the URL `http://<tempo-host>:<port>/api/mcp`. For example, in Cursor you can add the server with `type: "streamableHttp"` in your MCP configuration. If your client doesn't support Streamable HTTP natively, you can use the [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) package as a bridge. --- ### Sources Tempo Api Docs Pushing Spans With Http (`docs/sources/tempo/api_docs/pushing-spans-with-http.md`) --- title: Push spans menuTitle: Push spans description: Learn how to push spans to Tempo using HTTP or gRPC with the OpenTelemetry Protocol (OTLP) receiver. aliases: - /docs/tempo/latest/guides/pushing-spans-with-http/ --- # Push spans with HTTP or gRPC Sometimes using a tracing system is intimidating because it seems like you need complex application instrumentation or a span ingestion pipeline to push spans. This guide shows a basic technique for pushing spans with HTTP/JSON or gRPC using the [OpenTelemetry](https://opentelemetry.io/docs/specs/otlp/) receiver. ## Before you begin This procedure uses an example Docker Compose setup to run Tempo, so you don't need an existing installation. The Docker image also includes a Grafana container which lets you visualize traces. To use this procedure, you need: - Docker and `docker compose` installed - (Optional) [OpenTelemetry `telemetrygen`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen) installed for the gRPC examples ## Start Tempo using the quickstart Use the instructions in the [Quick start for Tempo documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/getting-started/docker-example/) to start a local instance of Tempo and Grafana. ## Push spans with OTLP/HTTP Now that Tempo is running and listening on port 4318 for [OTLP/HTTP spans](https://opentelemetry.io/docs/specs/otlp/#otlphttp), you can push a span to Tempo using `curl`. {{< shared id="push-spans-http-otlp" >}} Before you can use this example, you need to update the start and end time. The `startTimeUnixNano` and `endTimeUnixNano` fields are in nanoseconds, where 100000000 nanoseconds is 100 milliseconds. The nanosecond value can be obtained by any tool that provides the epoch date in nanoseconds. For example, under Linux, you can use `date +%s%N`. You can also use an online tool such as [Epoch Converter](https://www.epochconverter.com/) to get the current time in nanoseconds. 1. Copy and paste the `curl` command into a text editor. ```bash curl -X POST -H 'Content-Type: application/json' http://localhost:4318/v1/traces -d ' { "resourceSpans": [{ "resource": { "attributes": [{ "key": "service.name", "value": { "stringValue": "my.service" } }] }, "scopeSpans": [{ "scope": { "name": "my.library", "version": "1.0.0", "attributes": [{ "key": "my.scope.attribute", "value": { "stringValue": "some scope attribute" } }] }, "spans": [ { "traceId": "5B8EFFF798038103D269B633813FC700", "spanId": "EEE19B7EC3C1B100", "name": "I am a span!", "startTimeUnixNano": 1689969302000000000, "endTimeUnixNano": 1689970000000000000, "kind": 2, "attributes": [ { "key": "my.span.attr", "value": { "stringValue": "some value" } }] }] }] }] }' ``` 1. Replace `startTimeUnixNano` and `endTimeUnixNano` with current values for the last 24 hours to allow you to search for them using a 24 hour relative time range. You can get this in seconds and milliseconds from the [Unix Epoch Clock](https://www.epochconverter.com/). For `startTimeUnixNano`, use the current time minus 24 hours. Use the link above to figure out the value. For `endTimeUnixNano`, use the current time. Multiply the milliseconds value by 1,000,000 to turn it into nanoseconds. You can do this from a bash terminal with the following command. Replace `<epochTimeMilliseconds>` with the milliseconds value. ```bash echo $((<epochTimeMilliseconds> * 1000000)) ``` 1. Copy the updated curl command to a terminal window and run it. {{< /shared >}} 1. View the trace in Grafana: 1. Open a browser window to http://localhost:3000. 1. Open the **Explore** page and select the Tempo data source. 1. Select the **Search** query type. 1. Select **Run query** to list available traces. 1. Select the trace ID (yellow box) to view details about the trace and its spans. ## Push spans with OTLP/gRPC Tempo also listens on port 4317 for [OTLP/gRPC](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc) spans. You can use [OpenTelemetry `telemetrygen`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen) to push test traces over gRPC. 1. Install `telemetrygen` following the [installation instructions](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/cmd/telemetrygen). 1. Push traces to Tempo over gRPC: ```bash telemetrygen traces --otlp-insecure --rate 20 --duration 5s --otlp-endpoint localhost:4317 ``` This sends 100 traces (20 per second for 5 seconds) to the Tempo distributor's OTLP/gRPC receiver. 1. Verify the traces arrived by searching for them. The `telemetrygen` tool uses the service name `telemetrygen` by default: ```bash curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ resource.service.name = "telemetrygen" }' | jq . ``` ### Multi-tenancy If you've enabled multi-tenancy (`multitenancy_enabled: true`), pass the tenant ID using the `X-Scope-OrgID` header in gRPC metadata: ```bash telemetrygen traces --otlp-insecure --rate 20 --duration 5s \ --otlp-endpoint localhost:4317 \ --otlp-header "X-Scope-OrgID=my-tenant" ``` For more information about configuring OpenTelemetry SDKs to export traces over gRPC, refer to the [OpenTelemetry documentation](https://opentelemetry.io/docs/specs/otlp/#otlpgrpc). ## Retrieve traces You can retrieve the trace by running a `curl` command against Tempo. The returned format is [OTLP](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/trace/v1/trace.proto). 1. Replace the trace ID in the `curl` command with the trace ID that was generated from the push. This information is in the data that's sent with the `curl`. You can use the Grafana **Explore** page to find this, as shown in the previous section. ```bash curl http://localhost:3200/api/v2/traces/5b8efff798038103d269b633813fc700 {"trace": {"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]},"scopeSpans":[{"scope":{"name":"my.library","version":"1.0.0"},"spans":[{"traceId":"W47/95gDgQPSabYzgT/HAA==","spanId":"7uGbfsPBsQA=","name":"I am a span!","kind":"SPAN_KIND_SERVER","startTimeUnixNano":"1689969302000000000","endTimeUnixNano":"1689970000000000000","attributes":[{"key":"my.span.attr","value":{"stringValue":"some value"}}],"status":{}}]}]}]}} ``` 1. Copy and paste the updated `curl` command into a terminal window. ### Use TraceQL to search for a trace Alternatively, you can also use [TraceQL](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/) to search for the trace that was pushed. You can search by using the unique trace attributes that were set: ```bash curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ resource.service.name = "my.service" }' {"traces":[{"traceID":"5b8efff798038103d269b633813fc700","rootServiceName":"my.service","rootTraceName":"I am a span!","startTimeUnixNano":"1694718625557000000","durationMs":10000,"spanSet":{"spans":[{"spanID":"eee19b7ec3c1b100","startTimeUnixNano":"1694718625557000000","durationNanos":"10000000000","attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]}],"matched":1},"spanSets":[{"spans":[{"spanID":"eee19b7ec3c1b100","startTimeUnixNano":"1694718625557000000","durationNanos":"10000000000","attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]}],"matched":1}]}],"metrics":{"inspectedBytes":"292781","completedJobs":1,"totalJobs":1}} ``` To format this in a more human-readable output, consider using a [tool such as `jq`](https://jqlang.github.io/jq/), which lets you to run the same `curl` command and pipe it to `jq` to format the block. For example: ```bash curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ resource.service.name = "my.service" }' | jq ``` --- ### Sources Tempo Configuration Anonymous Usage Reporting (`docs/sources/tempo/configuration/anonymous-usage-reporting.md`) --- title: Anonymous usage reporting description: Learn about anonymous usage statistics reporting in Tempo. weight: 950 --- # Anonymous usage reporting By default, Tempo reports anonymous usage data about itself to Grafana Labs. This data is used to understand which features are commonly enabled, as well as which deployment modes, replication factors, and block formats are most popular. By providing information on how people use Tempo, usage reporting helps the Tempo team decide where to focus their development and documentation efforts. The following configuration values are used: - Receivers enabled - Frontend concurrency and version - Storage cache, backend, and configured block format - Ring replication factor, and `kvstore` - Features toggles enabled No private information is collected, and all reports are completely anonymous. ## Configure anonymous usage reporting Reporting is controlled by the `usage_report` configuration option and can be disabled. For instructions, refer to [the Configuration documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#usage-report). ## View usage stats report Tempo provides a usage stats report that you can view in your browser. To view the report, go to the following URL on the Tempo instance: `http://localhost:3200/status/usage-stats` An example report output looks like this: ```json { "clusterID": "", "createdAt": "0001-01-01T00:00:00Z", "interval": "0001-01-01T00:00:00Z", "intervalPeriod": 14400, "target": "all", "version": { "version": "v3.0.0", "revision": "31e2dddb5", "branch": "main", "buildUser": "", "buildDate": "", "goVersion": "go1.24.3" }, "os": "linux", "arch": "arm64", "edition": "oss", "metrics": { "ring_kv_store": "inmemory", "memstats": { "heap_inuse": 14106624, "stack_inuse": 2490368, "pause_total_ns": 872084, "num_gc": 3, "gc_cpu_fraction": 0.08786506719453682, "heap_alloc": 11640400, "alloc": 11640400, "total_alloc": 16491760, "sys": 27874568 }, "num_cpu": 8, "feature_enabled_multitenancy": 0, "receiver_enabled_jaeger": 0, "storage_block_format": "vParquet5", "storage_cache": "", "feature_enabled_auth_stats": 0, "frontend_version": "v1", "storage_backend": "local", "receiver_enabled_otlp": 0, "ring_replication_factor": 1, "num_goroutine": 813, "cache_memcached": 1, "cache_redis": 0, "distributor_bytes_received": { "total": 0, "rate": 0 }, "distributor_spans_received": { "total": 0, "rate": 0 }, "receiver_enabled_kafka": 0, "receiver_enabled_zipkin": 0 } } ``` ## Which information is collected? Tempo collects and reports the following information to Grafana Labs. The report from your Tempo instance may vary from the provided example. Each field provides insight into the Tempo instance, its environment, and configuration. The fields are grouped by their purpose. This information helps Grafana Labs understand how Tempo is used, which features are enabled, and the typical deployment environments, without collecting any private or user-identifying data. {{< admonition type="note">}} Tempo maintainers commit to keeping the list of tracked information updated over time, and reporting any change both via the CHANGELOG and the release notes. {{< /admonition>}} ### Instance identification - **`clusterID`**: A unique, randomly generated identifier for the Tempo cluster. This value helps Grafana Labs distinguish between different deployments. - **`createdAt`**: The timestamp when anonymous usage reporting was first enabled and the cluster ID was created. - **`interval`**: The timestamp marking the start of the current reporting interval. - **`intervalPeriod`**: The length of the reporting interval, in seconds. ### Deployment and version information - **`target`**: The deployment mode or target for the Tempo instance, such as `all` for monolithic mode. - **`version`**: An object containing detailed version information: - **`version`**: The Tempo version, for example `v3.0.0`. - **`revision`**: The Git commit hash or revision used to build the binary. - **`branch`**: The Git branch used for the build. - **`buildUser`**: The user who built the binary. - **`buildDate`**: The date and time when the binary was built. - **`goVersion`**: The Go language version used for the build. ### Environment details - **`os`**: The operating system the Tempo instance is running on, such as `linux`. - **`arch`**: The system architecture, such as `arm64`. - **`edition`**: The edition of Tempo, such as `oss` for open source. ### Metrics and configuration - **`metrics`**: An object containing runtime metrics and configuration: - **`ring_kv_store`**: The key-value store used for the ring, for example `inmemory`. - **`memstats`**: Memory usage statistics, including heap and stack usage, garbage collection metrics, and total allocations. - **`num_cpu`**: The number of logical CPU cores available. - **`feature_enabled_multitenancy`**: Indicates if multitenancy is enabled (`1`) or not (`0`). - **`receiver_enabled_jaeger`**, **`receiver_enabled_otlp`**, **`receiver_enabled_kafka`**, **`receiver_enabled_zipkin`**: Flags indicating if each trace receiver is enabled (`1`) or not (`0`). - **`storage_block_format`**: The effective `storage.trace.block.version` value used for newly written blocks. Existing blocks may use older formats, and older Tempo versions omit this field. - **`storage_cache`**: The cache backend used for storage, if any. - **`feature_enabled_auth_stats`**: Indicates if authentication statistics are enabled. - **`frontend_version`**: The version of the frontend component. - **`storage_backend`**: The storage backend in use, such as `local`. - **`ring_replication_factor`**: The replication factor for the ring. - **`num_goroutine`**: The number of active Go routines. - **`cache_memcached`**, **`cache_redis`**: Flags indicating if Memcached or Redis caching is enabled. - **`distributor_bytes_received`**, **`distributor_spans_received`**: Objects showing the total and rate of bytes and spans received by the distributor. --- ### Sources Tempo Configuration Manifest (`docs/sources/tempo/configuration/manifest.md`) --- title: Manifest description: This manifest lists of all Tempo options and their defaults. weight: 110 --- [//]: # 'THIS FILE IS GENERATED AUTOMATICALLY BY go run pkg/docsgen/generate_manifest.go' [//]: # 'DO NOT EDIT THIS FILE DIRECTLY' # Manifest This document is a reference for all Tempo options and their defaults. If you are just getting started with Tempo, refer to [Tempo examples](https://github.com/grafana/tempo/tree/main/example/docker-compose) and other [configuration documentation](../). Most installations will require only setting 10 to 20 of these options. ## Complete configuration ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Sources Tempo Configuration Parquet (`docs/sources/tempo/configuration/parquet.md`) --- title: Apache Parquet block format menuTitle: Apache Parquet description: Learn about Parquet block format in Tempo. weight: 300 --- # Apache Parquet block format Tempo has a default columnar block format based on Apache Parquet. This format is required for tags-based search as well as [TraceQL](../../traceql/), the query language for traces. The columnar block format improves search performance and enables an ecosystem of tools, including [Tempo CLI](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#analyse-blocks), to access the underlying trace data. Starting in Tempo 3.1, Tempo writes new blocks in `vParquet5` by default. Existing `vParquet4` blocks remain readable. No data migration is required. ## Considerations The Parquet block format has been the default since Tempo 2.0 and is the only supported block format in Tempo 3.x. If you install using the [Tempo Helm charts](https://grafana.com/docs/tempo/<TEMPO_VERSION>/setup/helm-chart/), then Parquet is enabled by default. No data conversion or upgrade process is necessary. As soon as a block format version is enabled, Tempo starts writing data in that format, leaving existing data as-is. ## Block format versions Only Parquet-based formats are supported. ### Removed and deprecated block formats The `v2` block format has been removed in Tempo 3.0. The `vParquet3` block format has been removed in Tempo 3.2. Tempo can no longer read `vParquet3` blocks, and refuses to start if configured to write them. Convert any remaining `vParquet3` blocks with `tempo-cli` from Tempo 3.1 before you upgrade, or let them expire. Write new blocks in `vParquet5` (default) or `vParquet4`. ### vParquet5 `vParquet5` is the default block format as of Tempo 3.1. It builds on vParquet4 with the following improvements: - Expanded dedicated columns: Up to 20 dedicated string columns and 5 dedicated integer columns per scope (span, resource, and event), compared with 10 string columns per scope in vParquet4. - Event-scoped dedicated columns: Dedicated attribute columns can target event-scoped attributes such as `exception.message`. - Blob column support: High-cardinality or high-length string attributes (for example, stack traces or UUIDs) can use `zstd` compression instead of dictionary encoding for better efficiency. - Array-valued dedicated columns: Dedicated columns can store multiple values per attribute using the `options: ["array"]` configuration. - Materialized timestamp columns for faster metrics queries using common step intervals of 15s, 60s, 300s, and 1h, and any even multiples of those. The best value is chosen by automatically based on the query time range when the step interval is left as `auto`. For details on configuring dedicated attribute columns with vParquet5 features, refer to [Dedicated attribute columns](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/dedicated_columns/). ### vParquet4 `vParquet4` is available as an opt-in alternative to the default `vParquet5` format. `vParquet4` introduces columns that enable querying for data in array attributes as well as events and links. For more information, refer to [Dedicated attribute columns](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/dedicated_columns/). ## Change the block format version To change the block format version, set the `version` option in the [Storage section](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#storage) of the configuration file: ```yaml storage: trace: block: version: <version> ``` If you omit `version`, Tempo writes `vParquet5`. Replace `<version>` with `vParquet4` or `vParquet5`. To keep writing the previous `vParquet4` format, set the `version` option to `vParquet4`. ## Parquet configuration parameters Some parameters in the Tempo configuration are specific to Parquet. For more information, refer to the [storage configuration documentation](../#storage). ### Trace search parameters These configuration options impact trace search. | Parameter | Default value | Description | |---|---|---| | `[read_buffer_size_bytes: <int>]` | 10485676 | Size of read buffers used when performing search on a vParquet block. This value times the read_buffer_count is the total amount of bytes used for buffering when performing search on a Parquet block. | | `[read_buffer_count: <int>]` | 32 | Number of read buffers used when performing search on a vParquet block. This value times the read_buffer_size_bytes is the total amount of bytes used for buffering when performing search on a Parquet block. | The `cache_control` section contains the follow parameters for Parquet metadata objects: | Parameter | Default value | Description | |---|---|---| | <code>[footer: <bool> \| default = false]</code> | `false` | Specifies if the footer should be cached | | `[column_index: <bool> \| default = false]` | `false` | Specifies if the column index should be cached | | `[offset_index: <bool> \| default = false]` | `false` | Specifies if the offset index should be cached | --- ### Sources Tempo Configuration Polling (`docs/sources/tempo/configuration/polling.md`) --- title: Polling description: Learn about polling cycle configuration options in Tempo. weight: 700 aliases: - /docs/tempo/configuration/polling --- # Polling The polling cycle is controlled by a number of configuration options detailed here. ``` storage: trace: # How often to repoll the backend for new blocks. Default is 5m [blocklist_poll: <duration>] # Number of blocks to process in parallel during polling. Default is 50. [blocklist_poll_concurrency: <int>] # By default components will pull the blocklist from the tenant index. If that fails the component can # fallback to scanning the entire bucket. Set to false to disable this behavior. Default is true. [blocklist_poll_fallback: <bool>] # Maximum number of workers that should build the tenant index. All other components will download # the index. Default 2. [blocklist_poll_tenant_index_builders: <int>] # The oldest allowable tenant index. If an index is pulled that is older than this duration, # the polling will consider this an error. Note that `blocklist_poll_fallback` applies here. # If fallback is true and a tenant index exceeds this duration, it will fall back to listing # the bucket contents. # Default 0 (disabled). [blocklist_poll_stale_tenant_index: <duration>] ``` Due to the mechanics of the [tenant index](../../operations/monitor/polling/), the blocklist will be stale by at most twice the configured `blocklist_poll` duration. This 2x staleness bound is the basis for the minimum values of `query_backend_after` and `compacted_block_retention` described below. For a detailed explanation of how compaction interacts with the polling cycle, refer to [Compaction](../../operations/compaction/). ## How blocks reach object storage Block-builders consume trace data from Kafka and organize spans into blocks based on a configurable time window. Once a block is complete, the block-builder flushes it to object storage. After the block is written, queriers won't discover it until their next polling cycle completes. ## Handling blocklist staleness Two mechanisms ensure data availability despite blocklist staleness: - Live-stores cover the recent data gap - Compacted block retention prevents premature deletion ### Live-stores cover the recent data gap Live-stores serve recent trace data directly from Kafka, covering the window between when a block-builder flushes a new block and when queriers discover it through polling. The query-frontend uses `query_backend_after` to control when backend storage is searched: - Time ranges more recent than `query_backend_after` are searched only in live-stores. - Time ranges older than `query_backend_after` are searched in backend storage. `query_backend_after` must be set to at least twice the `blocklist_poll` duration. A newly flushed block will not appear in any querier's blocklist for up to `2 * blocklist_poll` after it is written. If `query_backend_after` is shorter than this window, the query-frontend may route requests to the backend before the block is discoverable, causing data gaps. With the default `blocklist_poll` of 5 minutes, `query_backend_after` should be at least 10 minutes. ``` query_frontend: search: # Time after which the query-frontend starts searching backend storage. Default is 15m. [query_backend_after: <duration>] ``` ### Compacted block retention prevents premature deletion The `compacted_block_retention` keeps a block in object storage for a period of time after it has been compacted and the data has been merged into a new block. This allows queriers with a stale blocklist to still access these blocks until they complete their polling cycles and have up-to-date blocklists. At a minimum, this should be twice the configured `blocklist_poll` duration. ``` backend_worker: compaction: # How long to leave a block in the backend after it has been compacted successfully. Default is 1h [compacted_block_retention: <duration>] ``` Additionally, the querier `blocklist_poll` duration needs to be greater than or equal to the worker `blocklist_poll` duration. Otherwise, a querier may not correctly check all assigned blocks and incorrectly return 404. It is recommended to simply set both components to use the same poll duration. --- ### Sources Tempo Configuration Tenant Ids (`docs/sources/tempo/configuration/tenant-ids.md`) --- aliases: - ../configuring/about-tenant-ids/ description: Learn about tenant IDs. menuTitle: Tenant IDs title: Tenant IDs weight: 40 --- # Tenant IDs Within a Grafana Tempo cluster, the tenant ID is the unique identifier of a tenant. Tenant IDs enable multi-tenancy in Tempo, allowing multiple organizations or tenants to share the same Tempo cluster while maintaining data isolation. Tempo uses the `X-Scope-OrgID` HTTP header to identify and enforce tenant boundaries. This header is set to the tenant ID value and is used for: - **Scoped writes (ingest)**: Each span is stored under its specified tenant, ensuring data isolation at the storage level - **Scoped reads (queries)**: Queries return only data belonging to the specified tenant For more information about setting up multi-tenancy, refer to [Enable multi-tenancy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/multitenancy/). ## Characters and length restrictions Tenant IDs must be less-than or equal-to 150 bytes in length. The length is measured in bytes, not characters, which is important for multi-byte characters. Tenant IDs can only include the following supported characters: - Alphanumeric characters (`0-9`, `a-z`, `A-Z`) - Special characters (`!`, `-`, `_`, `.`, `*`, `'`, `(`, `)`) All other characters, including slashes (`/`), backslashes (`\`), and whitespace, aren't supported. Tenant IDs are case-sensitive. For example, `Tenant-123` and `tenant-123` are treated as different tenant IDs. The tenant ID should not be empty. For security reasons, `.` and `..` aren't valid tenant IDs. These values are restricted to prevent path traversal attacks. ## Cross-tenant queries Cross-tenant queries are supported for search, search-tags, and trace-by-ID search operations. Refer to [Cross-tenant query federation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/cross_tenant_query/) for more information. ## Multitenancy Tenant IDs are transmitted to Tempo via the `X-Scope-OrgID` HTTP header. This header must be included in all requests to Tempo when multi-tenancy is enabled. Multi-tenancy on ingestion is supported with both gRPC and HTTP for OTLP (OpenTelemetry Protocol). You can add the header in: - OpenTelemetry Collector configuration - Grafana Alloy configuration - Any HTTP/gRPC client using `curl` or other relevant tools Refer to [Enable multi-tenancy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/multitenancy/) for more information. --- ### Sources Tempo Configuration Usage Tracker (`docs/sources/tempo/configuration/usage-tracker.md`) --- title: Usage tracker description: Learn how to configure the usage tracker for cost attribution. weight: 700 aliases: - /docs/tempo/configuration/usage-tracker --- # Usage tracker The usage tracker accurately tracks the amount of ingested traffic using a set of custom labels on a per-tenant basis, providing fine-grained control over your data. Use the `cost_attributes` option to configure usage trackers in the distributor, which expose metrics of ingested traffic grouped by configurable attributes exposed on `/usage_metrics`. ## Enable the usage tracker To use this feature, you need to enable it in the [distributor](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#distributor) and configure the overrides to handle the data. 1. Open your configuration file, for example, `tempo.yaml`. 1. In the `distributor` section, locate the `cost_attribution` line. If it is not present, then copy and paste the example below into your distributor section and update the options. ``` usage: cost_attribution: # Enables the "cost-attribution" usage tracker. Per-tenant attributes are configured in overrides. [enabled: <boolean> | default = false] # Maximum number of series per tenant. [max_cardinality: <int> | default = 10000] # Interval after which a series is considered stale and will be deleted from the registry. # Once a metrics series is deleted, it won't be emitted anymore, keeping active series low. [stale_duration: <duration> | default = 15m0s] ``` ### Configure overrides You also need to configure the dimensions to break down your the usage data in the standard overrides. In the overrides section, you can define attributes to group ingested data by and you can rename and combine attributes. For more information, refer to the [standard overrides](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#standard-overrides) section of the Configuration documentation. In this example, usage data is grouped by `service.name`. ``` # Overrides configuration block overrides: # Global ingestion limits configurations defaults: # Cost attribution usage tracker configuration cost_attribution: dimensions: service.name: "" ``` You can also configure per tenant in the [runtime-overrides](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#runtime-overrides) or in the [user-configurable-overrides](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#user-configurable-overrides). ## Review usage metrics Once everything is configured, the usage metrics are exposed in the distributor: ``` GET /usage_metrics ``` Example: ``` curl http://localhost:3200/usage_metrics # HELP tempo_usage_tracker_bytes_received_total bytes total received with these attributes # TYPE tempo_usage_tracker_bytes_received_total counter tempo_usage_tracker_bytes_received_total{service_name="article-service",tenant="single-tenant",tracker="cost-attribution"} 7327 tempo_usage_tracker_bytes_received_total{service_name="auth-service",tenant="single-tenant",tracker="cost-attribution"} 8938 tempo_usage_tracker_bytes_received_total{service_name="billing-service",tenant="single-tenant",tracker="cost-attribution"} 2401 tempo_usage_tracker_bytes_received_total{service_name="cart-service",tenant="single-tenant",tracker="cost-attribution"} 4116 tempo_usage_tracker_bytes_received_total{service_name="postgres",tenant="single-tenant",tracker="cost-attribution"} 3571 tempo_usage_tracker_bytes_received_total{service_name="shop-backend",tenant="single-tenant",tracker="cost-attribution"} 17619 ``` You can configure Prometheus to scrape this endpoint to collect usage metrics. Example: ```yaml scrape_configs: - job_name: 'tempo-usage' static_configs: - targets: ['<DISTRIBUTOR-HOST>:3200'] metrics_path: '/usage_metrics' ``` ## Scope dimensions by resource or span You can configure dimensions to look for attributes at specific scopes using prefixes: - `resource.<attribute>` - Look only at resource-level attributes - `span.<attribute>` - Look only at span-level attributes - `<attribute>` (no prefix) - Look at both resource and span levels Example: ```yaml overrides: defaults: cost_attribution: dimensions: resource.service.name: "service" # Scoped by resource, only look at `service.name` in resource attributes. span.db.system: "database" # Scoped by span, only look at `db.system` in span attributes. k8s.namespace.name: "namespace" # Unscoped, look at `k8s.namespace.name` in both resource and span attributes ``` {{< admonition type="note" >}} When using unscoped attributes (no prefix), span-level attributes will always overwrite resource-level attributes if both exist. This can lead to mixed values from both scopes, use scope prefixes (`resource.` or `span.`) for predictable behavior when attributes exist at both and you only want values from one of them. {{< /admonition >}} This table provides more information about the behavior: | Configuration | Resource has `service.name="api"` | Span has `service.name="worker"` | Result | |-------------------------|-----------------------------------|----------------------------------|------------| | `resource.service.name` | Uses `"api"` | Ignores span value | `"api"` | | `span.service.name` | Ignores resource value | Uses `"worker"` | `"worker"` | | `service.name` | Uses `"api"` initially | Overwrites with `"worker"` | `"worker"` | ## Configure target labels You can customize the label names that appear in the `/usage_metrics` metrics by configuring a custom label name in the dimensions configuration. ```yaml dimensions: <ATTRIBUTE_NAME>: "<LABEL_NAME>" ``` {{< admonition type="note" >}} The `_<LABEL_NAME>_` should be a [valid Prometheus label name](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels). If you provide an invalid label name, Tempo sanitizes it. {{< /admonition >}} Example: ```yaml overrides: defaults: cost_attribution: dimensions: service.name: "" # Results in: service_name span.service.name: "" # Results in: service_name resource.service.name: "" # Results in: service_name span.db.system: "" # Results in: db_system span.db.system: "database.type" # Results in: database_type k8s.namespace.name: "namespace" # Results in: namespace ``` --- ### Sources Tempo Configuration Use Trace Data (`docs/sources/tempo/configuration/use-trace-data.md`) --- title: Use tracing data in Grafana menuTitle: Use tracing data in Grafana description: Learn how to configure and query Tempo data with Grafana. aliases: - ./querying/ # /docs/tempo/<TEMPO_VERSION>/configuration/querying weight: 900 --- # Use tracing data in Grafana You can use Tempo as a data source in Grafana. Grafana Cloud comes pre-configured with a Tempo data source. If you are using Grafana on-prem, you need to [set up the Tempo data source](/docs/grafana/<GRAFANA_VERSION>/datasources/tempo). {{< admonition type="tip" >}} If you want to explore tracing data in Grafana, try the [Intro to Metrics, Logs, Traces, and Profiling example](../../getting-started/docker-example/). {{< /admonition >}} This video explains how to add data sources, including Loki, Tempo, and Mimir, to Grafana and Grafana Cloud. Tempo data source set up starts at 4:58 in the video. {{< youtube id="cqHO0oYW6Ic" >}} ## Configure the data source For detailed instructions on the Tempo data source in Grafana, refer to [Tempo data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/). To configure Tempo with Grafana: 1. Point the Grafana data source at your Tempo query frontend (or monolithic mode Tempo). 1. Enter the URL: `http://<tempo hostname>:<http port number>`. For most of [the Tempo examples](https://github.com/grafana/tempo/tree/main/example/docker-compose) the following works. The port of 3200 is a common port used in our examples. Tempo default HTTP port is 3200. ## Query the data source Refer to [Tempo in Grafana](https://grafana.com/docs/tempo/<TEMPO_VERSION>/visualize-traces/) for an overview about how tracing data can be viewed and queried in Grafana. For information on querying the Tempo data source, refer to [Tempo query editor](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/tempo/query-editor/). --- ### Sources Tempo Introduction Architecture (`docs/sources/tempo/introduction/architecture.md`) --- title: Tempo architecture description: Learn how Grafana Tempo ingests, stores, and queries trace data. keywords: - Grafana Tempo - architecture - components - traces aliases: - ../architecture # https://grafana.com/docs/tempo/<TEMPO_VERSION>/architecture/ - ../operations/architecture/ # /docs/tempo/latest/operations/architecture/ weight: 400 --- # Tempo architecture The journey of a trace is usually from an instrumented application, to a trace receiver/collector, to a trace backend store, and then those traces are retrieved and visualized in a tool like Grafana. Grafana Tempo acts in two major capacities: - Ingesting trace spans, sorting the span resources and attributes into columns in an Apache Parquet schema, before sending them to object storage for long-term retention. - Retrieving trace data from storage, either by specific trace ID or by search parameters using TraceQL. This page gives a high-level overview of how Tempo works: how it ingests, stores, and queries trace data, and the components that make up a Tempo deployment. For in-depth operational detail, refer to the [Tempo architecture reference](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/). ## Design goals The Tempo architecture is built around a few goals that shape how it ingests, stores, and queries traces. - Cost-efficient storage: Tempo stores all trace data in object storage. - Independent scaling: In microservices mode, Tempo separates the write path from the read path, so you can scale ingestion and query capacity independently. - Durability without extra replication: In microservices mode, a Kafka-compatible queue acts as a write-ahead log (WAL). After Kafka acknowledges a write, the data is durable, so Tempo doesn't replicate data across instances on the write path and can operate with a replication factor of 1 (RF1), which reduces cost. - Efficient attribute queries: Tempo stores blocks in Apache Parquet, a columnar format, so a query reads only the columns it needs instead of scanning entire traces. ## Deployment modes All Tempo components are compiled into the same binary; the `-target` parameter controls which components run in a given process, whether that's one process running everything or many processes each running one component. This lets you run Tempo in two modes: - Monolithic mode: All required components run in a single process (`-target=all`, the default). No Kafka is required; the distributor pushes trace data in-process to the live-store and metrics-generator. - Microservices mode: Each component runs as a separate process with its own `-target`. This mode requires a Kafka-compatible system and is recommended for production. Object storage is recommended for production in both modes; a local filesystem backend is available for development and testing. For more information, refer to [Deployment modes](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/deployment-modes/) and [Set up for tracing](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/). ## How data flows Tempo separates the write path, which gets trace data into storage, from the read path, which serves queries. ### Lifecycle of a write The following steps describe the write path in microservices mode. In monolithic mode, the distributor pushes data in-process to the live-store and metrics-generator instead of writing to Kafka. 1. Trace data is sent from the tracing pipeline (OpenTelemetry Collector, Grafana Alloy) to a **distributor**. 2. The distributor validates the request, shards traces by trace ID, and writes to **Kafka**. 3. Kafka acknowledges the write, and the distributor returns a response to the client. After the write is acknowledged, downstream components consume the data asynchronously: 4. **Live-stores** consume from Kafka and make data available for recent queries. 5. **Block-builders** consume from Kafka and build blocks for long-term object storage. 6. **Metrics-generators** (optional) consume from Kafka to derive metrics from traces. ### Lifecycle of a read 1. The **query frontend** receives a query and shards it into jobs for recent data and long-term storage. 2. **Queriers** retrieve recent data from **live-stores**. 3. For older data, queriers fetch blocks from **object storage**. 4. Results are aggregated and returned to the user. ## Components The following table summarizes the Tempo components and which deployment modes use them. | Component | Role | Used in | |---|---|---| | [Distributor](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/distributor/) | Receives and validates spans, then routes them to the write path | Both modes | | [Kafka-compatible queue](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/kafka/) | Durable write-ahead log between the distributor and downstream consumers | Microservices mode | | [Block-builder](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/block-builder/) | Builds Apache Parquet blocks and flushes them to object storage | Microservices mode | | [Live-store](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/live-store/) | Serves recent trace data for queries | Both modes | | [Query frontend](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/query-frontend/) | Receives queries and shards them into parallel jobs | Both modes | | [Querier](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/querier/) | Executes query jobs against live-stores and object storage | Both modes | | [Backend scheduler and worker](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/) | Handles compaction, retention, and blocklist maintenance | Both modes | | [Metrics-generator](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/metrics-generator/) | Optional; derives request rate, error rate, and duration (RED) metrics and service graphs from traces | Both modes | | [Object storage](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/object-storage/) | Long-term storage for all trace data | Both modes | ## Storage Tempo stores all trace data in object storage. It supports three major object storage APIs: - Amazon S3 (and S3-compatible systems, such as MinIO) - Google Cloud Storage (GCS) - Microsoft Azure Blob Storage A local filesystem backend is available for development and testing. Trace data is stored in Apache Parquet blocks, a columnar format that lets queries read only the attributes they need. Backend workers enforce retention by expiring data in object storage after the configured retention period. For more information, refer to [Object storage](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/object-storage/) and [Block format](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/block-format/). ## What you can query Tempo answers two kinds of read requests: lookups for a specific trace by trace ID, and searches across traces using [TraceQL](https://grafana.com/docs/tempo/<TEMPO_VERSION>/traceql/), the Tempo trace query language. You can also compute [TraceQL metrics](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/) (for example, span rates and latency quantiles) directly from trace data, and use the optional metrics-generator to produce rate, error, and duration (RED) metrics and service graphs. Tempo ingests traces in OpenTelemetry (OTLP), Jaeger, and Zipkin formats. Tempo is multi-tenant: trace data is isolated per tenant at the storage level. For more information, refer to [multi-tenancy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/multitenancy/). For the span fields and attributes you can query, refer to [Trace structure](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/trace-structure/). For AI-driven access to trace data through the MCP server and LLM-optimized APIs, refer to [Tempo and AI](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/tempo-and-ai/). ## Component details The following sections describe each component in a little more detail. For full operational reference, follow the link at the end of each section. ### Distributor The distributor is the entry point for trace data. It accepts spans in OTLP (recommended), Jaeger, and Zipkin formats and validates them against per-tenant ingestion limits. In microservices mode, it shards traces by trace ID and writes them to Kafka; in monolithic mode, it pushes data in-process to the live-store and metrics-generator. For more information, refer to [Distributor](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/distributor/). ### Kafka-compatible queue In microservices mode, Tempo uses a Kafka-compatible system (such as Apache Kafka or WarpStream) as a durable write-ahead log between the distributor and downstream consumers. Because Kafka provides durability after it acknowledges a write, Tempo can operate at a replication factor of 1. Kafka isn't used in monolithic mode. For more information, refer to [Kafka](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/kafka/). ### Block-builder The block-builder consumes trace data from Kafka, organizes spans into Apache Parquet blocks, and flushes them to object storage for long-term retention. It runs only in microservices mode; in monolithic mode, the live-store flushes blocks to object storage directly. For details, refer to [Block-builder](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/block-builder/). ### Live-store The live-store serves recent trace data, holding traces in memory and in a local write-ahead log so they're queryable within seconds of ingestion. In microservices mode, it consumes trace data from Kafka; in monolithic mode, it receives data directly from the distributor. For high availability, live-stores can be deployed across availability zones. For more information, refer to [Live-store](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/live-store/). ### Query frontend The query frontend is the entry point for queries. It receives TraceQL queries and trace ID lookups, shards each into parallel jobs that it distributes to queriers, and merges the returned results into a final response. For more information, refer to [Query frontend](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/query-frontend/). ### Querier The querier executes the jobs dispatched by the query frontend. It fetches recent data from live-stores and historical data from object storage, then returns results to the query frontend for merging. For more information, refer to [Querier](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/querier/). ### Backend scheduler and worker The backend scheduler and worker handle compaction, retention, and blocklist maintenance for data in object storage. The scheduler creates jobs and assigns them to workers, which compact small blocks into larger ones and expire data past its retention period. Together they replace the legacy compactor. For more information, refer to [Compaction](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/). ### Metrics-generator The metrics-generator is an optional component that derives rate, error, and duration metrics and service graphs from traces and remote-writes them to a metrics backend such as Prometheus or Grafana Mimir. For more information, refer to [Metrics-generator](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/metrics-generator/). ### Object storage Object storage is the long-term store for all trace data. Tempo supports Amazon S3 (and S3-compatible systems), Google Cloud Storage, and Azure Blob Storage, with a local filesystem backend for development and testing. For more information, refer to [Object storage](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/object-storage/). ## Things to keep in mind There are three fundamentally important things to keep in mind with traces: - There's no concept of the 'end' of a trace. A trace can start with any span which holds a unique trace ID that hasn't been seen by Tempo previously. However, spans can be continually added at any point in the future. - When a trace ID is queried in Tempo, it returns all of the currently stored/ingested spans that belong to that trace and presents them in a mapped response (that is, a graph of parent/child/sibling relationships between spans) that allows, for example, Grafana to then visually render those traces. - TraceQL queries return all matching traces from Tempo for the filters presented to it. This doesn't necessarily mean in chronological order, as some queriers may return results more promptly than others. When the maximum trace limit for a TraceQL query has been hit, the query frontend returns a response with the traces and spans it has collected so far and stops waiting for outstanding queriers. ## Next steps - Plan and deploy Tempo: [Set up for tracing](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/). - Explore the architecture in depth: [Tempo architecture reference](https://grafana.com/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/). - Learn the parts of a span: [Trace structure](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/trace-structure/). --- ### Sources Tempo Introduction Glossary (`docs/sources/tempo/introduction/glossary.md`) --- description: 'Glossary for traces' keywords: - Grafana - traces - tracing title: Glossary weight: 500 --- # Glossary The following terms are often used when discussing traces. Active series : {{< docs/glossary "active series" >}} Cardinality : {{< docs/glossary "cardinality" >}} Child span : A span that is nested within a parent span. Each child span represents a sub-operation or downstream call within the broader operation represented by its parent. A child span can itself be a parent of other child spans. Data source : {{< docs/glossary "data source" >}} Event attribute : A key-value pair attached to a span event, which represents a unique point in time during the span's duration. For more information, refer to [Span events](https://opentelemetry.io/docs/concepts/signals/traces/#span-events) in the OpenTelemetry documentation. Exemplar : {{< docs/glossary "exemplar" >}} Intrinsics : The core, built-in fields that are fundamental to the identity and lifecycle of spans and traces. Intrinsic fields include name, duration, status, and kind. These fields are defined by the OpenTelemetry specification and are always present. Link attribute : A key-value pair attached to a span link. A span link associates one span with one or more causally related spans. For more information, refer to [Span Links](https://opentelemetry.io/docs/concepts/signals/traces/#span-links) in the OpenTelemetry documentation. Log : {{< docs/glossary "log" >}} Metric : {{< docs/glossary "metric" >}} Resource attribute : A key-value pair that represents information about the entity producing a span, such as the cluster name, namespace, Pod, or container name. Root span : The first span in a trace, representing the initial request or operation. A root span has no parent span and serves as the top of the span tree. Semantic attribute : A standardized naming scheme for attributes shared across languages, frameworks, and runtimes, as defined by the OpenTelemetry specification. For more information, refer to [Trace Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/general/trace/) in the OpenTelemetry documentation. Span : {{< docs/glossary "span" >}} Span attribute : A key-value pair that contains metadata to annotate a span with information about the operation it tracks. For example, a span tracking an "add to cart" operation might include the user ID, item ID, and cart ID as span attributes. Spanset : {{< docs/glossary "spanset" >}} Trace : {{< docs/glossary "trace" >}} --- ### Sources Tempo Introduction Telemetry (`docs/sources/tempo/introduction/telemetry.md`) --- description: "Traces and telemetry" keywords: - Grafana - traces - tracing title: Traces and telemetry weight: 200 --- # Traces and telemetry Metrics, logs, traces, and profiles form the pillars of observability. Correlating between the four pillars of observability helps create a holistic view of your application and infrastructure. ## Metrics Metrics provide a high level picture of the state of a system. Metrics are the foundation of alerts because metrics are numeric values and can be compared against known thresholds. Alerts constantly run in the background and trigger when a value is outside of an expected range. This is typically the first sign that something is going on and are where discovery first starts. Metrics indicate that something is happening. ## Logs Logs provide an audit trail of activity from a single process that create informational context. Logs act as atomic events, detailing what's occurring in the services in your application. Whereas metrics are quantitative (numeric) and structured, logs are qualitative (textual) and unstructured or semi-structured. They offer a higher degree of detail, but also at the expense of creating significantly higher data volumes. Logs let you know what's happening to your application. ## Traces Traces add further to the observability picture by telling you what happens at each step or action in a data pathway. Traces provide a map of where something is going wrong. A trace provides a graphic representation of how long each step in the data flow pathway takes to complete. For example, how long a HTTP request, a database lookup, or a call to a third party service takes. It can show where requests initiate and finish, as well as how your system responds. This data helps you locate problem areas and assess their impact, often in places you never would have anticipated or found without this ability to trace the request flow. ## Profiles Profiles help you understand how your applications utilize compute resources such as CPU time and memory. This helps identify specific lines of code or functions to optimize and improve performance and efficiency. ## Why traces? Metrics in themselves aren't sufficient to find the root cause and solve complex issues. The same can be said for logs, which can contain a significant amount of information but lack the context of the interactions and dependencies between the different components of your complex environment. Each pillar of observability has its own unique strength when it comes to root causing issues. To get the most value of your observability strategy, you need to be able to correlate them. Traces have the unique ability to show relationships between services. They help identify which services are upstream from your service, which is helpful when you want to understand which services might be negatively impacted by problems in your service. Traces also help identify which services are downstream from your service. This is valuable since your application relies on their downstream services, and problems with those services may be the cause of elevated errors or latency reported by your service. For example, you can directly see the failing database and all impacted failing edge endpoints. Using traces and [exemplars](https://grafana.com/docs/grafana/next/fundamentals/exemplars/), you can go from a metric data point and get to an associated trace. Or from traces to logs: And vice versa, from logs to traces. --- ### Sources Tempo Introduction Tempo And Ai (`docs/sources/tempo/introduction/tempo-and-ai.md`) --- title: Tempo and AI description: Tempo exposes an MCP server and LLM-optimized APIs that let AI agents query, explore, and analyze distributed tracing data. keywords: - Grafana Tempo - AI - LLM - MCP - Model Context Protocol - gcx - agent weight: 450 --- # Tempo and AI Tempo exposes a Model Context Protocol (MCP) server and LLM-optimized API endpoints that let AI agents query and compare traces, compute metrics, and discover attributes directly. Grafana also provides command-line tools that connect agents to the broader observability platform. ## Model Context Protocol server The [MCP server](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/mcp-server/) at `/api/mcp` lets agents search for traces with TraceQL, retrieve or compare traces by ID, compute metrics from span data, and discover available attributes. The server also serves TraceQL syntax documentation as MCP resources, so agents can look up query syntax on demand instead of relying on training data. The MCP server uses the `streamable-http` transport and supports the same authentication and [multi-tenancy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/multitenancy/) as other Tempo API endpoints. {{< admonition type="warning" >}} The MCP server returns tracing data to the calling agent, which may forward it to an LLM provider. Consider the content of your tracing data and your organization's policies before connecting agents to this endpoint. {{< /admonition >}} To try it locally, refer to the [MCP server quick start](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/mcp-server/#quick-start). ## LLM-optimized API responses The [trace by ID v2](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#query-v2) and [tag values v2](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#search-tag-values-v2) endpoints accept an `Accept: application/vnd.grafana.llm` header. This returns a simplified JSON format that strips unnecessary detail and reduces token usage, so agents can process larger traces within their context window. {{< admonition type="caution" >}} The simplified LLM format is subject to change and shouldn't be relied on for programmatic use. {{< /admonition >}} Use this header for interactive or experimental agent use. For stable integrations, use the MCP server or request the standard JSON or protobuf formats. ## Tempo features relevant to AI workflows These Tempo features aren't AI-specific, but they directly improve what agents can do with tracing data. ### TraceQL metrics at general availability The MCP server's `traceql-metrics-instant` and `traceql-metrics-range` tools let agents compute rate, error, and duration metrics directly from trace data. Because [TraceQL metrics](https://grafana.com/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/) moved from experimental to GA in Tempo 3.0, these tools query a production-ready engine. ### OR conditions for tag autocomplete The [search tags v2](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#search-tags-v2) and [search tag values v2](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/#search-tag-values-v2) APIs support OR conditions, so autocomplete requests can match multiple values in a single call instead of making separate requests. When an agent discovers available attributes through the MCP tools, fewer round-trips mean faster, cheaper exploration of what's queryable. ### Control what agents can access Before you open up trace data to agents through the MCP server or API endpoints, you can control what's available. If your traces contain personally identifiable information or security tokens, use the Tempo CLI to remove affected traces from object storage first. [Redact traces](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#redact-traces) removes traces containing sensitive data from object storage. To remove specific traces directly from the CLI, [drop traces by ID](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#drop-traces-by-id). ## Grafana tools for AI workflows These capabilities aren't specific to Tempo. You have them by working in the Grafana product family, and they apply across the platform. Their AI value is composability: an agent that starts in Tempo's trace data can pivot outward to act on Grafana resources, chain investigations across other signals, and look up current documentation. For additional information about AI capabilities in Grafana, refer to the [AI and machine learning](https://grafana.com/docs/grafana-cloud/machine-learning/) documentation. ### `gcx` [`gcx`](https://github.com/grafana/gcx) is a CLI for managing Grafana resources, including dashboards, data sources, alerting rules, and Grafana Cloud products like Synthetic Monitoring, SLO, and Adaptive Telemetry. It works with Grafana Cloud, Grafana Enterprise, and Grafana OSS (v12 or later). An agent can use `gcx` alongside the Tempo MCP server to act on what it finds in traces. For example, after identifying a latency issue, the agent could query related Prometheus metrics or inspect alerting rules through `gcx` without leaving the terminal. ### Grafana Assistant CLI Grafana Assistant is an LLM-powered tool built into Grafana Cloud that queries data, builds dashboards, and helps you understand errors using natural language. It requires a Grafana instance where Grafana Assistant is enabled (for example, a Grafana Cloud stack). The [assistant-cli](https://github.com/grafana/assistant-cli) connects agents to Grafana Assistant through the Agent-to-Agent (A2A) API. This lets an agent chain investigations across observability signals from the terminal. For example, an agent could find a failing trace in Tempo, then use the `assistant-cli` to correlate the failure with error logs or query related metrics. ### Documentation as Markdown Grafana documentation is [available as Markdown](https://grafana.com/whats-new/2026-05-15-grafana-documentation-now-ships-as-markdown/), so agents can fetch current reference material instead of relying on potentially outdated training data. ## Next steps - [Set up the MCP server](https://grafana.com/docs/tempo/<TEMPO_VERSION>/api_docs/mcp-server/) to give agents access to your tracing data. - [Install gcx](https://github.com/grafana/gcx) to manage Grafana resources from the terminal or an agent workflow. - [Install the assistant-cli](https://github.com/grafana/assistant-cli) to connect agents to Grafana Assistant. --- ### Sources Tempo Introduction Trace Structure (`docs/sources/tempo/introduction/trace-structure.md`) --- description: Learn about the structure and parts of a span. keywords: - structure - spans - tracing title: Trace structure weight: 300 --- # Trace structure [//]: # 'Shared content for trace structures' [//]: # 'This content is located in /tempo/docs/sources/shared/trace-structure.md' {{< docs/shared source="tempo" lookup="trace-structure.md" version="<TEMPO_VERSION>" >}} --- ### Sources Tempo Metrics From Traces Where To Generate Metrics (`docs/sources/tempo/metrics-from-traces/where-to-generate-metrics.md`) --- title: Choose where to generate metrics from traces menuTitle: Where to generate metrics description: Learn how to choose between generating span metrics and service graphs in Tempo or in the collector pipeline. keywords: - metrics-generator - span metrics - sampling - Alloy weight: 50 --- # Choose where to generate metrics from traces You can generate metrics from traces in Grafana Tempo or in the collector pipeline. These metrics include RED metrics (rate, error, and duration) for request volume, failures, and latency, and service graphs that map how services call each other. Generating metrics from traces lets you build dashboards and alerts from your tracing pipeline without a separate metrics instrumentation path. Where you generate the metrics depends on how you sample traces before they reach Tempo. Sampling reduces the traces Tempo stores. It doesn't change the traffic your services produced. Refer to [Sampling](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/tail-sampling/) for head and tail sampling. The [metrics-generator](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-generator/) builds span metrics and service graphs from traces after ingest. Span metrics are request counts, durations, and error rates derived from individual spans. Service graphs are maps of relationships between services, built from parent-child span pairs. If a collector already dropped most of those traces, the metrics describe the sample, not the full system. Request rate, error rate, and latency no longer match the traffic your services produced. Service graphs miss relationships that never arrived. Alerts that depend on those series miss discarded traffic. If you send traces to Grafana Cloud Traces, you can use [Adaptive Traces](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/adaptive-telemetry/adaptive-traces/) instead of building and operating your own tail sampling pipeline. Adaptive Traces is a managed tail sampling capability that analyzes your trace data, recommends sampling policies (for example, keep traces with errors or high latency), and applies them for you. Adaptive Traces can also generate metrics from all received traces. ## Choose a generation path Use the table to choose a generation path, then follow the matching section. | Option | Use when | Tradeoff | | --- | --- | --- | | [Tempo metrics-generator](#use-the-metrics-generator) | Traces reach Tempo unsampled | One component generates metrics and collectors stay thin. Metrics include only the traces Tempo ingested. | | [Alloy or OpenTelemetry Collector](#generate-metrics-in-alloy-or-the-opentelemetry-collector) | You tail sample before traces reach Tempo | RED metrics and service graphs describe all traffic. Each collector replica emits series. | | [Both generators](#avoid-running-both) | You have a specific requirement for two generation paths | Duplicate active series, extra compute, and extra cost. | | [Scale ratio-based samples in Tempo](#scale-ratio-based-samples-in-tempo) | You sample at a fixed ratio and record that ratio on the span | Generation stays in Tempo. This option doesn't apply to tail sampling. | ## Use the metrics-generator The metrics-generator creates span metrics and service graphs after traces are ingested. Use it when traces reach Tempo unsampled. One component generates the metrics, so collectors stay thin and you don't emit the same series from every collector replica. That reduces active series and compute compared with generating metrics on every collector. Grafana Cloud Traces uses this model. Refer to [Use the metrics-generator to create metrics from spans](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics/span-metrics-metrics-generator/). For Grafana Cloud Traces, refer to [Metrics-generator in Grafana Cloud](https://grafana.com/docs/grafana-cloud/send-data/traces/metrics-generator/). ## Generate metrics in Alloy or the OpenTelemetry Collector Grafana Alloy and the OpenTelemetry Collector can create span metrics and service graphs in the collector pipeline, before sampling. Use this option when you tail sample before traces reach Tempo. Tail sampling evaluates a complete trace before deciding whether to keep or drop it. Generating metrics before the sampler keeps RED metrics and alerts aligned with all traffic, not only the traces Tempo stores. Alloy and the OpenTelemetry Collector provide the same generation capability in this context. Use the collector you already run. To generate metrics in the collector: 1. Configure the span metrics and service graph connectors in Alloy or the OpenTelemetry Collector. 1. Place the connectors ahead of the sampler so they see every span. For Alloy settings, refer to [Use Alloy to generate metrics from spans](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics/span-metrics-alloy/). For OpenTelemetry Collector settings, refer to [Generate metrics from spans](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/otel-collector/#generate-metrics-from-spans). For pipeline order, refer to [Pipeline workflows](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/tail-sampling/#pipeline-workflows). ## Avoid running both You can run a collector generator and Tempo metrics-generator at the same time. The two paths don't replace each other. Each path writes its own series, so you can end up with two sets of active series. Running both generators increases active series, compute, and Grafana Cloud cost. Use a single generation path unless you have a specific requirement. If you already run both, look for two sets of span metrics or service graph series in your metrics backend. Disable one path: turn off the metrics-generator processors in Tempo, or remove the span metrics and service graph connectors from the collector pipeline. ## Scale ratio-based samples in Tempo Ratio-based sampling keeps a fixed fraction of traces and can record that fraction on the span. When configured, Tempo multiplies the metric so counts match the unsampled population. Use this option when you sample at a fixed ratio, not when you tail sample. Generation stays in Tempo, and you don't need a collector-side generator. Refer to [Handling sampled traces](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics/span-metrics-metrics-generator/#handling-sampled-traces). --- ### Sources Tempo Operations Authentication (`docs/sources/tempo/operations/authentication.md`) --- title: Manage authentication menuTitle: Authentication description: Describes how to add authentication to Grafana Tempo. weight: --- # Manage authentication Grafana Tempo does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services. We recommend that in all [deployment modes](https://grafana.com/docs/tempo/<TEMPO_VERSION>/setup/deployment/) you add a reverse proxy in front of Tempo to direct client API requests to the various components. A list of open-source reverse proxies you can use: - [HAProxy](https://docs.haproxy.org/ ) - [NGINX](https://docs.nginx.com/nginx/) using their [guide on restricting access with HTTP basic authentication](https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-http-basic-authentication/) - [OAuth2 proxy](https://oauth2-proxy.github.io/oauth2-proxy/) - [Pomerium](https://www.pomerium.com/docs), which has a [guide for securing Grafana](https://www.pomerium.com/docs/guides/grafana) {{< admonition type="note" >}} When using Tempo in multi-tenant mode, Tempo requires the HTTP header `X-Scope-OrgID` to be set to a string identifying the tenant. It's assumed that clients setting `X-Scope-OrgID` are trusted clients, and the responsibility of populating this value should be handled by the authenticating reverse proxy. For more information, read the [multi-tenancy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/multitenancy/) documentation. {{< /admonition >}} --- ### Sources Tempo Operations Backend Search (`docs/sources/tempo/operations/backend_search.md`) --- title: Tune search performance menutitle: Tune search performance description: How to tune Tempo to improve search performance. weight: 500 aliases: - ./serverless_aws/ # https://grafana.com/docs/tempo/next/operations/serverless_aws/ - ./serverless_gcp/ # https://grafana.com/docs/tempo/next/operations/serverless_gcp/ --- # Tune search performance Regardless of whether or not you are using TraceQL or the original search API, Tempo searches all of the blocks in the specified time range. Depending on your volume, this may result in slow queries. This document explains how read path in Tempo works, what controls you have to tune the read path, details of the core configuration options, and how to tune these configuration options to get the most out of your Tempo cluster. The general advice is to scale your workers and queriers. Additional queriers can more effectively run jobs in parallel while additional workers more aggressively reduce the length of your blocklist and copies of data (if using `RF=3`). {{< admonition type="note" >}} All forms of search (TraceQL and tags based) require `vParquet` block formats. The v2 block format has been removed in Tempo 3.0. [Dedicated attribute columns](../dedicated_columns/) are another effective method to improve search performance. {{< /admonition>}} ## Before you begin You should understand the basic Tempo architecture. For more information, refer to the [Tempo architecture](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/architecture/#query-frontend) to learn about how Tempo works. ### Glossary query : A search query, issued by end user. For example, `{ traceDuration > 1s }` is a TraceQL query. job : A shard of a search query, the lowest unit of work. A query is broken down into multiple jobs and processed. batch : A group of jobs is called a batch. frontend : Another name for query-frontend querier : Responsible for executing (processing) the jobs, and sending the results back to query-frontend. ## Tempo query path Tempo has a query path made up of a query-frontend, querier, live-store, and backend. You can think about each component in a query path as a generic producer and worker model: * The query-frontend is a producer and it has multiple workers connected to it. The query-frontend takes a single query and shards it into multiple jobs (units of work). * The queriers are workers. They enqueue work from a queue, process it, and send the results back to the producer (query-frontend). * The querier either reads data from backend and processes the query, or it delegates the query to live-stores based on the type of job, time range, and query type. ### Lifecycle of a query A search request comes from the user, and it is split into multiple jobs in the query-frontend. These jobs are then added to a queue and queriers pick up jobs in batches to process. A single search query can create hundreds of jobs based on the time range and other variables. These jobs are batched and sent to queriers. The querier receives the batch and processes the jobs in the batch, builds results, and sends the results of the whole batch back to the query-frontend. The query-frontend merges and deduplicates the results, and sends them back to the client. When querier starts, it opens multiple connections to each query-frontend. Job batches are sent over these connections, and return the results. This process is synchronized so a single job batch can block a connection. The number of connections control the number of batches a querier processes concurrently. The number of connections is controlled by `querier.max_concurrent_queries` OR `frontend_worker.parallelism`. ## General guidelines Tuning the search pipeline can be difficult as it requires balancing a number of different configuration parameters. The below tips can you get your head around the general problem, but the specifics require experimentation. - Review the query-frontend logs for lines like the following to get a feeling for how many jobs your queries are creating: ``` level=info ts=2023-07-19T19:38:01.354220385Z caller=searchsharding.go:236 msg="sharded search query request stats and SearchMetrics" ... ``` - For a single TraceQL query the maximum number of parallel jobs is constrained by: - `query_frontend.search.concurrent_jobs`: This is the maximum number of jobs the frontend dispatches for one TraceQL query. - `# queriers * querier.max_concurrent_queries * query_frontend.max_batch_size`: This is the maximum job capacity of your Tempo cluster. If a given TraceQL query produces less jobs then these two values it should be executed entirely in parallel on the queriers. - Increasing `querier.max_concurrent_queries` is a great way to get more out of your queriers. However, if queriers are OOMing or saturating other resources then this should be lowered. Lowering `query_frontend.max_batch_size` will also reduce the total work attempted by one querier. ## Querier and query-frontend configuration Queriers and query-frontends have additional configuration related to search of the backend datastore. ### Querier ```yaml querier: # Control the amount of work each querier will attempt. The total number of # jobs a querier will attempt this is this value * query_frontend.max_batch_size max_concurrent_queries: 20 ``` ### Query-frontend [Query frontend](../../configuration/#query-frontend) lists all configuration options. These suggestions help deal with scaling issues. ```yaml server: # At larger scales, searching starts to feel more like a batch job. # Increase the server timeout intervals. http_server_read_timeout: 2m http_server_write_timeout: 2m query_frontend: # When increasing concurrent_jobs, also increase the queue size per tenant, # or search requests will be cause 429 errors. This is the total number of jobs # per tenant allowed in the queue. max_outstanding_per_tenant: 2000 # The number of jobs the query-frontend will batch together when passing jobs to the queriers. This value # This value * querier.max_concurrent_queries is your the max number of jobs a given querier will try at once. max_batch_size: 3 search: # At larger scales, increase the number of jobs attempted simultaneously, # per search query. concurrent_jobs: 2000 # The query-frontend will attempt to divide jobs up by an estimate of job size. The smallest possible # job size is a single parquet row group. Increasing this value will create fewer, larger jobs. Decreasing # it will create more, smaller jobs. target_bytes_per_job: 50_000_000 ``` ## Settings that are safe to increase without major impact. Scaling up queriers is a safe way to add more query capacity. At Grafana Labs, we prefer to scale queries horizontally by adding more replicas. If you see out of memory (OOM) errors, it might be worth scaling the queriers vertically. We recommend running at least two replicates of query-frontend. These replicas should be scaled vertically instead of horizontally. If you need to scale, scale query-frontends vertically by adding more CPU and RAM. Currently, query-frontends aren’t scaled horizontally, but this might change in the future. The reason we keep only two replicas is because each query-frontend has its own request queue and it also impacts the amount of jobs sent to each querier. If you add more query-frontends, you need to tune other configuration parameters to account for the change. We decided to keep query-frontend to two replicas for now. In a dedicated cluster, you can increase `query_frontend.max_outstanding_per_tenant` because the cluster is dedicated to a single customer. In a shared cluster, you need to use more caution when increasing `querier.max_outstanding_per_tenant` ### Parameter interactions The `query_frontend` configuration options control how Tempo shards a query into multiple jobs. These options also control the size of a single job, number of jobs, and the number of jobs added to the work queue at once (concurrently) for queriers to be picked up. The `max_batch_size` option controls how many jobs are sent to queriers at once in a single batch. Querier configuration options control processing of these jobs, queries operate at a job level. They're workers: they pick a job from the queue and execute the job, and return the results back to the query-frontend. Querier options control how many jobs it processes at once by opening concurrent connections to query-frontends, and each connection processes a single. ## Guidelines on key configuration parameters The following sections provide recommendations for key configuration parameters. ### `query_frontend.max_outstanding_per_tenant` parameter The `query_frontend.max_outstanding_per_tenant` parameter decides the maximum number of outstanding requests per tenant per frontend; requests beyond this error with HTTP `429`. This configuration controls how much load a single tenant can put on a cluster. A return of `429` indicates users to slow down. This configuration can be used to maintain quality of service (QoS) for tenants by asking tenants to slow down by sending `429` to the tenant that's overloading the system. #### Guidelines * In a dedicated cluster with one big tenant, it’s okay to increase the number for `query_frontend.max_outstanding_per_tenant.` * In a shared cluster with a lot of small tenants, keep the number small. * If a single tenant is overwhelming the whole cluster, you should decrease this parameter. It reduces the amount of work the tenant can enqueue at once. It starts returning `429` to the tenant. * If your tenants are complaining about getting `429`, you might want to increase this parameter and scale out queries to handle the query load. * If you increase other configurations that increase the query throughput or scale queriers, you might want to reduce this parameter to control the amount of work that can be enqueued. ### `query_frontend.max_batch_size` parameter The number of jobs to batch together in one HTTP request to the querier. Large search queries over a longer time range produce thousands of jobs. Instead of sending jobs one by one, batch the jobs and send them at once. The querier picks up and process them. Batching pushes jobs faster to the queriers, and reduce the time spent waiting around for jobs to be picked up by the querier. #### Guidelines * Default value of `max_batch_size` is set to `7`. * We DO NOT recommend changing the batch size from the default value of `7`. Based on testing at Grafana Labs, `7` is a good default. * If the batch size is big, you push more jobs at once but it takes longer for the querier to process the batch and return the results back. * Big batch size will increase the latency of the querier requests, and they might start hitting timeouts of 5xx, which will increase the rate of retries. * Bigger `max_batch_size` results in pushing too many jobs to the queriers. The jobs then have to be canceled if a query is exited early. ### `query_frontend.search.concurrent_jobs` parameter The number of concurrent jobs to execute when searching the backend. This controls how much work is produced concurrently. To put it another way, if a search job is sharded into 5000 jobs, and `concurrent_jobs` is set to 1000, then Tempo only executes 1000 jobs concurrently. The other 4000 jobs are processed one by one as the first 1000 jobs return the responses. If Tempo manages to answer the search query without executing all 5000 jobs, Tempo exits early and cancels the jobs that weren't started. #### Guidelines * This params controls the numbers of concurrent jobs to the number of jobs to put in the queue at once for processing a query. * If this number is low, big queries that create lots of jobs processed slowly. * If this number is high, the frontend pushes work faster to queries, and the frontend needs to cancel those jobs if the search exits early because the query is fulfilled. * If you see high load, spikes, or OOMs in the queriers, you can lower this configuration for a fair QoS and stability. * Having this number to a high value will allow a single tenant to push lots of work at once and overwhelm the querier. So keeping it to a lower number in a shared cluster allows for fair scheduling. * This parameter controls how fast jobs are pushed into the queue for queries to be picked up by queries * Adjusting this can impact your batch size so that metric should be watched while making changes to this configuration, and make sure that actual batch size is around the `max_batch_size`. * We recommend keeping it high in dedicated clusters with a single big tenant. * We recommend keeping it low in shared clusters because it ensures that a single tenant can't overwhelm the read path. ### `query_frontend.max_retries` parameter This option controls the number of times to retry a request sent to a querier. We only retry on 5xx from a queriers (_translated from the equivalent gRPC error_). #### Guidelines * This option controls the retries from query-frontend to queriers. * High `max_retries` amplifies the slow queriers and result in a lot more load then required. * Low `max_retries` can lead to users having bad experience due failed queriers, because a job failed and wasn’t retried. * If there are too many retries in a cluster, it can be a sign that queriers are struggling to finish work and needs to be scaled up. * Big jobs (high `target_bytes_per_job` or `max_batch_size`) can result in increased retries. Make the jobs smaller by reducing the `target_bytes_per_job` or `max_batch_size` to reduce the retries. If not, scale up the queriers. * Don't set `query_frontend.max_retries` to a high number. The default of `2` is a good default. * Higher retries impact the backend when you have a big query that’s generating lots of jobs. * High `max_retries` can overwhelm the queriers when they're under heavy load and failing to process the jobs. Retries can snowball, and degrade the query performance. ### `query_frontend.search.target_bytes_per_job` parameter The target number of bytes for each job to handle when performing a backend search. This parameter controls the size of a single job, and size of a job dictates how much data a single job reads. You can tune this to make big or small jobs. This option controls the upper limit on the size of a job, and can be used as a proxy for how much data a single search job scans or the amount of work a querier needs to do to process a job. #### Guidelines * Setting this to a small value produces too many jobs, and results in more overhead, setting it too high produces big jobs. Queriers might struggle to finish those jobs and it can lead to high latency. * In testing at Grafana Labs, 100MB to 200MB is a good range for this configuration, and works across different sizes of clusters. * We recommend keeping this fixed within the recommended range. ### `querier.frontend_worker.parallelism` parameter Number of simultaneous queries to process per query-frontend or query-scheduler. This configuration controls the number of concurrent requests per query-frontend a querier process. If parallelism is set to 5 and there are two query-frontends running, then a querier process opens 5 connections to each query-frontend, and in total of 10 connections. One batch (or job, if batching is disabled) is processed per connection, so this controls the maximum number of concurrent jobs processed. A single batch is processed in sync and size of the batch is controlled by `max_batch_size`, a connection is blocked until it's process and returns the result of a batch. You can also disable it and use `max_concurrent_connections` but we use this to make sure we queriers are picking up work from both query-frontends and jobs are being scheduled fairly. If you want to make sure you always have connections defined in this config, you should set `match_max_concurrent: false` in your configuration to ensure you are not limited by `max_concurrent_connections`. #### Guidelines * Maximum number of jobs a querier processes when using parallelism is equal to `parallelism * query_frontend replicas * max_batch_size`. * As you add more queriers, connections to an individual frontend increase. Query-frontend has a shared lock on these connections so if you see issues when you scale out queriers, lower the parallelism to reduce contention. * It’s recommended to set `match_max_concurrent: false` and not set `max_concurrent_queries` when using parallelism. ### `querier.max_concurrent_queries` parameter This controls the maximum number of jobs a querier processes concurrently. It doesn't distinguish between the types of queries. These jobs can be from two different queries, or different tenants as well. Querier processes these in sync, blocks the connection, and once it returns the results, it picks up a new job from the query-frontend queue. #### Guidelines * The `max_concurrent_queries` parameter controls the number of queries that are processed by querier at once (concurrently). * if queriers are under-utilized, increase this value to process more jobs concurrently and process the queries faster. * If queriers are over utilized, seeing spikes, or OOMs, you can reduce this config to do less work per queriers and scale out queriers to add more capacity to the cluster. * This setting depends on the resources, in small queriers, it should be set to a lower number. * If you have large queriers, this configuration should be set to a higher value to fully utilize the capacity of the querier. ## Querier memory sizing If you are seeing that your queries are using more memory then you prefer, reduce the amount of work a querier is doing concurrently. That reduces resource usage. If you want your queries to use more memory then they're currently using, increase the amount of work a querier is doing concurrently and it increases the resource usage. You can tune `query_frontend.search.target_bytes_per_job`, `querier.frontend_worker.parallelism`, and `querier.max_concurrent_queries` to tune the amount of work a querier is doing concurrently. Queriers memory request roughly translates to job size times the concurrent work and some buffer. ## Scaling cache Tempo can be configured to use multiple caches for different types of data. When configured, Tempo uses these caches to improve the performance of queries. Here are few general heuristics to know when to scale a cache: * Look at cache latency. If cache latency is hitting `cache_timeout`, that means that cache is under scaled and it’s taking too long to read or write, scale the cache. * Scale the cache if a cache has a high eviction rate. The cache might be under provisioned. * Lower level cache like bloom cache, parquet-page cache, parquet-footer cache sees higher hit rates (usually around 90% of above). If you have a consistent query traffic and these lower level caches have low hit rate, they're undervalued and needs to be scaled up. * Higher level cache like frontend-search cache has a low hit rate and is only useful when the same query is being repeated. Size these according to the amount of data you want you want to cache * Cache sizes are also dictated by how much data you want to cache at each tier, it’s better to cache more at lower level caches because they have higher hit rate and is useful across queries. --- ### Sources Tempo Operations Caching (`docs/sources/tempo/operations/caching.md`) --- title: Improve performance with caching menuTitle: Improve performance with caching description: Learn how to improve query performance by using caching. weight: 600 --- # Improve performance with caching Tempo uses an external cache to improve query performance. Tempo supports [Memcached](https://memcached.org/) and [Redis](https://redis.io/) (experimental). ## Cache roles Tempo caches different types of data, each assigned a **role**. You configure one or more cache instances under the top-level `cache:` block and assign roles to each instance. This lets you size and tune each cache independently based on the workload it handles. | Role | What it caches | Volume | |---|---|---| | `bloom` | Bloom filters used for trace ID lookup. | Moderate | | `trace-id-index` | Trace ID index used to locate traces within blocks. | Moderate | | `parquet-footer` | Parquet file footer metadata. Useful for both search and trace-by-ID queries. | Low | | `parquet-column-idx` | Parquet column index sections. | Low | | `parquet-offset-idx` | Parquet offset index sections. | Low | | `parquet-page` | Parquet data pages. Caches most Parquet reads. | **High** | | `frontend-search` | Query-frontend search job outcomes. | Varies | You can assign multiple roles to a single cache instance, or split high-volume roles (like `parquet-page`) onto a dedicated instance. For example, you might use a large Memcached pool for `parquet-page` and a smaller one for `bloom` and `parquet-footer`. For configuration parameters and an example, refer to the [Cache](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#cache) section of the Tempo configuration reference. For information about search performance, refer to [Tune search performance](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/backend_search/). ## Monitor cache item sizes Tempo emits the `tempodb_cache_store_size_bytes` histogram for every item written to the tempodb backend cache. The metric is labeled by `role`, so you can compare item sizes for backend cache roles such as `bloom`, `trace-id-index`, `parquet-footer`, `parquet-column-idx`, `parquet-offset-idx`, and `parquet-page`. Use this metric when you tune Memcached slab classes or decide whether to split high-volume cache roles onto a dedicated cache. For example, chart the P95 cache item size by role: ```promql histogram_quantile( 0.95, sum by (role, le) ( rate(tempodb_cache_store_size_bytes_bucket[5m]) ) ) ``` If a role regularly writes larger items than the cache can store efficiently, move that role to a cache sized for larger objects or adjust the cache configuration before increasing query concurrency. ## Monitor cache hit ratio Tempo emits two counters that report the cache traffic Tempo itself drives, labelled by `role` (the cache role, such as `bloom`, `parquet-page`, and so on) and `outcome` (`hit` or `miss`): - `tempodb_cache_requests_total{role,outcome}` counts cache lookups. - `tempodb_cache_request_bytes_total{role,outcome}` counts the bytes returned on `hit` and the bytes fetched from the backend on `miss`. Use these to track the live hit ratio per cache role and decide where to grow capacity. For example, the hit ratio for `parquet-page` over the last five minutes: ```promql sum(rate(tempodb_cache_requests_total{role="parquet-page",outcome="hit"}[5m])) / sum(rate(tempodb_cache_requests_total{role="parquet-page"}[5m])) ``` A persistently low hit ratio on a high-volume role (commonly `parquet-page`) is a signal to size up that cache pool or split it onto a dedicated cache instance. Pair the request counter with `tempodb_cache_request_bytes_total` to see whether misses are dominated by many small lookups or by a few large ones. ## Memcached Memcached is used by default in the Tanka and Helm examples. Refer to [Deploy Tempo](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/). ### Connection limit As a cluster grows in size, the number of instances of Tempo connecting to the cache servers also increases. By default, Memcached has a connection limit of 1024. Memcached refuses connections when this limit is surpassed. You can resolve this issue by increasing the connection limit of Memcached. You can use the `tempo_memcache_request_duration_seconds_count` metric to observe these errors. For example, by using the following query: ```promql sum by (status_code) ( rate(tempo_memcache_request_duration_seconds_count{}[$__rate_interval]) ) ``` This metric is also shown in [the monitoring dashboards](../monitor/) (the left panel): Note that the already open connections continue to function. New connections are refused. Additionally, Memcached logs the following errors when it can't accept any new requests: ``` accept4(): No file descriptors available Too many open connections accept4(): No file descriptors available Too many open connections ``` When using the [memcached_exporter](https://github.com/prometheus/memcached_exporter), you can observe the number of open connections at `memcached_current_connections`. ## Cache size control Tempo querier accesses bloom filters of all blocks while searching for a trace. This essentially mandates the size of cache to be at-least the total size of the bloom filters (the working set). However, in larger deployments, the working set might be larger than the desired size of cache. When that happens, eviction rates on the cache grow high, and hit rate drop. Tempo provides two configuration parameters to filter down on the items stored in cache. ``` # Min compaction level of block to qualify for caching bloom filter # Example: "cache_min_compaction_level: 2" [cache_min_compaction_level: <int>] # Max block age to qualify for caching bloom filter # Example: "cache_max_block_age: 48h" [cache_max_block_age: <duration>] ``` Using a combination of these configuration options, you can narrow down on which bloom filters are cached, thereby reducing the cache eviction rate, and increasing the cache hit rate. In order to decide the values of these configuration parameters, you can use a cache summary command in the [tempo-cli](../tempo_cli/) that prints a summary of bloom filter shards per day and per compaction level. The outcome looks something like this: This image shows the bloom filter shards over 14 days and 6 compaction levels. This can be used to decide the configuration parameters. --- ### Sources Tempo Operations Compaction (`docs/sources/tempo/operations/compaction.md`) --- title: Compaction description: Learn how Tempo compacts blocks in the backend and how to configure the compaction process. weight: 200 --- # Compaction Tempo stores trace data as immutable blocks in object storage. Over time, many small blocks accumulate from frequent flushes. Compaction merges groups of small blocks into fewer, larger blocks, reducing storage costs and improving query performance by limiting the number of blocks a querier must scan. ## Block states Every block has one of two states, signaled by which metadata file is present in the backend: - **Live** (`meta.json`): the block is queryable and included in query plans. - **Compacted** (`meta.compacted.json`): the block's data has been merged into a new block. The file is renamed from `meta.json` to `meta.compacted.json` at the time of compaction. Blocks are never deleted immediately after compaction. Because the blocklist is polled periodically, queriers may not yet know about the new merged block when the old blocks are first marked compacted. The `compacted_block_retention` window keeps the old blocks readable until all pollers have caught up. ## Blocklist and poller Tempo components rely on an in-memory blocklist to know which blocks exist in object storage. They do not scan object storage directly on each request. The blocklist is maintained by the poller, a background process that reads the backend on a fixed interval controlled by `blocklist_poll` (default 5 minutes). The poller is how compaction results become visible to queriers and other components: newly merged blocks appear in the blocklist and compacted blocks are removed. A component can only serve data from blocks the poller has discovered. If the poller falls behind or fails, the component's view of the backend becomes incomplete. The poller reads a pre-built tenant index from the backend rather than scanning all objects on each cycle. If the tenant index is missing or older than `blocklist_poll_stale_tenant_index`, the component falls back to scanning the backend directly. Setting `blocklist_poll_fallback: true` (the default) enables this fallback; disabling it means a stale or missing index causes the poll to fail rather than recover. Because the poller runs on an interval, the blocklist is always slightly out of date. In a healthy system, the blocklist will be stale by at most twice the configured `blocklist_poll` duration. This is because a writer may update the backend at any point within a polling cycle, and the reader may have just started its own cycle at that same moment. ## Timing requirements The 2x staleness bound drives two concrete configuration requirements. ### query_backend_after must be at least 2x blocklist_poll The query-frontend uses `query_backend_after` to decide where to search: - Traces more recent than `query_backend_after` are served from live-stores only. - Traces older than `query_backend_after` are served from the backend. A newly flushed block will not appear in the querier's blocklist until up to `2 * blocklist_poll` after it was written. During that window, the block would be missed if the query-frontend routed the request to the backend. Setting `query_backend_after` to at least `2 * blocklist_poll` ensures that such blocks are still covered by live-stores and are not yet expected to appear in the backend. With the default `blocklist_poll` of 5 minutes, `query_backend_after` should be at least 10 minutes. The default of 15 minutes for search provides a comfortable margin. ### compacted_block_retention must be at least 2x blocklist_poll When a block is compacted, its data is merged into a new block and the old block is marked compacted. A querier with a stale blocklist may still reference the old block until it completes its next poll cycle. Setting `compacted_block_retention` to at least `2 * blocklist_poll` ensures those queriers can still read the old block before it is deleted. The default of 1 hour is well above the minimum for typical configurations. ## Block selection Before any compaction work begins, the block selector determines which blocks to compact together. The time window block selector groups blocks by their time range and compaction level. Blocks are only compacted with other blocks that share the same compaction window, data encoding, format version, and dedicated column configuration. The selector divides the blocklist into two zones based on the **active window** (the most recent 24 hours): - **Inside the active window**: blocks are grouped by compaction level, then by time window. Lower compaction levels are prioritized, and within each group the smallest blocks are selected first. - **Outside the active window**: blocks are grouped by time window only, ignoring compaction level. This allows older data to be consolidated regardless of how many times it has already been compacted. The selector only returns a group for compaction if it contains at least the minimum number of input blocks (default 2). Each input set is also bounded by `max_block_bytes` (default 100 GB). If a full group would exceed this limit, the selector picks the largest contiguous subset of blocks that fits; any remaining blocks in the group are eligible in subsequent compaction runs. The number of input blocks per compaction job is configurable via `min_input_blocks` and `max_input_blocks`, defaulting to a minimum of 2 and a maximum of 4. ## How compaction works Compaction is handled by the backend scheduler and backend workers. The scheduler is a singleton that assigns compaction jobs to a pool of workers. Workers request jobs from the scheduler and report back when complete; the scheduler tracks active jobs and avoids handing out overlapping work. The scheduler uses a priority queue to choose which tenant to work on next. Outstanding blocks are those the block selector has identified as eligible for compaction -- blocks that meet the window, encoding, and minimum-group-size requirements. Tenants are ranked by the sum of their total blocklist length and their outstanding block count, so tenants with more overall backend work receive jobs first. A tenant with no outstanding blocks receives a priority of zero and is not scheduled. The scheduler processes one tenant at a time, issuing jobs until the tenant's eligible blocks are exhausted or the per-tenant job limit (`max_jobs_per_tenant`, default 1000) is reached, then moves to the next tenant. This cap is the mechanism that prevents a single busy tenant from monopolizing the workers. Within a tenant, the block selector prioritizes lower compaction levels first. Higher-level blocks are still compacted, but after lower-level work is cleared. The compaction provider waits for a poll notification before selecting new work, preventing idle spin when no blocks are eligible. A minimum cycle interval (`min_cycle_interval`, default 30 seconds) rate-limits tenant prioritization when the queue is empty. ### Autoscaling Outstanding block counts are measured for all tenants on a periodic ticker (`measure_interval`, default 1 minute), giving an aggregate view of total backend work. Autoscaling thresholds should be set against this aggregate total rather than per-instance averages. For a reference KEDA autoscaling configuration driven by outstanding block metrics, see `operations/jsonnet/microservices/autoscaling.libsonnet`. ## Configuration reference For a full configuration reference, refer to the [configuration documentation](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/). The following fields are most relevant to compaction behavior. ### Polling interval (`storage.trace`) ```yaml storage: trace: # How often to poll the backend for new blocks. Default: 5m. blocklist_poll: 5m ``` ### Compaction settings (`backend_worker.compaction`) ```yaml backend_worker: compaction: # Size of the time window used to group blocks. Default: 1h. compaction_window: 1h # Maximum size of a compacted output block in bytes. Default: 100GB. max_block_bytes: 107374182400 # How long to keep a block after compaction. Must be >= 2x blocklist_poll. Default: 1h. compacted_block_retention: 1h ``` ### Query cutoff (must be >= 2x blocklist_poll) ```yaml query_frontend: search: # Default: 15m. query_backend_after: 15m ``` ### Scheduler compaction (`backend_scheduler.provider.compaction`) ```yaml backend_scheduler: provider: compaction: # Maximum jobs to issue per tenant before switching. Default: 1000. max_jobs_per_tenant: 1000 # Minimum number of input blocks per job. Default: 2. min_input_blocks: 2 # Maximum number of input blocks per job. Default: 4. max_input_blocks: 4 # Minimum time between tenant selection cycles. Default: 30s. min_cycle_interval: 30s ``` {{< admonition type="note" >}} The `backend_worker.compaction` configuration block contains two fields that are not used by the current scheduled compaction path: `max_time_per_tenant` and `compaction_cycle`. These fields were used by the ring-based compaction loop, which has been removed. They are accepted by the configuration parser but have no effect. {{< /admonition >}} ## See also - [Polling configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/polling/) - [Monitor backend polling](../monitor/polling/) --- ### Sources Tempo Operations Dedicated Columns (`docs/sources/tempo/operations/dedicated_columns.md`) --- title: Dedicated attribute columns description: Learn how to use dedicated attribute columns to improve query performance. weight: 400 --- # Dedicated attribute columns Dedicated attribute columns improve query performance by storing the most frequently used attributes in their own columns, rather than in the generic attribute key-value list. Dedicated attribute columns are available when using `vParquet4` or later block formats. `vParquet5` expands on the feature with doubled string column limits, integer dedicated columns, event-scoped attributes, array-valued attributes, and blob attributes using the `options` field. As of Tempo 3.1, Tempo writes `vParquet5` by default, so these expanded limits apply without setting `storage.trace.block.version`. ## Configuration You can configure dedicated attribute columns in the storage block or via overrides. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Priority is given to the most specific configuration, so tenant-specific overrides will take precedence over global overrides. Similarly, default overrides take precedence over storage block configuration. ## Usage The number of dedicated attribute columns depends on the block format version: - **vParquet4**: Up to 10 string attributes per scope (span and resource). Integer dedicated columns and event scope are not supported. - **vParquet5**: Up to 20 string attributes and 5 integer attributes per scope (span, resource, and event). As a rule of thumb, good candidates for dedicated attribute columns are attributes that contribute the most to the block size, even if they aren't frequently queried. Reducing the generic attribute key-value list size significantly improves query performance. ### Integer attribute selection Integer dedicated columns are most effective when the attribute is present on at least 5% of rows in a scope. The `tempo-cli analyse block` command shows the percentage of rows for each integer attribute and marks candidates that meet this threshold with a recommendation. Sparse integer attributes (below 5% prevalence) typically perform better in the generic attribute storage. ### Array-valued attributes Dedicated attribute columns support array-valued attributes. This is useful for attributes that can have multiple values, such as HTTP headers or custom tags. To enable array support, add `options: ["array"]` to the dedicated attribute column configuration. The `options` field is only available in `vParquet5` and later. Earlier versions ignore this field and only support single-valued attributes. When `options: ["array"]` is set, the dedicated column stores multiple values per attribute. Without this option (default), only single values are supported. ```yaml storage: trace: block: version: vParquet5 parquet_dedicated_columns: # Single-valued string attribute (default behavior) - name: http.method type: string scope: span # Array-valued string attribute - name: http.request.header.accept type: string scope: span options: ["array"] # Array-valued integer attribute - name: custom.retry.counts type: int scope: span options: ["array"] ``` ### Blob attributes Blob attributes are designed for high-cardinality or high-length string values where dictionary encoding becomes inefficient. Examples include UUIDs, stack traces, request bodies, or any attribute with many unique values. When an attribute's dictionary size per row group exceeds a threshold (default 4MiB), the column should be marked as a blob. The `tempo-cli analyse block` command can help identify blob candidates by showing the estimated dictionary size per row group. To enable blob mode, add `options: ["blob"]` to the dedicated attribute column configuration. Blob columns use `zstd` compression instead of dictionary encoding, which is more efficient for high-cardinality data. ```yaml storage: trace: block: version: vParquet5 parquet_dedicated_columns: # Standard dedicated column (uses dictionary encoding) - name: http.method type: string scope: span # Blob column for high-cardinality data - name: db.statement type: string scope: span options: ["blob"] # Blob column for stack traces - name: exception.stacktrace type: string scope: event options: ["blob"] ``` Use the `tempo-cli analyse block` command with the `--blob-threshold` option to identify attributes that should be configured as blobs. Attributes exceeding the threshold are marked as "(blob)" in the output. ### Event-scoped attributes With `vParquet5`, dedicated columns support event-scoped attributes in addition to span and resource scopes. Event-scoped columns are useful for frequently queried event attributes such as exception details or custom event data. ```yaml storage: trace: block: version: vParquet5 parquet_dedicated_columns: # Event-scoped string attribute - name: exception.message type: string scope: event # Event-scoped attribute with blob option - name: exception.stacktrace type: string scope: event options: ["blob"] ``` ### Tempo-cli You can use the `tempo-cli` tool to find good candidates for dedicated attribute columns. The `tempo-cli` provides the commands `analyse block <tenant-id> <block-id>` and `analyse blocks <tenant-id>` that will output the top N attributes by size for a given block or all blocks in a tenant. Example: ```bash tempo-cli analyse blocks --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant ``` Refer to the [tempo-cli documentation](../tempo_cli/) for more information. --- ### Sources Tempo Operations Manage Trace Ingestion (`docs/sources/tempo/operations/manage-trace-ingestion.md`) --- title: Manage trace ingestion menuTitle: Manage trace ingestion description: Size ingestion limits, fix refused spans, and identify what is driving trace volume in Tempo. weight: 250 --- # Manage trace ingestion If you are seeing `RATE_LIMITED`, `LIVE_TRACES_EXCEEDED`, or `TRACE_TOO_LARGE` errors, or if your trace storage costs are rising unexpectedly, this page can help. Grafana Tempo enforces ingestion limits at multiple points in the write path. The distributor checks rate limits before writing spans to Kafka. Downstream, live-stores enforce per-trace size and live trace count limits asynchronously, and block-builders enforce per-trace size limits. If limits are too low for your workload, spans are discarded and data is lost. If limits are unchecked, ingestion volume can grow beyond what you intended. This page covers three tasks: - [Size ingestion limits](#size-ingestion-limits-for-your-workload) when deploying or reviewing your configuration. - [Find and fix discarded spans](#find-and-fix-discarded-spans) when you are actively losing data. - [Identify what is driving ingestion volume](#identify-what-is-driving-ingestion-volume) when costs are growing. For an overview of how trace data flows through the write path, refer to [Tempo architecture](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/architecture/). ## Size ingestion limits for your workload Tempo enforces three ingestion limits. Understanding what the defaults mean for your traffic helps you set appropriate values at deploy time, rather than discovering them through production incidents. ### Rate limit `rate_limit_bytes` (default: `15,000,000`) sets the sustained byte rate each distributor allows per tenant, measured in bytes per second. For a typical span size of around 500 bytes, the default accommodates roughly 30,000 spans per second per distributor. How this scales depends on your `rate_strategy`: - **`local`** (default): each distributor enforces the limit independently. With three distributors, the effective cluster limit is approximately 90,000 spans per second. - **`global`**: the configured rate is shared across all distributors. The total cluster rate equals the configured value regardless of how many distributors you run. `burst_size_bytes` (default: `20,000,000`) allows temporary spikes above the sustained rate, for example during application deployments. The burst allowance is always applied locally, regardless of rate strategy. ### Live trace limit `max_traces_per_user` (default: `10,000`) caps the number of concurrently active traces per tenant on each live-store. This limit is enforced asynchronously in the live-store, not at ingestion time in the distributor. Block-builders do not enforce this limit. If your services produce many short-lived traces in parallel, you may need to raise this. {{< admonition type="note" >}} The `max_global_traces_per_user` setting, which provides a cluster-wide cap for the ingester write path, has been moved to `ingestion.max_global_traces_per_user` in Tempo 3.0. {{< /admonition >}} ### Per-trace size limit `max_bytes_per_trace` (default: `5,000,000`) caps the total size of a single trace. This limit is enforced asynchronously in live-stores and block-builders. Traces that exceed this limit are partially dropped. Unusually large traces often indicate a retry loop or misconfigured instrumentation rather than normal application behavior. Use the `tempo_distributor_trace_size_bytes` histogram to observe the size distribution of every trace received by the distributor, per tenant. Unlike the limits above, this histogram records all traces, so you can see how close traces get to the limit before they are dropped. The distributor sees each push request separately, so a trace sent across many requests shows up as several smaller samples. Use the `tempodb_compaction_trace_size_bytes` histogram to observe the size of combined traces written by compaction, per tenant and compaction level. Each compaction level rewrites the trace, so filter to the highest level to approximate the final trace size. ### Example configuration To estimate the rate limit you need, multiply your average span size by your peak spans-per-second across all services for a given tenant. The following example raises the defaults for a high-throughput workload: ```yaml overrides: defaults: ingestion: rate_strategy: local rate_limit_bytes: 30000000 burst_size_bytes: 40000000 max_traces_per_user: 50000 global: max_bytes_per_trace: 10000000 ``` If you run a multi-tenant deployment, you can set different limits per tenant using runtime overrides instead of raising the global defaults. Refer to [Enable multi-tenancy](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/multitenancy/) for per-tenant override examples. For the full list of available settings, refer to [Ingestion limits](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#ingestion-limits) in the configuration reference. You can also manage per-tenant limits through the API using [user-configurable overrides](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/user-configurable-overrides/). ## Find and fix discarded spans When a span exceeds an ingestion limit or fails validation, Tempo discards it and increments the `tempo_discarded_spans_total` metric. The distributor rejects entire push requests that exceed the rate limit or contain invalid trace or span IDs, before any spans reach Kafka. Live-stores discard spans that exceed per-trace size or live trace count limits after consuming them from Kafka. Block-builders discard spans that exceed per-trace size limits. ### Error reference The following table lists the three error types, what each one means, and how to fix it. | Error | Cause | Fix | | ---------------------- | -------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `RATE_LIMITED` | The tenant's byte rate exceeded `rate_limit_bytes`. | Raise `rate_limit_bytes`, or add distributors if using `rate_strategy: local`. If volume is genuinely higher than intended, reduce it upstream with [sampling](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/tail-sampling/). | | `LIVE_TRACES_EXCEEDED` | The number of concurrent active traces on a live-store exceeded `max_traces_per_user`. | Raise `max_traces_per_user`, or add live-store instances to distribute the active trace count across more nodes. | | `TRACE_TOO_LARGE` | A single trace exceeded `max_bytes_per_trace` (default 5 MB). | Raise `max_bytes_per_trace` in the `global` overrides. Also investigate why the trace is so large. Common causes include retry loops and misconfigured instrumentation. | ### Check why spans are being discarded Query the `tempo_discarded_spans_total` metric. The `reason` label identifies why Tempo discarded each span: ```promql sum by (reason) (rate(tempo_discarded_spans_total[5m])) ``` The following table lists the possible `reason` values: | Reason | Meaning | Component | | ----------------------------- | ----------------------------------------------------------- | ---------------------- | | `rate_limited` | Tenant byte rate exceeded `rate_limit_bytes`. | Distributor | | `trace_too_large` | Single trace exceeded `max_bytes_per_trace`. | Live-store, block-builder | | `live_traces_exceeded` | Active trace count exceeded `max_traces_per_user`. | Live-store | | `invalid_trace_id` | Batch contained a trace ID that isn't 128 bits. | Distributor | | `invalid_span_id` | Batch contained a span ID that isn't 64 bits or was all zeros. | Distributor | | `too_far_in_future` | Span start or end time was more than `creation_grace_period` in the future. | Distributor | | `invalid_duration` | Span end time was before its start time. | Distributor | | `trace_too_large_to_compact` | Trace too large for the backend-worker to compact. | Backend-worker | | `unknown_error` | Unexpected error during span processing. | Live-store | ### Log discarded spans for debugging To log spans discarded by the distributor (rate-limited spans) with their trace IDs, enable `log_discarded_spans` in the distributor configuration: ```yaml distributor: log_discarded_spans: enabled: true ``` Set `include_all_attributes: true` for more verbose output that includes span attributes. Spans discarded by live-stores for `LIVE_TRACES_EXCEEDED` or `TRACE_TOO_LARGE` are logged at debug level by the live-store. To see these entries, set the live-store log level to `debug`. Refer to [Distributor refusing spans](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/send-traces/max-trace-limit-reached/) for additional troubleshooting steps. ### Traces missing from queries without errors If the distributor is not refusing spans but traces are missing from query results, the issue may be downstream. Live-stores consume trace data from Kafka and serve recent queries. If a live-store falls behind its Kafka partition, query results may be incomplete. Monitor the `tempo_live_store_lagged_requests_total` metric to detect when this happens. This counter increments every time a search or metrics query hits a live-store whose Kafka lag overlaps the requested time range, meaning results may be incomplete. The metric is labeled by `route` (`/tempopb.Querier/SearchRecent` or `/tempopb.Metrics/QueryRange`). The `fail_on_high_lag` setting (default `true`) controls how the live-store responds when lag is detected: - When `true`, the live-store returns an error when it can't guarantee complete results, rather than returning a partial response. This trades availability for correctness. - When `false`, the live-store returns whatever data it has, which may be incomplete. The metric still increments. Refer to [Unable to find traces](https://grafana.com/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/unable-to-see-trace/) for query-side troubleshooting. ## Identify what is driving ingestion volume When your cluster is healthy but ingestion is growing, the first step is finding which services are responsible for the most volume. ### Set up cost attribution The usage tracker breaks down ingested bytes by configurable attributes, giving you a per-service view of who is consuming capacity. Enable cost attribution in the distributor and configure which attributes to track in overrides: ```yaml distributor: usage: cost_attribution: enabled: true overrides: defaults: cost_attribution: dimensions: resource.service.name: "service" ``` ### Find the top contributors After enabling cost attribution, the distributor exposes the `tempo_usage_tracker_bytes_received_total` metric on the `/usage_metrics` endpoint, labeled by the dimensions you configured. You can query this endpoint directly: ```bash curl http://<distributor-host>:3200/usage_metrics ``` If you scrape this endpoint with Prometheus, you can use the following query to find which services are sending the most data: ```promql topk(10, sum by (service) ( rate(tempo_usage_tracker_bytes_received_total[1h]) ) ) ``` For the full set of configuration options, including scoping dimensions by resource or span and customizing label names, refer to [Usage tracker](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/usage-tracker/). ### Reduce volume from noisy services After you know which services are driving volume, the most effective way to reduce it is [sampling](https://grafana.com/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/instrument-send/set-up-collector/tail-sampling/) at the collector layer. Tail sampling lets you keep traces with errors or high latency while dropping routine ones, reducing volume without losing visibility into the problems that matter. --- ### Sources Tempo Operations Schema (`docs/sources/tempo/operations/schema.md`) --- title: Apache Parquet schema menuTitle: Parquet schema description: This document describes the schema used with the Parquet block format. weight: 900 aliases: - /docs/tempo/parquet/schema --- # Apache Parquet schema Apache Parquet is the only supported block format in Tempo 3.x. Refer to the [Parquet configuration options](../../configuration/parquet/) for more information. This document describes the schema used with the Parquet block format. ## Version applicability Tempo defaults to the vParquet5 schema. vParquet4 remains available and differs in some schema details. The column tables and collapsed schema example below document the vParquet5 layout. vParquet5 shares the same overall structure with changes listed in [Summary of vParquet5 differences](#summary-of-vparquet5-differences). The following sections apply to both vParquet5 and vParquet4: - Fully nested versus span-oriented schema - Static vs dynamic columns (see vParquet5 differences for changes to dedicated columns) - Compression and encoding - Bloom filters ## Fully nested versus span-oriented schema There are two overall approaches to a columnar schema: fully nested or span-oriented. Span-oriented means a flattened schema where traces are destructured into rows of spans. A fully nested schema means the current trace structures such as Resource/Scope/Spans/Events are preserved (nested data is natively supported in Parquet). In both cases, individual leaf values such as span name and duration are individual columns. We chose the nested schema for several reasons: - The block size is much smaller for the nested schema. This is due to the high data duplication incurred when flattening resource-level attributes such as `service.name` to each individual span. - A flat schema is not truly "flat" because each span still contains nested data such as attributes and events. - Nested schema is much faster to search for resource-level attributes because the resource-level columns are very small (1 row for each batch). - Translation to and from the OpenTelemetry Protocol Specification (OTLP) is straightforward. - Easily add computed columns (for example, trace duration) at multiple levels such as per-trace, per-batch, etc. ## Static vs dynamic columns Dynamic vs static columns add another layer to the schema. A dynamic schema stores each attribute such as `service.name` and `http.status_code` as its own column and the columns in each parquet file can be different. A static schema is unresponsive to the shape of the data, and all attributes are stored in generic key/value containers. The dynamic schema is the ultimate dream for a columnar format but it is too complex for a first release. However, the benefits of that approach are also too good to pass up, so we propose a hybrid approach. It is primarily a static schema but with some dynamic columns extracted from trace data based on some heuristics of frequently queried attributes. We plan to continue investing in this direction to implement a fully dynamic schema where trace attributes are blown out into independent Parquet columns at runtime. For more information, refer to the [Parquet design document](https://github.com/grafana/tempo/blob/main/docs/design-proposals/2022-04%20Parquet.md). ## Schema details The adopted Parquet schema is mostly a direct translation of OTLP but with some key differences. The table below uses these abbreviations: - `rs` - resource spans - `ss` - scope spans | | | | | :---------------------------------------------------- | :--------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Name | Type | Description | | TraceID | byte array | The trace ID in 16-byte binary form. | | TraceIDText | string | The trace ID in hexadecimal text form. | | StartTimeUnixNano | int64 | Start time of the first span in the trace, in nanoseconds since unix epoch. | | EndTimeUnixNano | int64 | End time of the last span in the trace, in nanoseconds since unix epoch. | | DurationNano | int64 | Total trace duration in nanoseconds, computed as difference between EndTimeUnixNano and StartTimeUnixNano. | | RootServiceName | string | The resource-level `service.name` attribute (rs.Resource.ServiceName) from the root span of the trace if one exists, else empty string. | | RootSpanName | string | The name (rs.ss.Spans.Name) of the root span if one exists, else empty string. | | ServiceStats | map | Per-service counts keyed by service name. Values include span count and error count. | | rs | | Short-hand for ResourceSpans | | rs.Resource.ServiceName | string | A dedicated column for the resource-level `service.name` attribute if present. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/#service | | rs.Resource.Cluster | string | A dedicated column for the resource-level `cluster` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. | | rs.Resource.Namespace | string | A dedicated column for the resource-level `namespace` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. | | rs.Resource.Pod | string | A dedicated column for the resource-level `pod` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. | | rs.Resource.Container | string | A dedicated column for the resource-level `container` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. | | rs.Resource.K8sClusterName | string | A dedicated column for the resource-level `k8s.cluster.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#cluster | | rs.Resource.K8sNamespaceName | string | A dedicated column for the resource-level `k8s.namespace.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#namespace | | rs.Resource.K8sPodName | string | A dedicated column for the resource-level `k8s.pod.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#pod | | rs.Resource.K8sContainerName | string | A dedicated column for the resource-level `k8s.container.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#container | | rs.Resource.DroppedAttributesCount | int | Number of resource attributes that were dropped. | | rs.Resource.Attrs.Key | string | All resource attributes that do not have a dedicated column are stored as a key value pair in these columns. The Key column stores the name, and then one of the Value columns is populated according to the attribute's data type. The other value columns will contain null. | | rs.Resource.Attrs.IsArray | bool | Indicates if the attribute is stored as an array. | | rs.Resource.Attrs.Value | string | The attribute value if string type (or array of strings), else null. | | rs.Resource.Attrs.ValueInt | int | The attribute value if integer type (or array of integers), else null. | | rs.Resource.Attrs.ValueDouble | float | The attribute value if float type (or array of floats), else null. | | rs.Resource.Attrs.ValueBool | bool | The attribute value if boolean type (or array of booleans), else null. | | rs.Resource.Attrs.ValueUnsupported | string | JSON-encoded AnyValue for unsupported or mixed-type values. | | rs.Resource.DedicatedAttributes | | Group containing spares for dedicated attribute columns with resource scope. | | rs.Resource.DedicatedAttributes.String01 ... String10 | string | 10 spare string columns for dedicated attributes (vParquet4). vParquet5 expands to String01 ... String20 and adds Int01 ... Int05. | | rs.ss | | Shorthand for ResourceSpans.ScopeSpans | | rs.ss.Scope | | Shorthand for ResourceSpans.ScopeSpans.Scope | | rs.ss.Scope.Name | string | Scope name if present, else empty string. https://opentelemetry.io/docs/specs/otel/glossary/#instrumentation-scope | | rs.ss.Scope.Version | string | The Scope version if present, else empty string. https://opentelemetry.io/docs/specs/otel/glossary/#instrumentation-scope | | rs.ss.Scope.Attrs | | Scope attributes, using the same columns as rs.Resource.Attrs.\* | | rs.ss.Scope.DroppedAttributesCount | int | Number of scope attributes that were dropped. | | rs.ss.Spans.SpanID | byte array | Span unique ID. | | rs.ss.Spans.ParentSpanID | byte array | The unique ID of the span's parent. For root spans without a parent this is null. | | rs.ss.Spans.ParentID | int32 | Trace local numeric parent ID. | | rs.ss.Spans.NestedSetLeft | int32 | Left bound of the nested set model. Also used as a trace local numeric span ID. | | rs.ss.Spans.NestedSetRight | int32 | Right bound of the nested set model. | | rs.ss.Spans.Name | string | Span name. | | rs.ss.Spans.StartTimeUnixNano | int64 | Start time the span in nanoseconds since unix epoch. | | rs.ss.Spans.DurationNano | int64 | Span duration in nanoseconds. | | rs.ss.Spans.Kind | int | The span's kind. Defined values: 0. Unset; 1. Internal; 2. Server; 3. Client; 4. Producer; 5. Consumer; https://opentelemetry.io/docs/reference/specification/trace/api/#spankind | | rs.ss.Spans.StatusCode | int | The span status. Defined values: 0: Unset; 1: OK; 2: Error. https://opentelemetry.io/docs/reference/specification/trace/api/#set-status | | rs.ss.Spans.StatusMessage | string | Optional message to accompany Error status. | | rs.ss.Spans.HttpMethod | string | A dedicated column for the span-level `http.method` attribute if present and of string type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#common-attributes | | rs.ss.Spans.HttpStatusCode | int | A dedicated column for the span-level `http.status_code` attribute if present and of integer type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#common-attributes | | rs.ss.Spans.HttpUrl | string | A dedicated column for the span-level `http.url` attribute if present and of string type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#http-client | | rs.ss.Spans.DroppedAttributesCount | int | Number of attributes that were dropped | | rs.ss.Spans.Attrs | | Span attributes, using the same columns as rs.Resource.Attrs.\* | | rs.ss.Spans.DedicatedAttributes | | Group containing spares for dedicated attribute columns with span scope | | rs.ss.Spans.DedicatedAttributes.String01 ... String10 | string | 10 spare string columns for dedicated attributes (vParquet4). vParquet5 expands to String01 ... String20 and adds Int01 ... Int05. | | rs.ss.Spans.DroppedEventsCount | int | The number of events that were dropped | | rs.ss.Spans.Events.TimeSinceStartNano | int64 | The event timestamp in nanoseconds, relative to the span start time. | | rs.ss.Spans.Events.Name | string | The event name or message. | | rs.ss.Spans.Events.DroppedAttributesCount | int | The number of event attributes that were dropped. | | rs.ss.Spans.Events.Attrs | | Event attributes, using the same columns as rs.Resource.Attrs.\* | | rs.ss.Spans.DroppedLinksCount | int | The number of links that were dropped. | | rs.ss.Spans.Links | | Repeated link records with TraceID, SpanID, TraceState, Attrs, and DroppedAttributesCount. | | rs.ss.Spans.TraceState | string | The span's TraceState value if present, else empty string. https://opentelemetry.io/docs/reference/specification/trace/api/#tracestate | To increase the readability the table omits the groups `list.element` that are added for nested list types in Parquet. For maps (for example, ServiceStats), Parquet also inserts map-specific group levels that are omitted here. {{< collapse title="vParquet4 block schema example" >}} ``` /* Detailed source-code truncated for AI context efficiency. */ ``` {{< /collapse >}} For the authoritative schema, refer to `tempodb/encoding/vparquet4/schema.go` and `tempodb/encoding/vparquet5/schema.go`. ### Summary of vParquet5 differences - Resource-level dedicated columns (Cluster/Namespace/Pod/Container/K8s\*) and span HTTP columns are removed; vParquet5 relies on dynamically assigned dedicated columns only. - Dedicated attribute columns are doubled from 10 to 20 string spares per scope and add 5 integer spares per scope. Event scope is also supported. Array attributes and optional blob configuration are available for selected columns. - Additional fields exist for optimization, including span `ChildCount`, rounded start time buckets, and trace-level `ServiceStats` as a list with explicit service names. ## Trace-level attributes For speed and ease-of-use, we are projecting several values to columns at the trace-level: - Trace ID - Don't store on each span. - Root service/span names/StartTimeUnixNano - These are selected properties of the root span in each trace (if there is one). These are used for displaying results in the Grafana UI. These properties are computed at ingest time and stored once for efficiency, so we don't have to find the root span. - `DurationNano` - The total trace duration, computed at ingest time. This powers the min/max duration filtering in the current Tempo search and is more efficient than scanning the spans duration column. However, it may go away with TraceQL or we could decide to change it to span-level duration filtering too. - `ServiceStats` - Per-service span and error counts for each trace. ## "Any"-type Attributes OTLP attributes have variable data types, which is simpler in formats like protocol-buffers, but doesn't translate directly to Parquet. Each column must have a concrete type. There are several possibilities here but we chose to have optional values for each concrete type. Unsupported or mixed-type values are stored as JSON-encoded AnyValue in `ValueUnsupported`. Only scalar values are stored in the dedicated attribute columns. Arrays and unsupported types remain in the generic attribute columns. ```yaml repeated group Attrs { required binary Key (STRING); required boolean IsArray; repeated binary Value (STRING); repeated int64 ValueInt (INTEGER(64,true)); repeated double ValueDouble; repeated boolean ValueBool; optional binary ValueUnsupported (STRING); } ``` ## Compression and encoding Parquet has robust support for many compression algorithms and data encodings. We've found excellent combinations of storage size and performance with the following: 1. Snappy Compression - Enable on all columns 1. Dictionary encoding - Enable on all string columns. Most strings are very repetitive so this works well to optimize storage size. However, you can greatly speed up search by inspecting the dictionary first and eliminating pages with no matches. 1. Time and duration UNIX nanos - Delta encoding 1. Rarely used columns such as `DroppedAttributesCount` - These columns are usually all zeroes, RLE works well. ### Bloom filters Parquet has native support for bloom filters. However, Tempo doesn't use them at this time. Tempo already has sophisticated support for sharding and caching bloom filters. --- ### Sources Tempo Operations Tempo Vulture (`docs/sources/tempo/operations/tempo-vulture.md`) --- title: Tempo Vulture description: Guide to using Tempo Vulture for data integrity testing keywords: ["tempo", "vulture", "tempo-vulture", "testing", "validation", "monitoring"] weight: 850 --- # Tempo Vulture Tempo Vulture is a testing tool that validates the end-to-end functionality of Tempo by continuously writing traces and verifying they can be read back correctly. It is useful for monitoring data integrity in production environments and for post-deployment validation testing. ## Overview Tempo Vulture performs the following operations: - Write traces: Pushes test traces to Tempo using OTLP over gRPC - Read traces by ID: Queries Tempo to retrieve traces by their trace ID - Search traces: Uses TraceQL to search for traces by attributes - Metrics queries: Validates TraceQL metrics queries (enabled by default) Tempo Vulture can run in two modes: - Continuous mode (default): Runs indefinitely, continuously writing and reading traces while exposing Prometheus metrics - Validation mode: Executes a fixed number of write/read cycles and exits with a status code indicating success or failure ## Run Tempo Vulture Tempo Vulture is available as a Docker image: ```bash docker run grafana/tempo-vulture:3.0.0 [arguments...] ``` You can also build it from source: ```bash make tempo-vulture ./tempo-vulture [arguments...] ``` ## Configuration flags Configure Tempo Vulture using command-line flags. ### Required flags | Flag | Description | Example | | ------------------- | ----------------------------------------------------------------- | ------------------- | | `--tempo-query-url` | The URL (scheme://hostname) at which to query Tempo | `http://tempo:3200` | | `--tempo-push-url` | The URL (scheme://hostname:port) at which to push traces to Tempo | `http://tempo:4317` | ### Optional flags | Flag | Default | Description | | ---------------------------------------- | ---------- | --------------------------------------------------------------------------------- | | `--tempo-org-id` | `""` | The org ID to use when querying Tempo (for multi-tenant deployments) | | `--tempo-push-tls` | `false` | Whether to use TLS when pushing spans to Tempo | | `--tempo-write-backoff-duration` | `15s` | Time to pause between write operations | | `--tempo-long-write-backoff-duration` | `1m` | Time to pause between long write operations | | `--tempo-read-backoff-duration` | `30s` | Time to pause between read operations | | `--tempo-search-backoff-duration` | `1m` | Time to pause between search operations. Set to `0s` to disable search validation | | `--tempo-metrics-backoff-duration` | `10s` | Time to pause between TraceQL metrics operations. Set to `0s` to disable | | `--tempo-retention-duration` | `336h` | The block retention that Tempo is using | | `--tempo-recent-traces-backoff-duration` | `14m` | Cutoff between recent and old traces query checks | | `--tempo-query-livestore` | `false` | Whether to query live stores | | `--prometheus-listen-address` | `:80` | Address to listen on for Prometheus scrapes | | `--prometheus-path` | `/metrics` | Path to publish Prometheus metrics | ### Validation mode flags | Flag | Default | Description | | ---------------------- | ------- | ----------------------------------------------------------------- | | `--validation-mode` | `false` | Run in validation mode: execute a fixed number of cycles and exit | | `--validation-cycles` | `3` | Number of write/read cycles to perform in validation mode | | `--validation-timeout` | `5m` | Maximum time to run validation mode before timing out | ## Continuous mode In continuous mode, Tempo Vulture runs indefinitely and exposes Prometheus metrics that can be used for monitoring and alerting. ### Example ```bash docker run grafana/tempo-vulture:3.0.0 \ --tempo-query-url=http://tempo:3200 \ --tempo-push-url=http://tempo:4317 \ --tempo-org-id=my-tenant \ --tempo-write-backoff-duration=15s \ --tempo-read-backoff-duration=30s \ --tempo-search-backoff-duration=60s ``` ### Prometheus metrics Tempo Vulture exposes the following key metrics: | Metric | Type | Description | | --------------------------------- | ------- | ------------------------------------------ | | `tempo_vulture_trace_total` | Counter | Total number of trace operations performed | | `tempo_vulture_trace_error_total` | Counter | Total number of trace errors by error type | Error types include: - `incorrectresult`: Retrieved trace doesn't match the expected trace - `incorrect_metrics_result`: Metrics query returned unexpected results - `missingspans`: Trace has missing spans - `notfound_byid`: Trace not found by ID - `notfound_search`: Trace not found via search - `notfound_traceql`: Trace not found via TraceQL - `notfound_metrics`: Trace not found via metrics query - `notfound_search_attribute`: No searchable attribute found in trace - `inaccurate_metrics`: Metrics count doesn't match actual span count - `requestfailed`: Request to Tempo failed ### Alerting You can configure alerts based on Tempo Vulture metrics. For example, the Tempo mixin includes a [`TempoVultureHighErrorRate` alert](https://github.com/grafana/tempo/blob/08325a7d5da6a330ac5564265eee52e13544abc9/operations/tempo-mixin/alerts.libsonnet#L426) that fires when the error rate exceeds a configurable threshold. ## Validation mode {{< admonition type="note" >}} Validation mode was added in Tempo 2.10. {{< /admonition >}} Validation mode runs Tempo Vulture as a one-shot test that can be integrated into CI/CD pipelines for post-deployment validation. Instead of running continuously, it executes a fixed number of write/read cycles and exits with a status code indicating success or failure. {{< admonition type="warning" >}} Running Vulture as a Deployment while in validation mode causes problems in Kubernetes. Because Deployments expect long-running processes, a container that finishes its task and exits triggers a `CrashLoopBackOff` (even with exit code `0`). Run validation mode as a Job instead. {{< /admonition >}} ### Exit codes | Exit code | Meaning | | --------- | ------------------------------------------------------ | | `0` | All validations passed | | `1` | One or more validations failed, or configuration error | ### Environment variables Validation mode requires authentication: | Variable | Description | | --------------------------- | ------------------------------------------------------------------------------- | | `TEMPO_ACCESS_POLICY_TOKEN` | Access policy token for authenticating with Tempo (required in validation mode) | ### Example ```bash docker run \ -e TEMPO_ACCESS_POLICY_TOKEN=your-token \ grafana/tempo-vulture:3.0.0 \ --tempo-query-url=https://tempo.example.com \ --tempo-push-url=https://tempo.example.com:4317 \ --tempo-org-id=my-tenant \ --validation-mode \ --validation-cycles=5 \ --validation-timeout=10m ``` ### CI/CD pipeline integration Validation mode is useful for: - **Post-deployment testing**: Verify Tempo is functioning correctly after a deployment - **Smoke testing**: Quick validation that write and read paths are working - **Integration testing**: Ensure end-to-end trace pipeline is operational Example CI/CD usage in a GitHub Actions workflow: ```yaml - name: Validate Tempo deployment run: | docker run \ -e TEMPO_ACCESS_POLICY_TOKEN=${{ secrets.TEMPO_TOKEN }} \ grafana/tempo-vulture:3.0.0 \ --tempo-query-url=${{ vars.TEMPO_URL }} \ --tempo-push-url=${{ vars.TEMPO_PUSH_URL }} \ --tempo-org-id=${{ vars.TEMPO_ORG_ID }} \ --validation-mode \ --validation-cycles=3 \ --validation-timeout=5m ``` ### Validation process In validation mode, Tempo Vulture performs these steps: 1. For each cycle (default: 3): - Write a test trace to Tempo - Wait briefly for ingestion - Read the trace back by ID and verify it matches 2. After all cycles complete: - Wait for traces to be searchable (if search is enabled) - Search for each trace using a random attribute - Verify all traces are found 3. Exit with code `0` if all validations pass, or `1` if any fail ## Docker Compose example The Tempo repository includes Docker Compose examples that demonstrate how to run Tempo Vulture. Refer to the [example configurations](https://github.com/grafana/tempo/tree/main/example/docker-compose) for complete setups. Example from a Docker Compose file: ```yaml vulture: image: grafana/tempo-vulture:3.0.0 command: - "-tempo-push-url=http://distributor:4317" - "-tempo-query-url=http://query-frontend:3200" - "-tempo-org-id=" ``` --- ### Sources Tempo Operations Tempo Cli (`docs/sources/tempo/operations/tempo_cli.md`) --- title: Tempo CLI description: Guide to using tempo-cli keywords: ["tempo", "cli", "tempo-cli", "command line interface"] weight: 800 --- # Tempo CLI Tempo CLI is a separate executable that contains utility functions related to the Tempo software. Although it's not required for a working installation, Tempo CLI can be helpful for deeper analysis or for troubleshooting. ## Tempo CLI command syntax The general syntax for commands in Tempo CLI is: ```bash tempo-cli command [subcommand] [options] [arguments...] ``` `--help` or `-h` displays the help for a command or subcommand. Example: ```bash tempo-cli -h tempo-cli command [subcommand] -h ``` ## Run Tempo CLI Tempo CLI is available as source code and as a Docker image. ### Run with Docker (published image) ```bash docker run --rm grafana/tempo-cli [arguments...] ``` ### Run directly with `go run` ```bash go run ./cmd/tempo-cli [arguments...] ``` ### Build and run a local binary To build a local binary, you need a working Go installation and build environment. ```bash make tempo-cli ./bin/$(go env GOOS)/tempo-cli-$(go env GOARCH) [arguments...] ``` ### Build and run a local Docker image ```bash make docker-tempo-cli docker run --rm tempo-cli [arguments...] ``` ## Backend options Tempo CLI connects directly to the storage backend for some commands, meaning that it requires the ability to read from S3, GCS, Azure or file-system storage. You can configure the backend using the following options: - Load an existing tempo configuration file using the `--config-file` (`-c`) option. This is the recommended option for frequent usage. Refer to [Configuration](../../configuration/) documentation for more information. - Specify individual settings: - `--backend <value>` The storage backend type, one of `s3`, `gcs`, `azure`, and `local`. - `--bucket <value>` The bucket name. The meaning of this value is backend-specific. Refer to [Configuration](../../configuration/) documentation for more information. - `--s3-endpoint <value>` The S3 API endpoint (i.e. s3.dualstack.us-east-2.amazonaws.com). - `--s3-user <value>`, `--s3-pass <value>` The S3 user name and password (or access key and secret key). Optional, as Tempo CLI supports the same authentication mechanisms as Tempo. Refer to [S3 permissions documentation](../../configuration/hosted-storage/s3/) for more information. - `--insecure-skip-verify` skip TLS verification, only applies to S3 and GCS. Each option applies only to the command in which it's used. For example, `--backend <value>` doesn't permanently change where Tempo stores data. It only changes it for command in which you apply the option. ## TLS options All `query api` commands support these options for HTTPS and gRPC TLS connections. Use an `https://` URL for `trace-id`, or `--secure` for the other commands. - `--tls-cert <path>`, `--tls-key <path>` PEM client certificate and matching unencrypted private key for mutual TLS (mTLS). Both must be provided together. - `--tls-ca <path>` PEM CA bundle used instead of the system trust roots for server certificate verification. - `--tls-server-name <name>` Override the server name used for certificate verification and SNI. The CA and server-name options can also be used without a client certificate. ## Query API command ### Trace ID Call the Tempo API and retrieve a trace by ID. ```bash tempo-cli query api trace-id <api-endpoint> <trace-id> ``` Arguments: - `api-endpoint` URL for the Tempo API. - `trace-id` Trace ID as a hexadecimal string. Options: - [TLS options](#tls-options) - `--org-id <value>` Organization ID (for use in multi-tenant setup). - `--header <key=value>` Extra HTTP header to send with the request. Can be specified multiple times. - `--v1` use v1 API (use /api/traces endpoint to fetch traces, default: /api/v2/traces). Example: ```bash tempo-cli query api trace-id http://tempo:3200 f1cfe82a8eef933b ``` Example with a custom authentication header: ```bash tempo-cli query api trace-id http://tempo:3200 f1cfe82a8eef933b --header "X-TOKEN=<API_TOKEN>" ``` Replace _`<API_TOKEN>`_ with your authentication token. ### Search Call the Tempo API and search using TraceQL. ```bash tempo-cli query api search <host-port> <trace-ql> [<start> <end>] ``` Arguments: - `host-port` A host/port combination for Tempo. The scheme is inferred from the options. - `trace-ql` TraceQL query. - `start` Start of the time range to search in RFC3339 format (e.g. `2024-01-01T00:00:00Z`) or relative (e.g. `now-1h`) - `end` End of the time range to search in RFC3339 format (e.g. `2024-01-01T01:00:00Z`) or relative (e.g. `now`) Options: - [TLS options](#tls-options) - `--org-id <value>` Organization ID (for use in multi-tenant setup). - `--header <key=value>` Extra header to send with the request (as gRPC metadata when using `--use-grpc`). Can be specified multiple times. - `--use-grpc` Use GRPC streaming - `--spss <value>` Number of spans to return for each spanset - `--limit <value>` Number of results to return - `--path-prefix <value>` String to prefix search paths with - `--secure` Use HTTPS or gRPC with TLS {{< admonition type="note" >}} Set the `stream_over_http_enabled` flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to [Tempo GRPC API documentation](../../api_docs/). {{< /admonition >}} Example searching for error spans using relative time: ```bash tempo-cli query api search localhost:3200 '{status = error}' now-1h now ``` Example searching for error spans using absolute time: ```bash tempo-cli query api search localhost:3200 '{status = error}' 2024-01-01T00:00:00Z 2024-01-01T01:00:00Z ``` Example using GRPC streaming with organization ID: ```bash tempo-cli query api search --use-grpc --org-id my-org localhost:3200 '{span.http.status_code >= 400}' now-1h now ``` Example with a custom authentication header over GRPC: ```bash tempo-cli query api search --use-grpc --header "X-TOKEN=<API_TOKEN>" localhost:9095 '{status = error}' now-1h now ``` ### Search tags Call the Tempo API and search attribute names. ```bash tempo-cli query api search-tags <host-port> [<start> <end>] ``` Arguments: - `host-port` A host/port combination for Tempo. The scheme will be inferred based on the options provided. - `start` Start of the time range to search in RFC3339 format (e.g. `2024-01-01T00:00:00Z`) or relative (e.g. `now-1h`) - `end` End of the time range to search in RFC3339 format (e.g. `2024-01-01T01:00:00Z`) or relative (e.g. `now`) Options: - [TLS options](#tls-options) - `--org-id <value>` Organization ID (for use in multi-tenant setup). - `--header <key=value>` Extra header to send with the request (as gRPC metadata when using `--use-grpc`). Can be specified multiple times. - `--use-grpc` Use GRPC streaming - `--path-prefix <value>` String to prefix search paths with - `--secure` Use HTTPS or gRPC with TLS {{< admonition type="note" >}} Set the `stream_over_http_enabled` flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to [Tempo GRPC API documentation](../../api_docs/). {{< /admonition >}} Example: ```bash tempo-cli query api search-tags localhost:3200 ``` Example with relative time range: ```bash tempo-cli query api search-tags localhost:3200 now-1h now ``` Example with absolute time range: ```bash tempo-cli query api search-tags localhost:3200 2024-01-01T00:00:00Z 2024-01-02T00:00:00Z ``` ### Search tag values Call the Tempo API and search attribute values. ```bash tempo-cli query api search-tag-values <host-port> <tag> [<start> <end>] ``` Arguments: - `host-port` A host/port combination for Tempo. The scheme is inferred from the options. - `tag` The fully qualified TraceQL tag to search for. For example, `resource.service.name`. - `start` Start of the time range to search in RFC3339 format (e.g. `2024-01-01T00:00:00Z`) or relative (e.g. `now-1h`) - `end` End of the time range to search in RFC3339 format (e.g. `2024-01-01T01:00:00Z`) or relative (e.g. `now`) Options: - [TLS options](#tls-options) - `--org-id <value>` Organization ID (for use in multi-tenant setup). - `--header <key=value>` Extra header to send with the request (as gRPC metadata when using `--use-grpc`). Can be specified multiple times. - `--query <value>` TraceQL query to filter attribute results by. - `--use-grpc` Use GRPC streaming - `--path-prefix <value>` String to prefix search paths with - `--secure` Use HTTP or gRPC with TLS {{< admonition type="note" >}} Set the `stream_over_http_enabled` flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to [Tempo GRPC API documentation](../../api_docs/). {{< /admonition >}} Example to find all service names: ```bash tempo-cli query api search-tag-values localhost:3200 resource.service.name ``` Example with query filter to find service names that have errors: ```bash tempo-cli query api search-tag-values --query '{status = error}' localhost:3200 resource.service.name ``` ### Metrics Call the Tempo API and generate metrics from traces using TraceQL. ```bash tempo-cli query api metrics <host-port> <trace-ql metrics query> [<start> <end>] ``` Arguments: - `host-port` A host/port combination for Tempo. The scheme will be inferred based on the options provided. - `trace-ql metrics query` TraceQL metrics query. - `start` Start of the time range to search in RFC3339 format (e.g. `2024-01-01T00:00:00Z`) or relative (e.g. `now-1h`) - `end` End of the time range to search in RFC3339 format (e.g. `2024-01-01T01:00:00Z`) or relative (e.g. `now`) Options: - [TLS options](#tls-options) - `--org-id <value>` Organization ID (for use in multi-tenant setup). - `--header <key=value>` Extra header to send with the request (as gRPC metadata when using `--use-grpc`). Can be specified multiple times. - `--use-grpc` Use GRPC streaming - `--instant` Perform an instant query instead of a range query. - `--path-prefix <value>` String to prefix search paths with - `--secure` Use HTTPS or gRPC with TLS {{< admonition type="note" >}} Set the `stream_over_http_enabled` flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to [Tempo GRPC API documentation](../../api_docs/). {{< /admonition >}} Example range query for request rates by service: ```bash tempo-cli query api metrics localhost:3200 '{} | rate() by (resource.service.name)' now-1h now ``` Example instant query for current error rates: ```bash tempo-cli query api metrics --instant localhost:3200 '{status = error} | rate()' now-1h now ``` ## Query trace-id command Iterate over all backend blocks and dump all data found for a given trace ID. ```bash tempo-cli query trace-id <trace-id> <tenant-id> ``` {{< admonition type="note" >}} This can be intense as it downloads every bloom filter and some percentage of indexes/trace data. {{< /admonition >}} Arguments: - `trace-id` Trace ID as a hexadecimal string. - `tenant-id` Tenant to search. Options: - [Backend options](#backend-options) - `--percentage <value>` Percentage of blocks to scan (for example, 0.1 for 10%). Useful for sampling large datasets. Example: ```bash tempo-cli query trace-id f1cfe82a8eef933b single-tenant ``` Example scanning only 10% of blocks: ```bash tempo-cli query trace-id --percentage 0.1 f1cfe82a8eef933b single-tenant ``` ## Query trace summary command Iterate over all backend blocks and dump a summary for a given trace ID. The summary includes: - number of blocks the trace is found in - span count - trace size - trace duration - root service name - root span info - top frequent service names ```bash tempo-cli query trace-summary <trace-id> <tenant-id> ``` {{< admonition type="note" >}} This can be intense as it downloads every bloom filter and some percentage of indexes/trace data. {{< /admonition >}} Arguments: - `trace-id` Trace ID as a hexadecimal string. - `tenant-id` Tenant to search. Options: - [Backend options](#backend-options) - `--percentage <value>` Percentage of blocks to scan (for example, 0.1 for 10%). Useful for sampling large datasets. Example: ```bash tempo-cli query trace-summary f1cfe82a8eef933b single-tenant ``` ## List blocks Lists information about all blocks for the given tenant, and optionally perform integrity checks on indexes for duplicate records. ```bash tempo-cli list blocks <tenant-id> ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. Options: - `--include-compacted` Include blocks that have been compacted. Default behavior is to display only active blocks. **Output:** Explanation of output: - `ID` Block ID. - `Lvl` Compaction level of the block. - `Objects` Number of objects stored in the block. - `Size` Data size of the block after any compression. - `Vers` Block version. - `Window` The window of time that was considered for compaction purposes. - `Start` The earliest timestamp stored in the block. - `End` The latest timestamp stored in the block. - `Duration` Duration between the start and end time. - `Age` The age of the block. - `Cmp` Whether the block has been compacted (present when --include-compacted is specified). Example: ```bash tempo-cli list blocks -c ./tempo.yaml single-tenant ``` ## List compaction summary Summarizes information about all blocks for the given tenant based on compaction level. This command is useful to analyze or troubleshoot compaction behavior. ```bash tempo-cli list compaction-summary <tenant-id> ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. Example: ```bash tempo-cli list compaction-summary -c ./tempo.yaml single-tenant ``` ## List cache summary Prints information about the number of bloom filter shards per day per compaction level. This command is useful to estimate and fine-tune cache storage. Read the [caching topic](../caching/) for more information. ```bash tempo-cli list cache-summary <tenant-id> ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. Example: ```bash tempo-cli list cache-summary -c ./tempo.yaml single-tenant ``` ## List column Lists values in a given column of a block. Useful for inspecting parquet block data directly. ```bash tempo-cli list column <tenant-id> <block-id> [column-name] ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. - `block-id` The block ID as UUID string. - `column-name` Column name to list values of (default: `TraceID`). Options: - [Backend options](#backend-options) Example: ```bash tempo-cli list column -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1 TraceID ``` ## View schema View block metadata, parquet schema structure, and column sizes for a given block. ```bash tempo-cli view schema <tenant-id> <block-id> ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. - `block-id` The block ID as UUID string. Options: - [Backend options](#backend-options) The output includes: - Block metadata - Parquet schema structure - Column sizes in KB Example: ```bash tempo-cli view schema -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1 ``` ## Benchmark profile Profile a local block for read-path benchmarking. Writes a JSON file recording what had to be measured from the block — its metadata, its row-group count, present and absent trace IDs to look up, and its attributes ranked by size — so that a benchmark run does not have to inspect the block, and every variant of an experiment works from the same measurements. ```bash tempo-cli benchmark profile <block-path> ``` Arguments: - `block-path` Path to the block directory on local disk, laid out as `<bucket>/<tenant-id>/<block-id>`. Options: - `--trace-ids` Number of present trace IDs to sample, or `all` to enumerate every ID at run time rather than embedding them. Defaults to `10000`. One absent ID is derived per present ID. Pass `0` to skip trace IDs, and with them the full scan they require. - `--attributes` Number of attributes to profile in each scope (resource, span) and kind of value (string, numeric, boolean), ranked by total bytes as `analyse block` counts them: a resource attribute once per resource, a span attribute once per span. The `name`, `status`, `kind` and `duration` intrinsics are always included. Each records its cardinality, the share of spans carrying it, and the share carrying each of its 100 most frequent values, or, for a numeric attribute with more than 100 distinct values, quantiles of its values. Defaults to `100`. Pass `0` to skip the full scan it requires. Supported for vParquet5 blocks. - `-o`, `--out` File to write the profile to. Defaults to stdout. Example: ```bash tempo-cli benchmark profile /data/traces/single-tenant/ca314fba-efec-4852-ba3f-8d2b0bbf69f1 --trace-ids=10000 -o profile.json ``` ## Benchmark run Run read-path benchmark queries against a local block and write the measurements as JSON. Takes a profile from `benchmark profile`, so the run does not inspect the block and every variant of an experiment measures the same queries. ```bash tempo-cli benchmark run <block-path> -p <profile.json> ``` Arguments: - `block-path` Path to the block directory on local disk, laid out as `<bucket>/<tenant-id>/<block-id>`. Options: - `-p`, `--profile` Profile of the block, from `benchmark profile`. Required. - `-o`, `--out` File to write the result to. Defaults to stdout. - `-k`, `--case` Cases to run, as glob patterns over case IDs in the syntax `benchmark compare` takes, such as `'metrics/ranked_*/*/high/*'`. Repeat the flag or separate patterns with commas. Defaults to every case. Patterns matching no case are an error. - `--repeat` Passes over the query set. Defaults to `1`. - `--target-bytes-per-request` Bytes per search shard, mirroring the query frontend option of the same name. Defaults to `100MiB`. - `--search-limit` Traces a search returns per shard. Defaults to `20`. - `--max-series` Series a metrics query returns. Defaults to `1000`. - `--exemplars` Exemplars a metrics query collects. Defaults to `0`. - `--max-tag-values`, `--stale-value-threshold` A tag-value request's `limit` and `maxStaleValues`. Each defaults to `0`, meaning no limit, as for a request without them. - `--read-buffer-size` Storage read buffer size, such as `8MiB`. Defaults to `0`, meaning Tempo's default. This is the knob an experiment varies. - `--backend-latency`, `--backend-bandwidth` Simulate an object store: every backend request waits the latency plus its size over the bandwidth, a size per second such as `100MiB`. Each defaults to `0`, which reads the local block as is. Without them, a local read costs almost nothing, so an option that trades request count for bytes read, like `--read-buffer-size`, only shows up in `backend.*` and not in latency. Each case records how many results it matched, and one `metrics` map. Two runs are only comparable if the match counts agree, so a difference there means the comparison is invalid rather than interesting. Every measurement has the same shape whatever its source, so nothing reading a result needs a rule per source: - `total` is the sum over the case for a `counter`, or the value left behind for a `gauge`. - `summary` describes the per-execution values, with quantiles so a box plot needs nothing else. A total on its own hides the tail, which on the read path is usually the interesting part. Keys are `source.name`: - `harness.*` is what the benchmark timed itself: `wallNs`, `cpuNs`, `allocBytes`, `allocCount`. - `backend.*` is object-store traffic: `reads`, `bytes`, `timeNs`. - `response.*` is what a query API reported, under Tempo's own metric names. A metric Tempo did not report is absent rather than zero, because the two are different claims, so a summary's `count` says how many executions reported it. - `process.*` is what Tempo emitted to its Prometheus registry, including the Go runtime and process collectors. Metrics are collected rather than listed, so a metric added to Tempo appears without a change to the benchmark. Only metrics that moved are kept. The query set has fixed cases, plus cases generated from the profile's ranked attributes. Fixed cases: - `traceid/present`, `traceid/absent`: trace lookups by ID, for IDs the block holds and IDs it does not. - `search/nopredicate`: an unfiltered search. - `metrics/rate`, `metrics/rate-by-service`: `rate()` and `rate() by (resource.service.name)`. - `metrics/drilldown/*`: the queries Traces Drilldown runs when it opens, such as `{nestedSetParent<0 && true} | rate() with(sample=true)`. - `metadata/tagnames/<scope>`: a tag-name lookup in each attribute scope. Generated cases, from the top 10 attributes with a dedicated column and the top 10 without: - `<group>` is the group an attribute was picked from: `ranked_dedicated` for an attribute with a column of its own, which every intrinsic and `resource.service.name` have, and `ranked_generic` for one in the generic key/value columns. - Each attribute is queried with up to three of its values, one per selectivity band: `high` is its most common value, `medium` the middle of its profiled values, and `low` the least common value the profile kept. Every value is one the block holds, so every query matches. - `search/<group>/<band>/<attribute>`: `{ attribute = value }`. - `metrics/<group>/<rate|quantile>/<band>/<attribute>`: `{ attribute = value } | rate()` and `{ attribute = value } | quantile_over_time(duration, 0.99, 0.95)`. - `metadata/<group>/tagvalues/<attribute>`: a tag-value lookup with no TraceQL, for the top 20 string attributes in each group. - Attributes with only quantile values are skipped, and a profile built with `--attributes 0` generates no cases. To run only some cases, pass `--case` a glob, such as `--case 'metrics/ranked_generic/*/low/*'`. Metrics queries run over the block's whole time range, stepping at `max(60s, window/30)` to land about 30 points. Searches and metrics queries are split into shards of row groups, mirroring how the query frontend splits a job. A case that fails is recorded with its error and the rest of the run continues. Benchmarking trace lookups reads the block's bloom filters, so a partial block copy without them can still be profiled but only its search cases will run. Example: ```bash tempo-cli benchmark profile /data/traces/single-tenant/ca314fba-efec-4852-ba3f-8d2b0bbf69f1 -o profile.json tempo-cli benchmark run /data/traces/single-tenant/ca314fba-efec-4852-ba3f-8d2b0bbf69f1 -p profile.json -o result.json ``` ## Benchmark compare Compare two or more results from `benchmark run`, and write the comparison as markdown to stdout, to read or paste into a pull request, or serve it as web pages to explore. For each metric it lays every case out as benchstat does: the baseline's value, then each other run's value and its change from the baseline. ```bash tempo-cli benchmark compare <result.json> <result.json>... ``` A change of 10% or more is in bold. A run that cannot be compared with the baseline on a case reads `not comparable`, with the reason under the table: its match or execution count per pass differs, or it is missing the case or failed it. When a run's name is too long to head a column, runs are numbered, and listed with their numbers. With `--http`, it serves the comparison as web pages instead, rendered on each request as pprof's web view is. The summary page has a table per metric, with links to switch the metric and the percentile. Each case has a page with the box plots and table of every metric at once, and hovering a box plot's row shows its numbers. A box spans the 25th to 75th percentile, with a mark at the median. Its whisker runs from the minimum to the 99th percentile, with a tick at the 90th. The axis stops near the highest 99th percentile, so a maximum far past it is marked at the edge and written out, rather than squashing every box to make room for it. Clicking a run on either page makes it the baseline. As with pprof, an address without a host, like `:8080`, is served on localhost only. Arguments: - `results` Results to compare, as `name=path` or a path. The first is the baseline. Options: - `-m`, `--metric` Metrics to show, as glob patterns over the metric keys. Defaults to `harness.wallNs`, `harness.cpuNs`, `harness.allocBytes`, `backend.bytes`, and `backend.reads`. - `-k`, `--case` Cases to show, as glob patterns over the case IDs, for example `traceid/*`. Defaults to every case. - `--percentile` Percentile the summaries show: `min`, `p25`, `p50`, `p75`, `p90`, `p99`, or `max`. Defaults to `p99`, since tail latency is what hurts most. - `--http` Serve the comparison as web pages on this address, like `:8080`, instead of writing markdown. A result given as a path is named after the settings that set it apart from the others: the run options and git SHA that differ between the runs, or, when those are all the same, the Go version, `GOMAXPROCS`, or host. When one setting differs, the name is its value, like `4MiB`. When several do, the name lists each, like `targetBytesPerRequest=2MiB readBufferSize=4MiB`. Runs set up alike, such as repeats of one setup, are named after their files instead. When the one setting that differs is a number, the runs are put in its order, with those left at Tempo's default first, so reading across the columns follows the setting as it grows. The baseline stays the run given first wherever it lands. Above the tables, each run has a line saying how it differs from the baseline, like `readBufferSize default → 4MiB`. The baseline's line shows what the others are measured from: its value of every setting that differs, then its git SHA, Go version, `GOMAXPROCS`, and host. A difference in where a run happened, its Go version, `GOMAXPROCS`, or host, is flagged with ⚠, since latencies from two environments are hard to compare. The summaries are per execution: one trace lookup, or one shard of a search, metrics, or tag-name query. The spread of a box is across those executions, not across repeated runs, so it describes how the inputs differ, not how noisy the measurement is. Example, where the runs are named `default`, `4MiB`, and `16MiB` from their read buffer sizes: ```bash tempo-cli benchmark compare main.json read-buffer-4mib.json read-buffer-16mib.json -k 'traceid/*' > comparison.md tempo-cli benchmark compare main.json read-buffer-4mib.json read-buffer-16mib.json --http=:8080 ``` ## Query search command Search blocks in a given time range for a specific key/value pair. ```bash tempo-cli query search <name> <value> <start> <end> <tenant-id> ``` {{< admonition type="note" >}} This can be intense as it downloads all relevant blocks and iterates through them. {{< /admonition >}} Arguments: - `name` Name of the attribute to search for, for example, `http.method`. - `value` Value of the attribute to search for, for example, `GET`. - `start` Start of the time range to search in RFC3339 format (e.g. `2024-01-01T00:00:00Z`) or relative (e.g. `now-1h`) - `end` End of the time range to search in RFC3339 format (e.g. `2024-01-01T01:00:00Z`) or relative (e.g. `now`) - `tenant-id` Tenant to search. Options: - [Backend options](#backend-options) Example using relative time: ```bash tempo-cli query search http.method GET now-1h now single-tenant --backend=gcs --bucket=tempo-trace-data ``` Example using absolute time: ```bash tempo-cli query search http.method GET 2024-01-01T00:00:00Z 2024-01-01T00:05:00Z single-tenant --backend=gcs --bucket=tempo-trace-data ``` ## Parquet convert A to B command Converts a vParquet file (actual data.parquet) of format A to a block of newer format B with an optional list of dedicated attribute columns. Actual supported versions for A and B vary by Tempo release. This utility command is useful when testing the impact of different combinations of dedicated columns. ### Convert vParquet4 to vParquet5 Converts a vParquet4 block to vParquet5 format with an optional list of dedicated attribute columns. ```bash tempo-cli parquet convert-4to5 <in file> [<out path>] [<list of dedicated columns>] ``` Arguments: - `in file` Path to an existing vParquet4 block directory. - `out path` Path to write the vParquet5 block to. The default is `./out`. - `list of dedicated columns` Optional list of columns to make dedicated. Columns use TraceQL syntax with scope. For example, `span.http.method`, `resource.namespace`, `event.exception.message`. Column prefixes: - `int/` marks the column as an integer type. For example, `int/span.http.status_code`. - `blob/` marks the column for blob encoding. For example, `blob/span.db.statement`. Example: ```bash tempo-cli parquet convert-4to5 ./block-in ./block-out "span.http.method" "int/span.http.status_code" "blob/span.db.statement" ``` ## Migrate tenant command Copy blocks from one backend and tenant to another. Blocks can be copied within the same backend or between two different backends. The data format isn't converted but the tenant ID in `meta.json` is rewritten. ```bash tempo-cli migrate tenant <source tenant> <dest tenant> ``` Arguments: - `source tenant` Tenant to copy blocks from - `dest tenant` Tenant to copy blocks into Options: - `--source-config-file <value>` (required) Configuration file for the source backend. - `--config-file <value>` Configuration file for the destination backend. Example: ```bash tempo-cli migrate tenant --source-config-file source.yaml --config-file dest.yaml my-tenant my-other-tenant ``` ## Migrate overrides config command Migrate the overrides section of a full Tempo config file from the legacy flat format to the new scoped format. The command reads the full config, converts any legacy overrides, and outputs only user-set values. ```bash tempo-cli migrate overrides-config <config-file> ``` Arguments: - `config-file` Path to the full Tempo config file. Options: - `-d, --config-dest <path>` Path to write the migrated overrides section. If not specified, output is printed to `stdout`. Example: ```bash tempo-cli migrate overrides-config config.yaml -d migrated-overrides.yaml ``` {{< admonition type="warning" >}} - Fields set to Go zero values (`false`, `0`, `""`) may be silently dropped due to `omitempty` tags. Compare against your original config to ensure nothing is lost. - Secret values (for example, `remote_write_headers`) are masked as `<secret>` in the output. You must manually restore the original values. - Some struct fields without `omitempty` may appear with zero values (for example, `exclude: null`) that were not in your original config. {{< /admonition >}} ## Migrate overrides per-tenant command Migrate a per-tenant overrides file from the legacy flat format to the new scoped format. The command handles both legacy and new format entries, and outputs only tenant-specific values. ```bash tempo-cli migrate overrides-per-tenant <overrides-file> ``` Arguments: - `overrides-file` Path to the per-tenant overrides file. Options: - `-d, --output-dest <path>` Path to write the migrated per-tenant overrides. If not specified, output is printed to `stdout`. Example: ```bash tempo-cli migrate overrides-per-tenant overrides.yaml -d migrated-overrides.yaml ``` {{< admonition type="warning" >}} - Fields set to Go zero values (`false`, `0`, `""`) may be silently dropped due to `omitempty` tags. Compare against your original config to ensure nothing is lost. - Secret values (for example, `remote_write_headers`) are masked as `<secret>` in the output. You must manually restore the original values. - Some struct fields without `omitempty` may appear with zero values (for example, `exclude: null`) that were not in your original config. {{< /admonition >}} ## Migrate config command Migrate a Tempo 2.x configuration file to a valid 3.0 configuration. The command removes obsolete configuration sections (such as `ingester`, `ingester_client`, and `compactor`), adds Kafka ingest configuration for microservices mode, disables compaction in overrides for parallel operation during migration, and strips the removed `local-blocks` metrics-generator processor. The tool works at the YAML map level rather than rewriting the file from fully decoded Tempo structs, so environment variable references like `${VAR}` are preserved. Top-level sections that are not recognized by the Tempo 3.0 configuration are dropped from the output. Unknown nested keys normally cause validation to fail, but validation is best-effort when the configuration contains `${VAR}` placeholders where non-string types are expected, which may let unknown nested keys through. ```bash tempo-cli migrate config [options] <config-file> ``` Arguments: - `config-file` Path to the 2.x Tempo configuration file. Options: - `--kafka-address <address>` Kafka broker address. Required when running in microservices mode. - `--kafka-topic <topic>` Kafka topic name. Defaults to `tempo`. - `--mode <monolithic|microservices>` Override automatic deployment mode detection. By default, the mode is detected from the `target` field (`all` or absent means monolithic, any other value means microservices). The migrated configuration is printed to `stdout`. Warnings are printed to `stderr`. ### Examples Monolithic mode (no Kafka flags needed): ```bash tempo-cli migrate config old-config.yaml > new-config.yaml ``` Microservices mode: ```bash tempo-cli migrate config --kafka-address=kafka:9092 --kafka-topic=tempo-traces old-config.yaml > new-config.yaml ``` {{< admonition type="warning" >}} - The output is a starting point for your 3.0 configuration. Always review it before deploying. - If your configuration uses legacy (flat) overrides, you must run `tempo-cli migrate overrides-config` first. - If your configuration references an external per-tenant overrides file (`per_tenant_override_config`), you must manually add `compaction_disabled: true` for each tenant in that file. - YAML comments and key ordering from the original file are not preserved. - Remove `compaction_disabled: true` from overrides after fully decommissioning your 2.x deployment. {{< /admonition >}} ## Analyse block Analyses a block and outputs a summary of the block's generic attributes. It's of particular use when trying to determine candidates for dedicated attribute columns. The output includes span, resource, and event attributes with cardinality and size information. Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. - `block-id` The block ID as UUID string. Options: - [Backend options](#backend-options) - `--num-attr <value>` Number of attributes to output (default: 20) - `--num-int-attr <value>` Number of integer attributes to display. If set to 0, uses the `--num-attr` value (default: 5) - `--blob-threshold <value>` Mark attributes as blob candidates when their dictionary size per row group exceeds this value. Set to 0 to disable. (default: 4MiB) - `--include-well-known` Include well-known attributes in the analysis. Enable when generating dedicated columns for vParquet5 or higher. (default: false) - `--generate-jsonnet` Generate Jsonnet overrides for dedicated columns - `--generate-cli-args` Generate command-line arguments for the parquet conversion command - `--int-percent-threshold <value>` Threshold for integer attributes in dedicated columns (default: 0.05) - `--str-percent-threshold <value>` Threshold for string attributes in dedicated columns (default: 0.03) - `--simple-summary` Print only a single line of top attributes (default: false) - `--print-full-summary` Print full summary of the analysed block (default: true) Example: ```bash tempo-cli analyse block --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0 ``` Example with blob detection: ```bash tempo-cli analyse block --blob-threshold=4MiB --generate-jsonnet --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0 ``` ## Analyse blocks Analyses all blocks in a given time range and outputs a summary of the blocks' generic attributes. It's of particular use when trying to determine candidates for dedicated attribute columns. The output includes span, resource, and event attributes with cardinality and size information. Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single-tenant setups. Options: - [Backend options](#backend-options) - `--num-attr <value>` Number of attributes to output (default: 20) - `--num-int-attr <value>` Number of integer attributes to display. If set to 0, uses the `--num-attr` value (default: 5) - `--min-compaction-level <value>` Minimum compaction level to include in the analysis (default: 3) - `--max-blocks <value>` Maximum number of blocks to analyze (default: 10) - `--max-start-time <value>` Oldest start time for a block to be processed. RFC3339 format (default: disabled) - `--min-start-time <value>` Newest start time for a block to be processed. RFC3339 format (default: disabled) - `--blob-threshold <value>` Mark attributes as blob candidates when their dictionary size per row group exceeds this value. Set to 0 to disable. (default: 4MiB) - `--include-well-known` Include well-known attributes in the analysis. (default: false) - `--jsonnet` Generate Jsonnet overrides for dedicated columns - `--cli` Generate command-line arguments for the parquet conversion command - `--int-percent-threshold <value>` Threshold for integer attributes in dedicated columns (default: 0.05) - `--str-percent-threshold <value>` Threshold for string attributes in dedicated columns (default: 0.03) - `--simple-summary` Print only a single line of top attributes (default: false) - `--print-full-summary` Print full summary of the analysed block (default: true) Example: ```bash tempo-cli analyse blocks --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant ``` Example with blob detection and Jsonnet output: ```bash tempo-cli analyse blocks --blob-threshold=4MiB --jsonnet --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant ``` ## Suggest columns Suggests dedicated columns for a tenant based on analysis of blocks. This command analyzes block data and outputs configuration recommendations in YAML or Jsonnet format that can be used to configure dedicated attribute columns. ```bash tempo-cli suggest columns <tenant-id> ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. Options: - [Backend options](#backend-options) - `--block-id <value>` Specific block ID to analyse. If not provided, analyzes multiple blocks. - `--min-compaction-level <value>` Minimum compaction level to analyse (default: 3) - `--max-blocks <value>` Maximum number of blocks to analyse (default: 10) - `--num-attr <value>` Number of attributes to display (default: 20) - `--num-int-attr <value>` Number of integer attributes to display. If set to 0, uses the `--num-attr` value (default: 5) - `--int-percent-threshold <value>` Threshold for integer attributes put in dedicated columns (default: 0.05) - `--str-percent-threshold <value>` Threshold for string attributes in dedicated columns (default: 0.03) - `--include-well-known` Include well-known attributes in the analysis (default: true) - `--blob-threshold <value>` Convert column to blob when dictionary size reaches this value (default: 4MiB) - `--max-start-time <value>` Oldest start time for a block to be processed. RFC3339 format. - `--min-start-time <value>` Newest start time for a block to be processed. RFC3339 format. - `-o, --out <value>` File to write output to. If not specified, output is printed to stdout. - `-f, --format <jsonnet|yaml>` Output format (default: yaml) Example outputting YAML to a file: ```bash tempo-cli suggest columns -c ./tempo.yaml --format yaml --out suggestions.yaml single-tenant ``` Example outputting Jsonnet for use in overrides: ```bash tempo-cli suggest columns -c ./tempo.yaml --format jsonnet single-tenant ``` ## Generate attribute index {{< admonition type="warning" >}} This command is EXPERIMENTAL and meant to facilitate experimentation with different kinds of indexes. {{< /admonition >}} Generate an attribute index for a parquet block. This creates an index file that can be used for faster attribute lookups. ```bash tempo-cli gen attr-index <input-path> ``` Arguments: - `input-path` Path to the input parquet block directory. Options: - `--add-intrinsics` Add intrinsic attributes to the index such as name, kind, status, and others. - `--index-types <rows|codes|rows,codes>` Type of index to generate (default: rows,codes) Example: ```bash tempo-cli gen attr-index ./path/to/block ``` Example with intrinsic attributes: ```bash tempo-cli gen attr-index --add-intrinsics ./path/to/block ``` ## Experimental traces diff {{< admonition type="warning" >}} This command is experimental. The output format and behavior may change in future releases. {{< /admonition >}} Compare two local trace JSON files. The default `trace-patch-v0` format returns the complete mechanical change list. You can instead request a compact native summary or a composed summary with a size-bounded patch. Use this command to compare traces captured at different times or from different environments, for example, to understand how a deployment changed trace structure. ```bash tempo-cli experimental traces-diff --trace-a <BASELINE_PATH> --trace-b <COMPARISON_PATH> ``` Arguments: - `--trace-a <path>` (required) Path to the baseline trace JSON file. - `--trace-b <path>` (required) Path to the comparison trace JSON file. Options: - `--format <value>` Output format: `trace-patch-v0` (default), `trace-summary-v0-native`, or `trace-summary-v0-composed`. - `-o, --out <path>` File to write output to. If not specified, output is printed to `stdout`. - `--pretty` Pretty-print JSON output. The input files can be either raw OpenTelemetry JSON traces or Tempo `TraceByIDResponse` JSON responses. The `trace-summary-v0-composed` format always includes a `trace-summary-v0-native` document. If the serialized `trace-patch-v0` document is no larger than 64 KiB, it is included in `patch`. Otherwise, `patchOmitted` reports its size and the reason `over_budget`; rerun with `--format trace-patch-v0` to retrieve the full patch. Example: ```bash tempo-cli experimental traces-diff --trace-a baseline.json --trace-b compare.json --pretty ``` Example writing output to a file: ```bash tempo-cli experimental traces-diff --trace-a baseline.json --trace-b compare.json -o diff-output.json ``` Example producing the native summary: ```bash tempo-cli experimental traces-diff \ --trace-a baseline.json \ --trace-b compare.json \ --format trace-summary-v0-native \ --pretty ``` The native summary calculates latency, the sum of inclusive span durations, topology, and significant per-service duration drift from the normalized traces. It combines those values with matcher-derived change and error counts. `changedServices` and service rollups cover all matcher changes and significant duration drift; `changedServices` also includes structure-only changes, which do not have service rollups. Warnings report partial inputs, high-cardinality span names, duplicate span IDs, ambiguous duplicate-span matching, empty traces, and invalid durations. Invalid spans contribute zero to summary duration aggregates. ## Drop traces by ID Rewrites all blocks for a tenant that contain specific trace IDs. The traces are dropped from the new blocks and the rewritten blocks are marked compacted so they will be cleaned up. ```bash tempo-cli rewrite-blocks drop-traces <tenant-id> <trace-ids> ``` Arguments: - `tenant-id` The tenant ID. Use `single-tenant` for single tenant setups. - `trace-ids` The comma-separated trace IDs to drop (also supports single trace ID). Options: - [Backend options](#backend-options) - `--drop-trace` By default, this command runs in dry run mode. Supplying this argument causes it to actually rewrite blocks with the traces dropped. - `--background` Run in background mode (default: false). Suppresses progress dots for use in automated scripts. ### Examples Dry run (default) to see which blocks would be affected: ```bash tempo-cli rewrite-blocks drop-traces --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2 ``` Drop one trace (actually perform the operation): ```bash tempo-cli rewrite-blocks drop-traces --drop-trace --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2 ``` Drop multiple traces: ```bash tempo-cli rewrite-blocks drop-traces --drop-trace --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2,111fa1850042aea83c17cd7e674210b8 ``` ## Redact traces Remove traces containing personally identifiable information or other sensitive data from object storage without waiting for retention to expire. {{< admonition type="warning" >}} Redaction rewrites blocks in object storage and can't be undone. You're responsible for verifying the selection. If you redact by query, confirm in Grafana Explore that the query selects the right traces, then run it with `--dry-run` and check the match count. Explore shows only a sample; redaction removes every matching trace across the tenant. {{< /admonition >}} The `redact` command submits a redaction request to the [backend scheduler](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/#backend-scheduler). The scheduler creates jobs that rewrite affected blocks in object storage to remove the specified traces. Unlike [`drop-traces`](#drop-traces-by-id), which operates directly on object storage from the CLI, `redact` delegates the work to the backend scheduler over gRPC. ```bash tempo-cli redact --tenant=<TENANT_ID> --trace-id=<TRACE_ID> [--trace-id=<TRACE_ID> ...] <scheduler-address> ``` ```bash tempo-cli redact --tenant=<TENANT_ID> --query=<TRACEQL_QUERY> [--start=<START> --end=<END>] <scheduler-address> ``` Arguments: - `scheduler-address` The backend scheduler gRPC address (`host:port`). Options: - `--tenant <value>` **(required)** Tenant ID. - `--trace-id <value>` Trace ID to redact, in hex format. Repeat the flag for several traces in one request (`--trace-id=<ID> --trace-id=<ID>`, not comma-separated), up to 1000. Every job the redaction creates carries the whole list, so a longer list costs one copy per block; use `--query` instead. Mutually exclusive with `--query`. - `--query <value>` TraceQL query selecting the traces to redact, for example `{ span.http.status_code = 500 }`. Mutually exclusive with `--trace-id`. The query is restricted to a single spanset filter over the matched span's own `resource.*` or `span.*` attributes, joined by `&&` or `||`: `=` comparisons, and the existence check `attr != nil`. Regular expressions, `!=` against a value, ordered comparisons, `parent.`-scoped attributes, and pipelines or aggregates aren't supported. - `--dry-run` Evaluate the selector without rewriting any blocks. After the dry-run jobs complete, match counts are added to `tempo_backend_scheduler_redaction_traces_found_total` (`mode="dry_run"`). The command doesn't print the count (default: `false`). - `--start <value>` Start of the time window. Accepts `now`, a relative offset such as `now-7d`, or an RFC3339 timestamp. Must be given with `--end`, must be before `--end`, and cannot be combined with `--trace-id`. Omit both bounds to redact the whole tenant. - `--end <value>` End of the time window. Same forms as `--start`. Must be given with `--start`. - `--tls` Use TLS for the gRPC connection (default: `false`). - `--tls-server-name <value>` Override the TLS server name (SNI). - `--tls-ca <value>` Path to a PEM-encoded CA certificate file. You must provide exactly one of `--trace-id` or `--query`. Providing both, or neither, returns an error before the request is submitted. `attr != nil` is rewritten to an existence check, so it selects spans that **have** the attribute, not the ones missing it. Other `!=` forms are refused. This can be broader than it looks on a large tenant (`span.user_id != nil` matches every span carrying a user ID), so preview with `--dry-run` first. A tenant can have only one redaction in progress at a time, dry runs included. A submission made while an earlier one is still running, or still in its quiescence period, is rejected. A trace-ID list larger than the cap therefore has to be redacted as successive batches rather than several at once, which is another reason to prefer `--query`. On success, the command prints the batch ID and the number of jobs created: ``` batch_id: <BATCH_ID> jobs_created: <COUNT> ``` When `--dry-run` is set, the command also prints a line indicating that no blocks were rewritten: ``` batch_id: <BATCH_ID> jobs_created: <COUNT> mode: dry-run (jobs will report match counts; no blocks will be rewritten) ``` ### Before you submit a redaction Redaction only rewrites blocks that already exist in object storage when you submit. Traces still held by block-builders aren't covered, so recently ingested traces can survive a run. 1. Stop ingesting the sensitive data at its source. 1. Wait for the current blocks to flush to object storage. Blocks flush on an interval of a few minutes; allow around 10 minutes to be safe, so the traces you want to remove land in blocks the redaction can reach. If you use `--query`: 1. [Check your redaction query in Grafana Explore](#check-your-redaction-query-in-grafana-explore) to confirm it selects the right traces. 1. Run the same query with `--dry-run`. The command returns as soon as jobs are created; it doesn't print the match count. Wait until those jobs complete. 1. Monitor progress on the [`/status/backendscheduler`](/docs/tempo/<TEMPO_VERSION>/api_docs/#backend-scheduler-job-status) endpoint. 1. Read `tempo_backend_scheduler_redaction_traces_found_total` for your tenant with `mode="dry_run"`. The metric is a counter that increments when each job finishes, so use an increase over the run or the **Dry-run Blast Radius / h** panel on the **Tempo - Backend Work** dashboard. A value of zero can mean no matches or that jobs haven't finished yet. For the metric and dashboard, refer to [Key metrics](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/#key-metrics). If the count is far larger than the Explore sample, narrow the query first. Then submit the redaction. For commands, refer to [Examples](#examples). ### Check your redaction query in Grafana Explore Use this procedure when you redact with `--query`. Enter the same query string you pass to `--query`. 1. In Grafana, go to **Explore** and select your Tempo data source. 1. For **Query type**, select **TraceQL**, then enter your redaction query, for example `{ span.http.status_code = 500 }`. 1. Select **Run query** and inspect the matching traces in the results. For help building and running TraceQL queries, refer to [TraceQL queries in Grafana](/docs/tempo/<TEMPO_VERSION>/traceql/). The `--query` option accepts only a restricted subset of TraceQL: a single spanset filter with positive equality matchers. That subset is deliberately narrow so a redaction query can't widen its own match set the way negation or regular expressions could. A broader query that works in Explore may be rejected at submission. Explore shows a sample; use `--dry-run` for the count before you apply the redaction. ### Redact a time window A redaction with no window covers every block the tenant has, which keeps the tenant's compaction paused for the whole run and lets its block list grow. `--start` and `--end` scope a redaction to a time range, so a large tenant can be redacted in slices with compaction recovering in between. ```bash tempo-cli redact --tenant=my-tenant --query='{resource.namespace = "checkout"}' --start=now-7d --end=now-6d localhost:9095 ``` Both bounds are inclusive and must both be supplied. Blocks whose data range overlaps the window are read, as are blocks whose recorded range is unusable — those are included rather than skipped, so that a block whose timestamps cannot be judged is never silently left behind. Inside each block the window bounds the scan, and a trace is redacted if any part of it overlaps: a trace that starts before the window and ends inside it is removed in full, including its earlier spans. A window cannot be combined with `--trace-id`. The window scopes which blocks are read and is not applied per trace, so the pair would remove each listed trace only from the blocks that happen to overlap and leave the rest of it in place while reporting success. Redact by trace ID without a window. The window is resolved to absolute timestamps when the command runs, so a long redaction does not drift forward into data that arrived after it started. Traces outside every window you run are left in place. Repeat the command for each slice, but expect to wait between slices. A finished redaction is held briefly before it is cleared, and a second submission for the same tenant is rejected with `AlreadyExists` until that completes — a couple of minutes at the default maintenance interval. {{< admonition type="note" >}} A single run doesn't remove everything. Two things bound what it covers: - Redaction snapshots the tenant's block list at submission and only rewrites those blocks (plus output from compactions that were already running). Traces still held by block-builders, or ingested after you submit, are untouched, so a single pass over a live tenant is never complete. - Search results are cached, so re-running the same search can still list redacted traces until that cache entry expires. To confirm a redaction, query by trace ID or vary the time range so the cache key differs. {{< /admonition >}} {{< admonition type="warning" >}} A backend-worker that predates `--start`/`--end` does not respect the window and scans each block it is given in full. The job still reports success, so there is no signal afterwards and the removed traces cannot be recovered. Wait until every worker for the cell supports the window before submitting a windowed redaction; an unwindowed redaction is unaffected. {{< /admonition >}} Monitor job progress through the [`/status/backendscheduler`](/docs/tempo/<TEMPO_VERSION>/api_docs/#backend-scheduler-job-status) endpoint. ### Examples Redact a single trace: ```bash tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d localhost:9095 ``` Redact multiple traces in one request: ```bash tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d --trace-id=00000000000000000000000000000001 localhost:9095 ``` Preview the traces a query would match, without rewriting any blocks: ```bash tempo-cli redact --tenant=my-tenant --query='{ span.http.status_code = 500 }' --dry-run localhost:9095 ``` After you preview, redact all traces matching that query: ```bash tempo-cli redact --tenant=my-tenant --query='{ span.http.status_code = 500 }' localhost:9095 ``` With TLS and a custom CA: ```bash tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d --tls --tls-ca=/path/to/ca.pem scheduler.example.com:9095 ``` --- ### Sources Tempo Operations Verify Container Images (`docs/sources/tempo/operations/verify-container-images.md`) --- title: Verify signed container images menuTitle: Verify container images description: Verify the cosign signature and SLSA build provenance of the container images Tempo publishes, before you deploy them. weight: 650 --- # Verify signed container images Starting with Tempo 3.1, Grafana Labs cryptographically signs every published Tempo container image and attaches a SLSA build provenance attestation to it. Use these artifacts to confirm that an image actually came from the `grafana/tempo` repository and to trace it back to the exact source commit and CI run that built it, before you pull it into a cluster. ## Which images are signed The following images published to [Docker Hub](https://hub.docker.com/u/grafana) are signed and attested: - `grafana/tempo` - `grafana/tempo-vulture` - `grafana/tempo-query` - `grafana/tempo-cli` Signing and attestation run as a best-effort step after each image is published. They don't gate the release or block a deployment, so a signing failure never prevents an image from being available. It only means that image doesn't verify until it's re-signed. ## What you get Each image ships with two supply chain artifacts: - **A keyless cosign signature.** Proves the image was produced by the `sign-and-attest.yml` workflow in the `grafana/tempo` repository and hasn't been tampered with since. Grafana Labs doesn't hold or manage a private signing key: the workflow signs using a short-lived certificate issued by [Sigstore's](https://www.sigstore.dev/) Fulcio CA in exchange for a GitHub Actions OIDC token, and records the signing event in the public Rekor transparency log. - **A SLSA build provenance attestation.** Binds the image's digest to the source commit, workflow, and run that built it. This is what lets you answer "was this image actually built by the Tempo CI pipeline from a specific commit, or did it come from somewhere else?" Both are pushed as OCI referrers alongside the image, so they travel with it wherever the image is stored or mirrored. ## Before you begin Install the verification tools: - [cosign](https://docs.sigstore.dev/cosign/system_config/installation/) v2 or later, to verify the signature - The [GitHub CLI](https://cli.github.com/) (`gh`), to verify the provenance attestation ## Verify an image Pull the image by digest, or resolve the digest for a tag you already have, so you verify the exact artifact you're about to deploy rather than a tag that could later move to a different image: ```bash docker buildx imagetools inspect grafana/tempo:<TEMPO_VERSION> --format '{{.Manifest.Digest}}' ``` ### Verify the signature Run `cosign verify` against the digest-pinned image, checking the certificate identity against the `sign-and-attest.yml` workflow in `grafana/tempo` and the GitHub Actions OIDC issuer: ```bash cosign verify \ --certificate-identity-regexp '^https://github\.com/grafana/tempo/\.github/workflows/sign-and-attest\.yml@' \ --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \ grafana/tempo@<DIGEST> ``` A successful verification prints the certificate details and a `Verified OK` message. ### Verify the provenance attestation Run `gh attestation verify` against the same digest-pinned image, scoped to the `grafana/tempo` repository: ```bash gh attestation verify oci://grafana/tempo@<DIGEST> --repo grafana/tempo ``` A successful verification confirms the attestation is signed by `grafana/tempo` and prints the predicate type and the workflow run that produced it. ## Result If both commands succeed, you've confirmed that the image digest you're about to deploy was built and published by the Tempo CI pipeline from a known commit, and hasn't been altered since. If either command fails, don't deploy the image, and treat it as unverified until you can confirm why verification failed. {{< admonition type="note" >}} The verification identity (the workflow path plus the OIDC issuer) is a public contract. It stays the same across all four signed images, so you can reuse the same `--certificate-identity-regexp` and `--certificate-oidc-issuer` values regardless of which component you're verifying. {{< /admonition >}} --- ### Sources Tempo Reference Tempo Architecture Block Format (`docs/sources/tempo/reference-tempo-architecture/block-format.md`) --- title: Block format menuTitle: Block format description: How Tempo stores trace data in Apache Parquet blocks. weight: 500 topicType: concept versionDate: 2026-08-25 --- # Block format Tempo stores trace data in [Apache Parquet](https://parquet.apache.org) format. Parquet is a columnar storage format that enables efficient querying of specific attributes without reading entire traces. ## Why Parquet Columnar storage is a natural fit for trace data. A TraceQL query like `{ span.http.status_code = 500 }` only needs to read the `http.status_code` column, not the entire trace. This dramatically reduces the amount of data read from storage. Columnar data also compresses well because values in a column tend to be similar (for example, all service names or all status codes). ## Block structure A block is a directory in object storage containing several files. | File | Purpose | | --------------- | ------------------------------------------------------------------------------------- | | `meta.json` | Block metadata: time range, tenant, block ID, `replaces` field for atomic replacement | | `data.parquet` | Trace data in columnar format | | Bloom filters | Probabilistic data structures for efficient trace ID lookups | | Index | Maps trace IDs to row groups within `data.parquet` | | `nocompact.flg` | Temporary flag preventing compaction (present during block-builder flushes) | Blocks are stored under `<tenant-id>/<block-id>/` in object storage. ### `meta.json` The `meta.json` file makes a block visible to the read path. It contains the block ID, tenant ID, start and end timestamps, total number of objects (traces). A block is invisible to queriers until `meta.json` exists. The block-builder uses this property to ensure atomicity during flushes. ## Schema Tempo uses a heavily nested Parquet schema. Each row represents a trace, and rows are sorted by trace ID within the file. Spans are stored inside the row as nested, repeated groups that mirror the OpenTelemetry hierarchy: resource spans, scope spans, and spans. The schema has columns for intrinsic fields (trace ID, span ID, parent span ID, span name, span kind, span status, duration, start time, root service name, root span name), resource attributes (for example, `service.name`, `deployment.environment`), and span attributes (for example, `http.method`, `http.status_code`). Each span contributes one value to every span-level column, so a single trace row holds many values per column. ### Intrinsic fields vs generic attributes A small number of fields are stored as top-level columns in the schema. These include `service.name` on resources and `status.code` on spans. Querying these fields is always efficient because they have their own Parquet columns. All other attributes (both resource and span) are stored in a generic `Attrs` column as key-value pairs. Querying these requires scanning the attribute map, which is slower. ### Dedicated attribute columns You can promote frequently queried attributes from the generic column to their own dedicated Parquet columns for better query performance. This is configured centrally in `storage.trace.block` and applies to all block-producing components (live-store and block-builder). Dedicated columns are assigned dynamically per block based on the configuration at the time the block is built. ```yaml storage: trace: block: parquet_dedicated_columns: - name: http.method type: string scope: span - name: deployment.environment type: string scope: resource ``` Refer to [Dedicated attribute columns](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/dedicated_columns/) for configuration details and recommendations. ## Block format versions Tempo uses versioned block formats. As of Tempo 3.1, new blocks are written in vParquet5 by default. | Version | Status | | --------- | ----------------------------------------- | | vParquet4 | Supported | | vParquet5 | Default and latest | If you omit `version`, Tempo writes vParquet5. To keep writing vParquet4, set the version explicitly: ```yaml storage: trace: block: version: vParquet4 ``` Existing blocks in older formats remain readable. New blocks are always written in the configured version. No data migration is required when the default changes. ## Related resources Refer to the [Apache Parquet schema](https://grafana.com/docs/tempo/<TEMPO_VERSION>/operations/schema/) documentation for the full schema details and the [Apache Parquet block format configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/parquet/) for configuration options. --- ### Sources Tempo Reference Tempo Architecture Deployment Modes (`docs/sources/tempo/reference-tempo-architecture/deployment-modes.md`) --- title: Deployment modes menuTitle: Deployment modes description: Monolithic and microservices deployment modes in detail. weight: 400 topicType: concept versionDate: 2026-03-20 --- # Deployment modes Tempo can be deployed in monolithic or microservices mode. Microservices mode requires a Kafka-compatible system. Monolithic mode doesn't use Kafka. _Monolithic mode_ was previously called _single binary mode_. {{< admonition type="note" >}} The previous _scalable monolithic mode_, also known as _scalable single binary mode_ or SSB, has been removed in v3.0. {{< /admonition >}} All components are compiled into the same binary. The `-target` command-line parameter, or `target` in configuration, determines which components run in a given process. The default target is `all`, which is the monolithic deployment mode. ```bash tempo -target=all ``` Refer to the [command line flags](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/command-line-flags/) documentation for more information on the `-target` flag. ## Monolithic mode In monolithic mode, the required components run in a single process using `-target=all`, which is the default. Components that are only needed in microservices mode, such as the block-builder, are excluded. No Kafka is required. The distributor pushes trace data in-process directly to the live-store and metrics-generator. Traces are flushed to the configured storage backend without an intermediate message queue. Object storage is recommended for production deployments. ### When to use monolithic mode Monolithic mode is suitable for getting started, development environments, and low to moderate trace volumes where operational simplicity matters more than independent scaling. ### Limitations Components share the same resource pool. A spike in query load can affect write throughput and vice versa. There is no independent scaling. At higher volumes, memory pressure from collocated components, particularly the live-store and querier, can cause out-of-memory issues. ### Resource considerations Monolithic instances need enough memory to handle the live-store's in-memory trace buffer, the querier's concurrent job execution, and the backend worker's memory for block merging. As volume increases, the instance is limited by whichever component is most resource-hungry. ### Example Refer to [Docker Compose examples in the Tempo repository](https://github.com/grafana/tempo/tree/main/example/docker-compose/) for sample deployments. For an annotated example configuration for Tempo, refer to the [Introduction to MLTP](https://github.com/grafana/intro-to-mltp) repository, which includes a [sample `tempo.yaml`](https://github.com/grafana/intro-to-mltp/blob/main/tempo/tempo.yaml) for a monolithic instance. ## Microservices mode In microservices mode, each component runs as a separate process with its own `-target`. This is the recommended mode for production. The configuration associated with each component's deployment specifies a `target`. For example, to deploy a `querier`, the configuration would contain `target: querier`. A command-line deployment may specify the `-target=querier` flag. ### When to use microservices mode Use microservices mode for production deployments, high trace volumes requiring independent scaling, and environments where high availability is important. ### Advantages Microservices mode provides independent scaling. You can scale block-builders for write throughput, queriers for query performance, and live-stores for recent data capacity, all independently. Failure domains are isolated: a querier OOM doesn't affect data ingestion, and a block-builder restart doesn't affect query availability. Live-stores can be deployed across availability zones for high availability. Each component gets exactly the resources it needs, avoiding the over-provisioning required in monolithic mode. ### Component scaling guidelines | Component | Scaling strategy | Notes | |---|---|---| | Distributor | Horizontal | Stateless. Scale based on ingestion rate. | | Block-builder | Horizontal | Bounded by Kafka partition count. Scale based on data volume. | | Live-store | Horizontal | Bounded by Kafka partition count. Scale based on recent data query volume and memory. | | Query frontend | Vertical | Keep to 2 replicas. Scale up CPU/RAM rather than adding replicas. | | Querier | Horizontal | Scale based on query concurrency and latency requirements. | | Backend worker | Horizontal | Handles compaction and retention. Scale based on block count and compaction lag. | | Metrics-generator | Horizontal | Scale based on trace volume and number of generated series. | ### Kafka as the connecting fabric In microservices mode, Kafka is the primary communication channel for the write path. Components don't communicate directly for data transfer. Distributors write to Kafka. Block-builders, live-stores, and metrics-generators each consume from Kafka independently. Queriers contact live-stores over gRPC for recent data. All components access object storage for block data. Adding or removing instances of any component doesn't require reconfiguring other components, aside from Kafka partition management. ### Example Refer to the [distributed Docker Compose example](https://github.com/grafana/tempo/tree/main/example/docker-compose/distributed) in the Tempo repository. ## Components by deployment mode Not all components and configuration blocks apply to both modes. The following table summarizes which components run in each mode and how shared components differ. | Component | Config block | Monolithic | Microservices | |---|---|---|---| | Distributor | `distributor` | Pushes data in-process to the live-store and metrics-generator | Writes data to Kafka | | Ingest | `ingest` | Not used | Kafka connection settings for the write path | | Block-builder | `block_builder` | Not used | Consumes from Kafka, builds Parquet blocks, flushes to object storage | | Live-store | `live_store` | Receives data directly from the distributor | Consumes from Kafka | | Live-store client | `live_store_client` | Querier-to-live-store client (runs in-process) | gRPC client for querier-to-live-store communication | | Query-frontend | `query_frontend` | Runs in-process | Runs as a separate process | | Querier | `querier` | Runs in-process | Runs as a separate process | | Backend scheduler | `backend_scheduler` | Runs in-process | Runs as a separate process | | Backend worker | `backend_worker` | Runs in-process | Runs as a separate process | | Metrics-generator | `metrics_generator` | Optional, runs in-process | Optional, runs as a separate process | | Storage | `storage` | Storage backend for trace data (object storage recommended; local supported for dev/test) | Object storage for trace data | | Memberlist | `memberlist` | Cluster membership | Cluster membership | | Overrides | `overrides` | Per-tenant limits | Per-tenant limits | For full configuration details, refer to the [configuration reference](/docs/tempo/<TEMPO_VERSION>/configuration/). ## Migrating between modes Moving from monolithic to microservices mode involves deploying individual components with appropriate `-target` flags, pointing all components at the same Kafka cluster, object storage, and memberlist, then scaling down the monolithic instances. Since Kafka provides durability, no data is lost during the transition. Live-stores replay from Kafka on startup, and block-builders continue from their last committed offset. ## Related resources Refer to [Plan your Tempo deployment](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/plan/) for deployment planning and sizing guidance. --- ### Sources Tempo Reference Tempo Architecture Object Storage (`docs/sources/tempo/reference-tempo-architecture/object-storage.md`) --- title: Object storage menuTitle: Object storage description: How Tempo uses object storage for long-term trace data retention. weight: 600 topicType: concept versionDate: 2026-03-20 --- # Object storage Object storage is the long-term storage backend for all trace data in Tempo. Block-builders write blocks to it, queriers read from it, and backend workers maintain it. ## Supported backends Tempo supports three major object storage APIs: - Amazon S3 (and S3-compatible systems, for example, MinIO) - Google Cloud Storage (GCS) - Microsoft Azure Blob Storage A local filesystem backend is also available for development and testing. ## Storage layout Data in object storage is organized by tenant and block: ``` <bucket>/ <tenant-id>/ <block-id>/ meta.json data.parquet bloom-0 bloom-1 ... index ``` Each tenant has its own directory. Within a tenant, each block is a directory containing the block's files. ### Blocklist The blocklist is the set of all known blocks for a tenant. Backend workers maintain it by periodically scanning object storage for `meta.json` files and writing a per-tenant block index. Queriers and query frontends read this tenant index to determine which blocks to search for a given query. They fall back to scanning object storage for `meta.json` files only when the tenant index is unavailable or too stale. The blocklist is distributed across the cluster so that not every component needs to poll storage directly. ### Tenant isolation Tenants are fully isolated at the storage level. Each tenant's blocks are in a separate directory prefix. There's no cross-tenant data sharing or block merging. ## Durability model With Tempo 3.0's Kafka-based architecture, durability works in layers. Kafka provides immediate durability. Once data is acknowledged by Kafka, it's safe even if all Tempo components crash. Object storage provides long-term durability. Once the block-builder flushes a block, the data is durably stored and independent of Kafka. Kafka retention bridges the gap: Kafka retains data long enough for block-builders to consume and flush it. If a block-builder is slow or restarting, Kafka holds the data until it's processed. There's no single point of failure for data durability. Kafka and object storage together provide end-to-end safety. ## Performance considerations ### Read path Query performance depends on how efficiently queriers can access blocks. Larger blocks (from compaction) reduce the number of blocks to search but increase individual block read time. Caching bloom filters, Parquet pages, and footers at the querier level significantly reduces object storage reads. Promoting frequently queried attributes to dedicated Parquet columns reduces the amount of data read per query. ### Write path Block-builder write performance is generally bounded by Kafka consumption rate, local disk speed (blocks are built on scratch disk before upload), and upload bandwidth to object storage. ### Cost Object storage costs are primarily driven by storage volume (total data retained, controlled by retention period and compaction efficiency) and API operations (GET/PUT/LIST calls). Compaction reduces LIST costs by consolidating blocks. Caching reduces GET costs. ## Configuration ```yaml storage: trace: backend: s3 # or gcs, azure, local s3: bucket: tempo-traces endpoint: s3.amazonaws.com ``` ## Related resources Refer to the [storage configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#storage) for the full list of options. --- ### Sources Tempo Reference Tempo Architecture Partition Ring (`docs/sources/tempo/reference-tempo-architecture/partition-ring.md`) --- title: Partition ring menuTitle: Partition ring description: How Tempo manages partition ownership and lifecycle. weight: 300 topicType: concept versionDate: 2026-03-20 --- # Partition ring The partition ring is the mechanism Tempo uses to track which partitions exist, their current state, and which components own them. By default, the partition ring propagates across the cluster via memberlist gossip and is central to how distributors, live-stores, and block-builders coordinate. ## Tempo partitions vs Kafka partitions Tempo maintains its own concept of partitions that are logically distinct from Kafka partitions. While there's typically a 1:1 mapping, the partition ring gives Tempo independent control over partition states (pending, active, inactive), ownership (which live-store owns each partition), and lifecycle management (creating, activating, and deactivating partitions without modifying Kafka's configuration). ## Partition states Each partition in the ring has one of three states. ### Pending Pending is the initial state when a new partition is created. No reads or writes occur. A partition enters pending state when a new live-store starts and creates a partition that doesn't yet exist in the ring. It stays in pending until enough owners have registered and a minimum waiting period has elapsed, at which point the owning live-store automatically promotes it to active. ### Active Active is the normal operating state. Distributors write data to active partitions, and queriers read from them. A partition transitions from pending to active once enough owners have registered for that partition and a configurable waiting duration has elapsed. This ensures that all availability zones have had time to register their live-store instances before traffic starts flowing. ### Inactive Inactive is the read-only state. Distributors stop writing to inactive partitions, but queriers can still read from them. A partition is marked inactive when scaling down. It must remain in this state long enough for the block-builder to flush all remaining data for this partition to object storage, and for queriers to stop relying on the live-store for this partition's recent data. After this grace period, you can safely remove the partition and its owning live-store. ## Ownership model ### Live-stores Each Tempo partition is owned by one live-store per availability zone. In a zone-aware deployment with two zones, each partition has two owners—one per zone. Both consume the same Kafka partition independently. When a live-store starts, it checks the ring for its assigned partition. If the partition exists, it joins as an owner. If not, it creates the partition in pending state and waits for enough owners to register. ### Distributors Distributors read the partition ring to determine which partitions are active. They only send data to active partitions. The ring tells distributors which Kafka partitions to write to. ### Block-builders Each block-builder instance computes which Kafka partitions it owns based on its ordinal ID and the `partitions_per_instance` setting. The partition ring indirectly affects block-builders because it determines which partitions receive data from distributors. ## Scaling ### Scaling up To scale up, deploy a new live-store instance. The live-store creates a new partition in the ring (pending state). After enough owners register and the waiting period elapses, the partition transitions to active, and distributors begin writing to the new partition. A corresponding Kafka partition must exist. Add Kafka partitions first if needed. ### Scaling down To scale down, mark the target partition as inactive while the live-store is still running. Distributors stop writing to it. Wait for the block-builder to flush remaining data to object storage, then remove the live-store instance. The partition is eventually cleaned up from the ring. Skipping the inactive step and abruptly removing a live-store causes recent data for that partition to become temporarily unavailable (unless a zone-aware replica exists). ## Memberlist propagation The partition ring state is propagated using memberlist, which uses a gossip protocol. Changes to the ring (new partitions, state transitions) propagate across the cluster within seconds under normal conditions. During network partitions or high cluster churn, propagation may be delayed. This can cause brief inconsistencies where different components have different views of the ring. Tempo handles this gracefully: distributors write to a partition that a live-store hasn't yet seen results in data that's picked up after the live-store catches up, and queriers contacting a live-store for a partition it doesn't own yet get an empty response, with the data eventually available from another live-store or from object storage. ## Related resources Refer to the [memberlist configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#memberlist) for ring propagation settings and the [ingest configuration](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#ingest) for partition-related settings. --- ### Sources Tempo Release Notes V3 0 (`docs/sources/tempo/release-notes/v3-0.md`) --- title: Version 3.0 release notes menuTitle: V3.0 description: Release notes for Grafana Tempo 3.0 weight: 700 --- # Version 3.0 release notes The Tempo team is pleased to announce the release of Grafana Tempo 3.0. This release gives you: - [New architecture](#new-architecture): Splits the write and read path, offering increased reliability and scalability, lower TCO and moving TraceQL metrics to general availability. - [TraceQL metrics generally available](#traceql-metrics): Production-ready trace-derived metrics at runtime with new comparison operators. Alerting on TraceQL metrics is an experimental capability. - [Metrics-generator improvements](#metrics-generator): Per-label cardinality limiting, service graph filtering, and span multiplier support for sampled environments. - [Trace redaction](#trace-redaction): New CLI command and API for hiding sensitive trace data. - [Span profiling](#span-profiling): Native Pyroscope integration links profiling data to trace spans. - [Query performance improvements](#query-performance-improvements): Doubled the number of dedicated columns and optimized TraceQL AST. To learn more about this release, refer to the [Tempo 3.0 blog post](https://grafana.com/blog/tempo-3-0-release-all-the-latest-features/). These release notes highlight the most important features and bug fixes. For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). {{< youtube id="JEOne1NHeyo" >}} ## New architecture Tempo 3.0 introduces a new architecture that replaces ingesters with a design that separates read and write paths. In microservices mode, a Kafka-compatible system provides durable buffering between ingestion and downstream components. In monolithic mode, Tempo runs all components in-process without Kafka. Previously known as "Project Rhythm" (experimental in 2.9 and 2.10), this architecture is now the default and only ingest path. For an overview of the components and data flow, refer to [Tempo architecture](/docs/tempo/<TEMPO_VERSION>/introduction/architecture/). ### Greater resilience In microservices mode, Kafka durably buffers trace data between ingestion and storage. Writes are acknowledged only after Kafka confirms receipt, ensuring durable ingestion even when downstream components are restarting or under load. The read and write paths are fully decoupled and isolated from each other. Failures or load spikes on one path don't affect the other. Live-stores serve recent data from multiple availability zones, providing high availability without deduplication overhead. ### Lower total cost of ownership The new architecture reduces costs in three ways. First, it eliminates duplicated responsibilities where ingesters and the metrics-generator both created and flushed blocks. Second, it reduces data in object storage. The previous architecture required replication factor 3 (RF3), which produced 2 to 2.5 times data duplication. The new design operates at RF1 with no duplication. Traces are sharded by ID to unique Kafka partitions, so each block is built exactly once. Third, better autoscaling support lowers resource waste during low-traffic periods. ### Simpler operations Each new component has a single, well-defined responsibility: block-builders create blocks, live-stores serve recent queries, and the backend scheduler and worker manage compaction and retention. This clear separation makes the system easier to reason about and operate. Compaction is now job-based, with centralized progress tracking, automatic rescheduling of failed jobs, and no duplicated work. ### What changed - Block-builders consume from Kafka and build blocks for object storage (microservices mode). - Live-stores serve recent data queries. - The backend scheduler and worker replace the compactor for compaction and retention. For full component details, refer to the [Tempo architecture](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/) pages. ### Upgrade to 3.0 For migration steps, breaking changes, and upgrade considerations, refer to [Migrate to 3.0](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/migrate-to-3/). You can also use the [`tempo-cli migrate config`](/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#migrate-config-command) command to migrate your configuration from 2.x to 3.0. [[PR 6982](https://github.com/grafana/tempo/pull/6982)] ## TraceQL metrics TraceQL metrics move to general availability in Tempo 3.0. You can now build dashboards directly from trace data using metrics queries, without relying on the metrics-generator to pre-compute them. Refer to [TraceQL metrics queries](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/) to learn more. {{< admonition type="note" >}} While TraceQL metrics are generally available, alerting on TraceQL metrics is an [experimental feature](https://grafana.com/docs/release-life-cycle/). Engineering and on-call support is not available for the alerting use case. {{< /admonition >}} ### Comparison operators in metrics queries With the new [comparison operators](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/functions/#comparison-operators), you can filter metrics results to only the data points that cross a threshold. Comparison operators are the foundation for alerting on trace-derived metrics: write a query that returns a value only when something is wrong, and point an alert rule at it. [[PR 6474](https://github.com/grafana/tempo/pull/6474)] For example, this query returns data only when a service exceeds 10 requests per second: ```traceql {} | rate() by (resource.service.name) > 10 ``` You can also filter on latency. This query finds endpoints where the average span duration exceeds one second: ```traceql { span:name = "GET /:endpoint" } | avg_over_time(span:duration) > 1s ``` ### Additional TraceQL improvements - The default [`max_duration`](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/) for metrics queries increases to one day, so you can analyze longer time ranges without adjusting configuration. Default step intervals now align with vParquet5 timestamp columns for better query performance. [[PR 6285](https://github.com/grafana/tempo/pull/6285), [PR 6413](https://github.com/grafana/tempo/pull/6413)] - An experimental faster read path is available for most metrics queries behind the query hint `spanonly_fetch=true` when `unsafe_query_hints` is enabled. A per-tenant override controls opt-in or opt-out. (PRs [#6359](https://github.com/grafana/tempo/pull/6359), [#6849](https://github.com/grafana/tempo/pull/6849)) - Tag name and tag value autocomplete ([search tags v2](/docs/tempo/<TEMPO_VERSION>/api_docs/#search-tags-v2)) now supports `OR` conditions. When you search for attribute values in Grafana or through the API, Tempo can match against multiple values in a single request, returning results faster. [[PR 6827](https://github.com/grafana/tempo/pull/6827)] ## Metrics-generator Tempo 2.10 introduced entity-based limiting for the metrics-generator. Tempo 3.0 builds on this with per-label cardinality control and new filter policies for both span metrics and service graphs. ### Per-label cardinality limiting The new per-label limiter prevents a single high-cardinality label from exhausting your entire series budget. Set `max_cardinality_per_label` as a per-tenant override to cap how many distinct values any one label can have, while leaving other labels unaffected. Accompanying demand-estimate metrics help you tune limits before they kick in. [[PR 6414](https://github.com/grafana/tempo/pull/6414), [documentation](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/metrics-generator/)] ### Span name sanitization (experimental) The `span_name` label often contributes the most to high cardinality in generated metrics. Applications that embed dynamic values in span names, for example, REST paths with user IDs (`GET /users/123`) or auto-generated operation names (`query-abc-def-ghi`), create a unique series for every distinct value. This drives up active series counts, increases storage and query costs, and can push you past cardinality limits. You can enable `span_name_sanitization` to solve this without changing instrumentation across your services. The option uses the DRAIN algorithm to learn patterns from incoming span names and replace variable segments with a `<_>` placeholder, reducing many unique span names down to a single representative series. Use span name sanitization when: - Your `span_name` label has high cardinality due to embedded IDs, timestamps, or request parameters. - You don't control the instrumentation that produces these span names, or changing it across all services isn't practical. - You want to reclaim active series budget without losing meaningful aggregation in your span metrics. To use the capability, set the `span_name_sanitization` option in the `overrides` block of your Tempo configuration: ```yaml overrides: defaults: metrics_generator: span_name_sanitization: "enabled" ``` Refer to [Reduce cardinality with span name sanitization](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-generator/reduce-cardinality/) for details. [[PR 6098](https://github.com/grafana/tempo/pull/6098)] ### Span metrics filter improvements The span metrics filter picks up an `include_any` policy, making it easier to capture specific spans, such as internal traffic from a single service, without opening up your filter to all internal spans. [[PR 6392](https://github.com/grafana/tempo/pull/6392), [documentation](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics/span-metrics-metrics-generator/#filtering)] ### Span multiplier If you use head-based sampling, your metrics don't reflect actual traffic volume. You can now set a `span_multiplier_key` per tenant to tell Tempo which span attribute holds the sampling ratio, so metric counts scale up to reflect your real request volume. [[PR 6260](https://github.com/grafana/tempo/pull/6260), [documentation](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics/span-metrics-metrics-generator/)] ### Service graph filter improvements Service graphs now support the same `filter_policies` that span metrics already had. This means you can exclude noisy internal traffic or health checks from your service graph, keeping it clean and focused on the service-to-service communication you actually care about. [[PR 6453](https://github.com/grafana/tempo/pull/6453), [documentation](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/)] ### Additional metrics-generator improvements - Support extracting span multiplier from W3C `tracestate` OpenTelemetry probability sampling threshold via the `enable_tracestate_span_multiplier` configuration option. [[PR 6684](https://github.com/grafana/tempo/pull/6684)] - Allow duplicate dimensions for span metrics and service graphs, supporting environments with different instrumentation libraries that use different attribute naming conventions. [[PR 6288](https://github.com/grafana/tempo/pull/6288)] - Allow `span_name_sanitization` to be set via the user-configurable overrides API. [[PR 6411](https://github.com/grafana/tempo/pull/6411)] - Deep validation for filter policies in the user-configurable overrides API. [[PR 6407](https://github.com/grafana/tempo/pull/6407)] ## Trace redaction Remove traces containing personally identifiable information or other sensitive data from object storage without waiting for retention to expire. The [`tempo-cli redact`](/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#redact-traces) command submits redaction jobs to the [backend scheduler](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/#backend-scheduler), which rewrites the affected blocks in object storage. You can monitor job progress through the [`/status/backendscheduler`](/docs/tempo/<TEMPO_VERSION>/api_docs/#backend-scheduler-job-status) endpoint. [[PR 6832](https://github.com/grafana/tempo/pull/6832)] When individual span redaction isn't enough, the `TraceRedactor` interface now supports hiding complete traces from query results through `ErrTraceHidden`. You can use this to enforce data governance policies that keep specific traces out of query results entirely. Refer to [Compaction](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/#job-types) for details on how redaction jobs are scheduled and executed. [[PR 6811](https://github.com/grafana/tempo/pull/6811)] ## Span profiling Tempo 3.0 adds span profiling through [`otelpyroscope`](https://github.com/grafana/otel-profiling-go), so you can jump from a slow span in a trace directly to the CPU or memory profile that shows what that span was doing. Instead of correlating traces and profiles manually, you get a direct link between the two in Grafana Pyroscope, making it faster to find the root cause of latency or resource issues at the code level. [[PR 7063](https://github.com/grafana/tempo/pull/7063)] Under the hood, Tempo attaches pprof goroutine labels (`span_id`, `span_name`) to OpenTelemetry spans and adds a `pyroscope.profile.id` attribute to root spans. Enable span profiling in your configuration file or with the CLI flag: ```yaml span_profiling: true ``` Span profiling requires an OTLP exporter. Set `OTEL_TRACES_EXPORTER`, `OTEL_EXPORTER_OTLP_ENDPOINT`, or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` in your environment. Refer to [command-line flags](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/command-line-flags/) for details. ## Query performance improvements Tempo 3.0 speeds up queries through storage and query engine improvements. vParquet5 is now production-ready with expanded dedicated column support, and the TraceQL engine rewrites queries more efficiently. ### vParquet5 vParquet5 is now production-ready. vParquet4 remains the default block format in this release. vParquet5 includes all the capabilities introduced as opt-in in 2.10, including event-scoped columns, blob detection, and array-valued dedicated columns, plus expanded dedicated column support. New in 3.0, the maximum number of dedicated string columns doubles to 20 per scope, so you can move even more of your most-searched attributes into fast dedicated storage. The `tempo-cli analyse blocks` command now recommends the optimal number for your data. [[PR 6282](https://github.com/grafana/tempo/pull/6282)] To enable vParquet5, set the block version in your storage configuration: ```yaml storage: trace: block: version: vParquet5 ``` After you enable vParquet5, Tempo writes new blocks in vParquet5 format while continuing to read existing vParquet4 blocks. No migration of existing data is required. For details, refer to [Apache Parquet block format](/docs/tempo/<TEMPO_VERSION>/configuration/parquet/) and [Dedicated attribute columns](/docs/tempo/<TEMPO_VERSION>/operations/dedicated_columns/). ### TraceQL AST optimization The TraceQL engine now automatically rewrites multiple conditions on the same attribute into a single array-aware check. This reduces the work the engine does per span and speeds up queries that filter on the same attribute more than once. For example, this query evaluates more efficiently in 3.0 because the engine combines both conditions on `span.http.status_code` into one pass: ```traceql { span.http.status_code >= 500 && span.http.status_code < 600 } ``` This is a breaking change that alters how `!=` and `!~` behave on array attributes and introduces stricter validation of regular expression literals. Refer to [TraceQL array matching changes](#traceql-array-matching-changes) in the upgrade considerations for details on what to check before upgrading. [[PR 6353](https://github.com/grafana/tempo/pull/6353)] ## Features and enhancements The most important features and enhancements in Tempo 3.0 are highlighted below. - Added `automemlimit` support for automatic GOMEMLIMIT configuration. Enable with `memory.automemlimit_enabled: true`. Refer to [Configuration](/docs/tempo/<TEMPO_VERSION>/configuration/#memory) for details. [[PR 6313](https://github.com/grafana/tempo/pull/6313)] - Added KEDA-based horizontal Pod autoscaling support for microservices deployment in Jsonnet. [[PR 6970](https://github.com/grafana/tempo/pull/6970)] - Enabled native histogram emission for all `promauto`-registered histograms, including `tempo_request_duration_seconds`. Both classic and native formats are emitted simultaneously; existing scrapers are unaffected. [[PR 6910](https://github.com/grafana/tempo/pull/6910)] - Added an extension mechanism for per-tenant overrides. Refer to [User-configurable overrides](/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/user-configurable-overrides/) for details. [[PR 6758](https://github.com/grafana/tempo/pull/6758)] - Added support for per-tenant left-padding of trace IDs. Refer to [Configuration](/docs/tempo/<TEMPO_VERSION>/configuration/#overrides) for the `left_pad_trace_ids` parameter. [[PR 6489](https://github.com/grafana/tempo/pull/6489)] - Improved attribute truncating observability with per-scope metric labels and rate-limited diagnostic logging. Refer to [Set max attribute size](/docs/tempo/<TEMPO_VERSION>/configuration/#set-max-attribute-size-to-help-control-out-of-memory-errors) for details. (PRs [#6400](https://github.com/grafana/tempo/pull/6400), [#6467](https://github.com/grafana/tempo/pull/6467)) - Exposed MinIO retry settings via S3 configuration. Refer to [Configuration](/docs/tempo/<TEMPO_VERSION>/configuration/#storage) for the `retry_max_attempts`, `retry_backoff_initial`, and `retry_backoff_max` parameters. [[PR 6561](https://github.com/grafana/tempo/pull/6561)] - Block builder now deduplicates spans within traces during block creation. Removed duplicates are tracked via the `tempo_block_builder_spans_deduped_total` metric. Refer to [Block-builder](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/block-builder/#span-deduplication) for details. [[PR 6539](https://github.com/grafana/tempo/pull/6539)] - Added `--header` flag to `query api` commands for passing custom headers. Refer to the [Tempo CLI documentation](/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#query-api-command) for usage details. [[PR 6768](https://github.com/grafana/tempo/pull/6768)] ## Upgrade considerations When [upgrading](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/upgrade/) to Tempo 3.0, be aware of these considerations and breaking changes. Tempo 3.0 is a major release with significant architectural changes. Use the `tempo-cli migrate config` command to automate configuration migration. Refer to the [Migrate from Tempo 2.x to 3.0](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/migrate-to-3/) guide for step-by-step instructions. ### Ingester removal The ingester module is removed entirely. All ingester-related configuration fields, CLI flags, alerts, and dashboard panels must be removed from your deployment. The write path is now exclusively handled by live-store and block-builder. Removed configuration sections: `ingester`, `ingester_client`, `compactor`, `metrics_generator_client`. The `ingest.enabled` field is also removed, but the `ingest` block itself is still required for microservices mode (for example, `ingest.kafka`). (PRs [#6959](https://github.com/grafana/tempo/pull/6959), [#6504](https://github.com/grafana/tempo/pull/6504), [#6667](https://github.com/grafana/tempo/pull/6667), [#6873](https://github.com/grafana/tempo/pull/6873)) ### Compactor removal The compactor component and the `v2` block encoding are removed. Compaction is now handled by the [backend scheduler and worker](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/), which track job progress centrally and automatically reschedule failed jobs. Remove all compactor-related configuration, alerts, and dashboard panels from your deployment. The following CLI commands are also removed because they were specific to the `v2` format: `list block`, `list index`, `view index`, `gen index`, and `gen bloom`. The compaction CLI flags drop their duplicate `compaction.` prefix. Update these flags in your configuration: - `compaction.compaction.block-retention` → `compaction.block-retention` - `compaction.compaction.max-objects-per-block` → `compaction.max-objects-per-block` - `compaction.compaction.max-block-bytes` → `compaction.max-block-bytes` - `compaction.compaction.compaction-window` → `compaction.compaction-window` [PRs [6273](https://github.com/grafana/tempo/pull/6273), [6369](https://github.com/grafana/tempo/pull/6369), [6909](https://github.com/grafana/tempo/pull/6909)] ### Target changes The `all` target is now 3.0-compatible and the `scalable-single-binary` target is removed. Refer to [Deployment modes](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/deployment-modes/) for details. [[PR 6283](https://github.com/grafana/tempo/pull/6283)] ### Block and WAL configuration centralization `block_builder` and `live_store` now always use `storage.trace.block` settings. Per-module block configuration fields are removed. If you configured block or WAL settings separately for these modules, move them to `storage.trace.block`. [[PR 6647](https://github.com/grafana/tempo/pull/6647)] ### Other removed components - The OpenCensus receiver is removed. Migrate to OTLP. [[PR 6523](https://github.com/grafana/tempo/pull/6523)] - The legacy `mem-ballast-size-mbs` CLI flag is removed. Use `GOMEMLIMIT` instead. [[PR 6403](https://github.com/grafana/tempo/pull/6403)] - `SpanMetricsSummary` is removed and querier code simplified. (PRs [#6496](https://github.com/grafana/tempo/pull/6496), [#6510](https://github.com/grafana/tempo/pull/6510)) ### Configuration changes - RetryInfo enabled by default: `distributor.retry_after_on_resource_exhausted` now defaults to `5s` (was `0`). OTLP clients receive a retry hint on `ResourceExhausted` errors. Set to `0` to disable cluster-wide, or set the per-tenant override `ingestion.retry_info_enabled: false` to disable for a single tenant. [[PR 7088](https://github.com/grafana/tempo/pull/7088)] - Read configuration consolidated: `query_frontend.search.query_ingesters_until` is removed in favor of `query_frontend.search.query_backend_after`. [[PR 6507](https://github.com/grafana/tempo/pull/6507)] - Legacy overrides disabled: Tempo refuses to start if legacy (flat, non-scoped) overrides are detected. Use `tempo-cli migrate overrides-config` to convert them. Set `enable_legacy_overrides: true` to opt back in temporarily. Refer to the [Migrate from Tempo 2.x to 3.0](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/migrate-to-3/) guide for details. [[PR 6741](https://github.com/grafana/tempo/pull/6741)] - Fail-on-high-lag enabled by default: `live_store.fail_on_high_lag` now defaults to `true` (was `false`). When a live-store's Kafka lag overlaps a query's time range, search and metrics requests return an error rather than silently incomplete results, trading availability for correctness. To restore the previous behavior, set `live_store.fail_on_high_lag: false`. Accordingly, `query_frontend.query_end_cutoff` now defaults to `30s` (was `0`), excluding the most recent 30 seconds from queries to avoid incomplete results; it must be less than `query_frontend.search.query_backend_after`. [[PR 7210](https://github.com/grafana/tempo/pull/7210)] ### Stricter filter policy validation Filter policies submitted through the [user-configurable overrides API](/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/user-configurable-overrides/) are now fully validated. Previously, invalid policies (for example, unsupported attribute scopes, malformed regular expressions, or unrecognized intrinsic values) were silently accepted. In Tempo 3.0, the API rejects them with a descriptive error. If you have existing filter policies, verify they use only `resource` and `span` scopes, valid `match_type` values (`strict` or `regex`), and recognized intrinsic values for `kind` and `status`. Refer to [Filtering](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/span-metrics/span-metrics-metrics-generator/#filtering) for the complete validation rules. [[PR 6407](https://github.com/grafana/tempo/pull/6407)] ### TraceQL array matching changes The TraceQL AST optimization changes the semantics of `!=` and `!~` operators when used with array attributes. `!=` now means `NOT IN` (was `CONTAINS NOT EQUAL`) and `!~` now means `MATCH NONE` (was `CONTAINS NON-MATCH`). Regex operands must be of type string or string array. Disable the optimization with the query hint `skip_optimization=true` if needed. [[PR 6353](https://github.com/grafana/tempo/pull/6353)] ### Tempo CLI timestamp format The `query search` command no longer accepts timestamps without timezone (for example, `2024-01-01T00:00:00`). Use RFC3339 format (for example, `2024-01-01T00:00:00Z`) or relative time (for example, `now-1h`). Refer to the [Tempo CLI documentation](/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/) for details. [[PR 6458](https://github.com/grafana/tempo/pull/6458)] ### Platform changes The 32-bit ARM binary archives are no longer published. Release artifacts continue to include amd64 and arm64 binaries. If you run Tempo on 32-bit ARM hardware, you need to build from source or migrate to arm64. [[PR 7106](https://github.com/grafana/tempo/pull/7106)] ### Go version upgrade Tempo 3.0 upgrades to Go 1.26.2. [[PR 6443](https://github.com/grafana/tempo/pull/6443)] ## Security fixes The following security issues are addressed. ### 3.0.3 - Updated Go to 1.26.5 to address [CVE-2026-39822](https://nvd.nist.gov/vuln/detail/CVE-2026-39822), [CVE-2026-27145](https://nvd.nist.gov/vuln/detail/CVE-2026-27145), [CVE-2026-42504](https://nvd.nist.gov/vuln/detail/CVE-2026-42504), [CVE-2026-42505](https://nvd.nist.gov/vuln/detail/CVE-2026-42505), and [CVE-2026-42507](https://nvd.nist.gov/vuln/detail/CVE-2026-42507). [[PR 7726](https://github.com/grafana/tempo/pull/7726)] - Updated `google.golang.org/grpc` to v1.82.1 to address [GHSA-hrxh-6v49-42gf](https://github.com/advisories/GHSA-hrxh-6v49-42gf). [[PR 7726](https://github.com/grafana/tempo/pull/7726)] - Updated `golang.org/x/net` to v0.56.0 to address [CVE-2026-46600](https://nvd.nist.gov/vuln/detail/CVE-2026-46600). [[PR 7726](https://github.com/grafana/tempo/pull/7726)] - Updated `golang.org/x/text` to v0.39.0 to address [CVE-2026-56852](https://nvd.nist.gov/vuln/detail/CVE-2026-56852). [[PR 7726](https://github.com/grafana/tempo/pull/7726)] - Updated `go.opentelemetry.io/otel` to v1.44.0 to address [CVE-2026-41178](https://nvd.nist.gov/vuln/detail/CVE-2026-41178). [[PR 7726](https://github.com/grafana/tempo/pull/7726)] ### 3.0.2 - Built Tempo with Go 1.26.3, which includes upstream security and bug fixes. [[PR 7423](https://github.com/grafana/tempo/pull/7423)] ### 3.0.0 - Fixed division by zero error in TraceQL expressions that could cause query failures. [[PR 6580](https://github.com/grafana/tempo/pull/6580)] - Fixed `intPow` function hanging for certain inputs, which could cause unbounded CPU consumption. [[PR 6581](https://github.com/grafana/tempo/pull/6581)] - Fixed integer overflow in query parameters by using `strconv.ParseUint` instead of `strconv.Atoi`/`strconv.ParseInt` for unsigned integer fields. [[PR 6612](https://github.com/grafana/tempo/pull/6612)] - Fixed allowlist header normalization when building the allowlist map. [[PR 6481](https://github.com/grafana/tempo/pull/6481)] ## Bug fixes For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). ### 3.0.3 - Fixed the version reported by `--version`, the build-info metric, and `/api/status/buildinfo`. The build version is now read from the top-level `VERSION` file instead of the most recently created git tag, which could belong to a different release. [[PR 7469](https://github.com/grafana/tempo/pull/7469)] - Fixed panic in livestore search when iterating blocks encounters an unexpected error. [[PR 7134](https://github.com/grafana/tempo/pull/7134)] - Fixed panic in old tag-style search when querying integer dedicated columns on vParquet5 blocks. [[PR 7135](https://github.com/grafana/tempo/pull/7135)] - Fixed incorrect search results for some queries on blob columns. [[PR 6815](https://github.com/grafana/tempo/pull/6815)] - Fixed active-series counter underflow in local series limiter when overflow series are deleted. [[PR 6568](https://github.com/grafana/tempo/pull/6568)] - Fixed per-label limiter and sanitizer being incorrectly applied to `target_info` and `host_info` metrics. [[PR 6660](https://github.com/grafana/tempo/pull/6660)] - Fixed `target_info` being skipped when resource attributes have empty values. [[PR 6774](https://github.com/grafana/tempo/pull/6774)] - Fixed limiter leak and permanent overflow when old series are replaced. [[PR 6653](https://github.com/grafana/tempo/pull/6653)] - Fixed dedicated column filtering issue. [[PR 6586](https://github.com/grafana/tempo/pull/6586)] - Fixed buffer-reuse bug where event attributes in dedicated columns could be persisted on additional spans and events. [[PR 6914](https://github.com/grafana/tempo/pull/6914)] - Fixed `span_name_sanitization` overrides not reloading during runtime. [[PR 6435](https://github.com/grafana/tempo/pull/6435)] - Returned 400 instead of 500 when `query_range` or `query_instant` requests have invalid start/end parameters. [[PR 6694](https://github.com/grafana/tempo/pull/6694)] --- ### Sources Tempo Release Notes V3 1 (`docs/sources/tempo/release-notes/v3-1.md`) --- title: Version 3.1 release notes menuTitle: V3.1 description: Release notes for Grafana Tempo 3.1 weight: 650 --- # Version 3.1 release notes The Tempo team is pleased to announce the release of Grafana Tempo 3.1. This release gives you: - [Kafka ingestion improvements](#kafka-ingestion-improvements): Connect to authenticated, TLS-encrypted Kafka, fetch in-rack to cut cross-zone transfer cost, and use `gzip` for Azure Event Hubs, all contributed by the Tempo community. - [Redaction by query](#redaction-by-query): Redact traces that match a TraceQL query instead of enumerating IDs, so a sensitive-data incident doesn't require hunting down every affected trace. - [TraceQL metrics extrapolation and arithmetic](#traceql-metrics-extrapolation-and-arithmetic): Extrapolate counts from head-sampled traces so rates reflect actual traffic, and combine aggregations with arithmetic to calculate ratios like error rate in a single query. - [Metrics-generator and service graph improvements](#metrics-generator-and-service-graph-improvements): Keep more service-map edges when you sample, recognize `db.system.name`, and cut per-span metrics-generator cost. - [Span pruning for trace-by-id v2](#span-pruning-for-trace-by-id-v2): Trim response size by pruning uninteresting spans. An experimental configuration option can turn pruning on by default. - [Traces diff and trace summary (experimental)](#traces-diff-and-trace-summary-experimental): Compare two traces to see what changed between a baseline and a candidate run, available through the API, `tempo-cli`, and the MCP server. - [vParquet5 as the default block format](#default-block-format-is-now-vparquet5): New blocks are written as vParquet5; existing vParquet4 blocks remain readable, with no migration. These release notes highlight the most important features and bug fixes. For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). ## Kafka ingestion improvements In microservices mode, Tempo uses Kafka as a durable write-ahead log: traces flow through Kafka so distributors and consumers scale independently, and writes survive a consumer restart. Until 3.1, the Tempo Kafka client only supported SASL `PLAIN` authentication; it couldn't encrypt in transit, prefer a local rack, or choose a compression codec. If your cluster required TLS or a stronger SASL mechanism, you couldn't use this path at all. Tempo 3.1 adds SASL `SCRAM-SHA-256`, `SCRAM-SHA-512`, `OAUTHBEARER`, and `AWS_MSK_IAM` authentication alongside the existing `PLAIN` support, plus TLS and mutual-TLS, so you can connect to encrypted Kafka clusters, including Amazon MSK. A `client_rack` option enables rack-aware fetching (KIP-392). Consumers fetch from a replica in the same availability zone instead of always crossing zones to the partition leader, which cuts cross-zone data transfer cost on high-throughput pipelines. An `ingest.kafka.producer_compression` option overrides the producer's compression codec, which Azure Event Hubs requires because it only supports `gzip`. (PRs [#7586](https://github.com/grafana/tempo/pull/7586), [#7594](https://github.com/grafana/tempo/pull/7594), [#7691](https://github.com/grafana/tempo/pull/7691)) Community contributors independently unblocked each of those gaps: SASL and TLS from [@heytrav](https://github.com/heytrav), rack-aware fetching from [@AvivGuiser](https://github.com/AvivGuiser), and gzip compression for Azure Event Hubs from [@fleighton](https://github.com/fleighton). Thanks to all three for this work. Refer to [Configure authentication and TLS](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/configure-kafka/#configure-authentication-and-tls) for SASL and TLS, and [Ingest](/docs/tempo/<TEMPO_VERSION>/configuration/#ingest) for `client_rack` and `producer_compression`. ## Redaction by query Tempo 3.0 introduced [trace redaction](/docs/tempo/<TEMPO_VERSION>/release-notes/v3-0/#trace-redaction), but you had to supply an explicit list of trace IDs. Tempo 3.1 lets you submit a redaction job with a TraceQL query instead, so you can match traces by attributes rather than enumerating IDs. [[PR 7663](https://github.com/grafana/tempo/pull/7663)] You can also bound a redaction to a time window with `--start` and `--end`, so a large tenant can be redacted in slices. [[PR 7702](https://github.com/grafana/tempo/pull/7702)] {{< admonition type="caution" >}} Don't submit a windowed redaction until every scheduler and worker is on Tempo 3.1. An older worker ignores the window and redacts every query match in the block, with no error. {{< /admonition >}} This release also includes several [backend scheduler redaction reliability](#backend-scheduler-redaction-reliability) fixes. Refer to [Redact traces](/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#redact-traces) for query syntax, constraints, and windowed redaction. ## TraceQL metrics extrapolation and arithmetic If you use head-based sampling to control costs, metrics derived from those traces reflect only the sampled fraction, not actual traffic. At 10% sampling, `{ } | rate()` reports approximately 50 spans/s when the unsampled rate is 500 spans/s. Tempo 3.1 lets TraceQL metrics extrapolate counts from the sampling probability recorded in each span, so rates and counts reflect actual traffic without a separate metrics pipeline. Opt in per query with the experimental `with(extrapolate=true)` hint. Extrapolation requires vParquet4 or later blocks; queries against vParquet3 blocks ignore the hint. [[PR 7452](https://github.com/grafana/tempo/pull/7452)] TraceQL metrics also adds arithmetic operators (`+`, `-`, `*`, `/`) so you can combine aggregations in a single query instead of stitching results client-side. (PRs [#6866](https://github.com/grafana/tempo/pull/6866), [#7199](https://github.com/grafana/tempo/pull/7199), [#7409](https://github.com/grafana/tempo/pull/7409)) This query estimates the fraction of spans with error status, adjusted for sampling probability: ```traceql ({status=error} | rate()) / ({} | rate()) with(extrapolate=true) ``` For supported functions, requirements, and syntax, refer to [TraceQL metrics functions](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/functions/), including the [arithmetic operators](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/functions/#arithmetic-expressions) and the [extrapolation hint](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/functions/#extrapolation-from-ingest-time-sampling-withextrapolatetrue-experimental). ## Metrics-generator and service graph improvements If you use the metrics-generator, you get RED metrics and a service map from traces you already ingest. The generator processes every span at ingest, so CPU, restart time, and dropped edges all affect the series you alert on. Service graphs get a new opt-in `traces_service_graph_connection_info` presence gauge, powered by new `service-graphs-*` subprocessors, for detecting service topology under heavy sampling, visualized in a new `tempo-service-graph.json` dashboard. [[PR 7202](https://github.com/grafana/tempo/pull/7202)] The presence gauge keeps observed connections visible when sampling makes request-rate metrics sparse. It uses the same edge-detection rules as the existing service graph metrics. Refer to [Connection information metric](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/#connection-information-metric) to enable the subprocessor and query the gauge. Additional service graph improvements: - Service graphs now recognize the `db.system.name` attribute (the OpenTelemetry v1.30.0 rename of `db.system`) for database node detection and virtual node naming. Both attributes remain supported, so database nodes still appear after you upgrade to OpenTelemetry SDKs that emit the renamed attribute. [[PR 7697](https://github.com/grafana/tempo/pull/7697), [documentation](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/service_graphs/#database-name-attributes)] - Expired service-graph edges are now labeled with the unmatched span kind, so you can tell which side of a connection was missing instead of an edge disappearing with no explanation. [[PR 7709](https://github.com/grafana/tempo/pull/7709), [documentation](/docs/tempo/<TEMPO_VERSION>/troubleshooting/metrics-generator/#expired-edges)] The span-metrics and service-graphs processors were both reworked to build series labels through the metrics-generator registry's pooled, borrowed-label path, cutting per-span and per-edge CPU and allocations. At high ingest rates, that per-span cost is what makes the generator expensive to run. Metric names, labels, and values are unchanged. (PRs [#7584](https://github.com/grafana/tempo/pull/7584), [#7587](https://github.com/grafana/tempo/pull/7587)) As part of this work, native histograms no longer attach an exemplar with an empty trace ID for spans that have no trace ID, matching classic histogram behavior. A new `skip_stale_backlog_on_startup` option seeks Kafka partitions forward to the ingestion-slack horizon on startup instead of replaying backlog the slack would discard anyway. [[PR 7611](https://github.com/grafana/tempo/pull/7611), [documentation](/docs/tempo/<TEMPO_VERSION>/configuration/#metrics-generator)] A restart or scale-up no longer spends time processing spans that would never become metrics. Stale per-partition ingest lag metrics are now pruned when a partition moves between consumers, fixing ever-growing `tempo_ingest_group_partition_lag` series after a partition handoff. [[PR 7665](https://github.com/grafana/tempo/pull/7665)] That series is how you tell whether the generator is keeping up; after a handoff it used to keep growing, which made health dashboards hard to trust. ## Span pruning for trace-by-id v2 The trace-by-id v2 endpoint can now prune uninteresting spans from the response to reduce payload size. [[PR 7566](https://github.com/grafana/tempo/pull/7566)] With no Tempo configuration change, pruning is per request: only a v2 request that sets `span_pruning=true` gets a pruned response. [[PR 7912](https://github.com/grafana/tempo/pull/7912)] Other requests, including Grafana, return the full trace. Traces already pruned on the write path are detected and aren't pruned again. Grafana doesn't expose a span pruning control, so you can't opt in or out from the UI. Use a client that can set the query parameter, such as a direct call to the v2 API. ### Configure the default The experimental `span_pruning_enabled_by_default` option is off unless you set it in the Tempo configuration. It's not active on an unchanged configuration file. When you set it to `true`, then traces are pruned automatically and you don't need to specify `span_pruning` in a request. [[PR 7628](https://github.com/grafana/tempo/pull/7628)] Use this when you want pruning on every request that doesn't set the parameter. A per-tenant `span_pruning_enabled` override lets individual tenants opt in or out of that default. [[PR 7693](https://github.com/grafana/tempo/pull/7693)] An explicit `span_pruning` value in the request always takes precedence over both settings. If you turn the default on, Grafana users receive pruned traces and have no UI control to turn pruning off. To fetch an unpruned trace, call the v2 API with `span_pruning=false` or turn the option off. To help evaluate the effect of pruning on query results, the TraceQL engine has a new span-watcher framework for collecting extra query metrics on demand. Enabled through the experimental per-tenant `span_pruning_awareness` override, it reports whether matched spans include span-pruning summary spans, for both search and metrics queries. [[PR 7532](https://github.com/grafana/tempo/pull/7532)] Refer to [Query V2](/docs/tempo/<TEMPO_VERSION>/api_docs/#query-v2) for the request parameters and [Query-frontend](/docs/tempo/<TEMPO_VERSION>/configuration/#query-frontend) for the configuration options. ## Traces diff and trace summary (experimental) {{< admonition type="note" >}} Traces diff and trace summary are [experimental features](https://grafana.com/docs/release-life-cycle/). Engineering and on-call support is not available. Experimental features might change or be removed in future releases and are not recommended for use in production environments. {{< /admonition >}} Tempo 3.1 adds an experimental way to compare two traces and see what changed between them, useful for debugging regressions between a baseline and a candidate run. [[PR 7539](https://github.com/grafana/tempo/pull/7539), [PR 7523](https://github.com/grafana/tempo/pull/7523)] The HTTP API defaults to `trace-patch-v0` (the full span-level patch). A `trace-summary-v0-composed` format returns a compact summary alongside the patch, capping the patch at 64 KiB; larger patches report that the patch was omitted rather than truncating it silently. [[PR 7593](https://github.com/grafana/tempo/pull/7593)] Comparisons use tolerance-based matching for span durations (20% relative, 1ms floor) and an allow-listed set of numeric attributes (5% relative), so timing noise between runs doesn't produce false positives; the output's duration field is now named `duration_nanos` and reports raw nanosecond values. [[PR 7544](https://github.com/grafana/tempo/pull/7544)] The combined size of both traces is checked against the `max_bytes_per_trace` per-tenant limit to protect the query frontend from oversized requests, returning `429` when exceeded. [[PR 7564](https://github.com/grafana/tempo/pull/7564)] `tempo-cli` also gets traces diff support for local work: `tempo-cli experimental traces-diff` compares two local trace JSON files and emits `trace-patch-v0` output, and an experimental `trace-summary-v0-native` format gives a compact overview of latency, summed span duration, errors, structural changes, and affected services. [[PR 7468](https://github.com/grafana/tempo/pull/7468), [PR 7510](https://github.com/grafana/tempo/pull/7510)] The MCP server also exposes a `traces-diff` tool for complete-trace comparisons, with a compact composed summary by default and patches up to 64 KiB. [[PR 7785](https://github.com/grafana/tempo/pull/7785)] Refer to [Traces diff](/docs/tempo/<TEMPO_VERSION>/api_docs/#traces-diff) for the HTTP API, [Compare traces](/docs/tempo/<TEMPO_VERSION>/api_docs/mcp-server/#compare-traces) for the MCP tool, and [Experimental traces diff](/docs/tempo/<TEMPO_VERSION>/operations/tempo_cli/#experimental-traces-diff) for the CLI. ## Features and enhancements The most important remaining features and enhancements in Tempo 3.1 are highlighted below. ### TraceQL correctness - Instant metrics queries now correctly reuse the per-block job results cache. [[PR 7602](https://github.com/grafana/tempo/pull/7602)] - The vParquet5 faster fetch layer no longer returns incorrect results for trace intrinsics such as `trace:rootService` and `span:childCount`, array operations, or metrics queries on event and link intrinsics. (PRs [#7508](https://github.com/grafana/tempo/pull/7508), [#7533](https://github.com/grafana/tempo/pull/7533)) - `max_metrics_duration` is now enforced against the user-provided range rather than the post-alignment range. [[PR 7170](https://github.com/grafana/tempo/pull/7170)] ### Performance improvements - The faster span-only fetch path for metrics queries, [introduced as experimental in Tempo 3.0](/docs/tempo/<TEMPO_VERSION>/release-notes/v3-0/#additional-traceql-improvements), is now enabled by default. Disable it per-tenant with the `metrics_spanonly_fetch: false` override. You can also disable it per-query with `with(spanonly_fetch=false)`, but this hint requires [`unsafe_query_hints`](/docs/tempo/<TEMPO_VERSION>/configuration/#overrides) to be enabled for the tenant. [[PR 7179](https://github.com/grafana/tempo/pull/7179), [documentation](/docs/tempo/<TEMPO_VERSION>/metrics-from-traces/metrics-queries/#faster-read-path)] - Storage and cache performance: tag-value scans now stop as soon as the response limit is reached instead of scanning every row group; `ByteInPredicate`/`ByteNotInPredicate` use a map lookup instead of a linear scan for large value sets; blocklist updates run in O(N+M) instead of O(N·M); and cache entries for retention-deleted blocks are evicted sooner. (PRs [#7696](https://github.com/grafana/tempo/pull/7696), [#7535](https://github.com/grafana/tempo/pull/7535), [#7140](https://github.com/grafana/tempo/pull/7140), [#7204](https://github.com/grafana/tempo/pull/7204)) - The Redis cache client supports a configurable `max_item_size`, and the Memcached client adds `connect_timeout` and `min_idle_conns_headroom_percentage` options. (PRs [#7311](https://github.com/grafana/tempo/pull/7311), [#7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo/<TEMPO_VERSION>/configuration/#cache)) ### Backend scheduler redaction reliability This release closes several correctness and scalability gaps in the backend scheduler's redaction pipeline: - A completed redaction batch now enters a short quiescence period before removal, so a block compacted right after the last job finishes still gets covered by a rescan. [[PR 7695](https://github.com/grafana/tempo/pull/7695)] - Redaction requests now source the tenant exclusively from the authenticated request context, closing a cross-tenant escalation path. [[PR 7153](https://github.com/grafana/tempo/pull/7153)] - A dry-run redaction no longer disables the tenant's compaction and retention or arms a rescan; those now only happen for a redaction in apply mode. [[PR 7700](https://github.com/grafana/tempo/pull/7700)] - A redaction job dropped at assignment no longer leaks an internal in-flight counter, which previously could block future submissions. [[PR 7703](https://github.com/grafana/tempo/pull/7703)] - Retention now gates on the redaction batch barrier instead of only in-flight jobs, and a new metric plus warning log surface when a job's target block is missing from the live blocklist. [[PR 7358](https://github.com/grafana/tempo/pull/7358)] - A new `tempo_backend_scheduler_redaction_traces_found_total` metric reports how many traces a redaction job matched, split by apply and dry-run mode. A new Redaction row on the Backend Work dashboard visualizes these. [[PR 7699](https://github.com/grafana/tempo/pull/7699)] - A new `tempo_backend_scheduler_jobs_pending` metric reports queue depth. [[PR 7772](https://github.com/grafana/tempo/pull/7772), [documentation](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/#key-metrics)] - Job and index lookups no longer scan proportionally to tenant or shard count under load. (PRs [#7141](https://github.com/grafana/tempo/pull/7141), [#6992](https://github.com/grafana/tempo/pull/6992)) For the full redaction job lifecycle and metrics, refer to [Compaction](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/compaction/). ### Query and TraceQL - The trace-by-id v2 endpoint adds filtering support with `q` (TraceQL filter) and `keep_hierarchy` query parameters. `match_depth` and `ancestor_depth` control how many descendant and ancestor hops to keep. [[PR 7483](https://github.com/grafana/tempo/pull/7483), [PR 7708](https://github.com/grafana/tempo/pull/7708), [documentation](/docs/tempo/<TEMPO_VERSION>/api_docs/#query-v2)] - New read-path observability: cache hit/miss counters, query-shape fields mirrored as span attributes, a `tempo_querier_backend_processing_duration_seconds` histogram, and stats metrics on additional querier methods. Query-shape span attribute names changed to `snake_case` (for example, `queryType` is now `query_type`). (PRs [#7504](https://github.com/grafana/tempo/pull/7504), [#7605](https://github.com/grafana/tempo/pull/7605), [#7525](https://github.com/grafana/tempo/pull/7525), [#7571](https://github.com/grafana/tempo/pull/7571), [#7568](https://github.com/grafana/tempo/pull/7568), [documentation](/docs/tempo/<TEMPO_VERSION>/reference-tempo-architecture/components/querier/#key-metrics)) - An opt-in `engine_bytes_tracking` override reports span and attribute size through the TraceQL engine as `tempo_query_frontend_engine_bytes_total`, so you can identify which queries scan the most data and plan capacity accordingly. [[PR 7689](https://github.com/grafana/tempo/pull/7689), [documentation](/docs/tempo/<TEMPO_VERSION>/configuration/#overrides)] - The Tempo MCP server adds a `docs-config` tool and refreshed TraceQL and metrics documentation resources. (PRs [#7387](https://github.com/grafana/tempo/pull/7387), [#7408](https://github.com/grafana/tempo/pull/7408), [documentation](/docs/tempo/<TEMPO_VERSION>/api_docs/mcp-server/#available-tools)) ### Configuration and build - New block-size histograms `tempo_block_builder_flush_size_bytes` and `tempodb_compaction_output_block_size_bytes` record the size of flushed and compacted blocks, helping you spot oversized blocks and tune compaction settings. [[PR 7773](https://github.com/grafana/tempo/pull/7773)] - A new `TempoDistributorKafkaProduceFailing` alert fires when the distributor can't produce records to Kafka. [[PR 7148](https://github.com/grafana/tempo/pull/7148)] - The distributor now auto-forgets unhealthy instances from its ring after twice the heartbeat timeout (10 minutes by default), removing the need to manually click "Forget" after a non-graceful Pod termination. [[PR 7098](https://github.com/grafana/tempo/pull/7098), [documentation](/docs/tempo/<TEMPO_VERSION>/operations/manage-advanced-systems/consistent_hash_ring/#distributor)] - Container image signing coverage is now complete: all four published images (`tempo`, `tempo-vulture`, `tempo-query`, `tempo-cli`) are signed with cosign and attested with SLSA build provenance, so you can verify image authenticity before deploying. (PRs [#7601](https://github.com/grafana/tempo/pull/7601), [#7543](https://github.com/grafana/tempo/pull/7543), [#7493](https://github.com/grafana/tempo/pull/7493), [documentation](/docs/tempo/<TEMPO_VERSION>/operations/verify-container-images/)) - KEDA-based autoscaling is now available for live-store and for the metrics-generator in Jsonnet. Set `autoscaling_prometheus_tenant` when the Prometheus source is a multi-tenant system such as Grafana Mimir. (PRs [#7142](https://github.com/grafana/tempo/pull/7142), [#7362](https://github.com/grafana/tempo/pull/7362), [#7099](https://github.com/grafana/tempo/pull/7099), [#7376](https://github.com/grafana/tempo/pull/7376), [documentation](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/deploy/kubernetes/tanka/#optional-enable-keda-autoscaling)) - New Grafana dashboards: `tempo-service-graph.json` visualizes service topology, and a new Livestore dashboard covers read latency, ingest lag, backpressure, and KEDA autoscaling. (PRs [#7207](https://github.com/grafana/tempo/pull/7207), [#7287](https://github.com/grafana/tempo/pull/7287), [#7710](https://github.com/grafana/tempo/pull/7710), [documentation](/docs/tempo/<TEMPO_VERSION>/operations/monitor/#dashboards)) - Darwin release builds are re-enabled. [[PR 7407](https://github.com/grafana/tempo/pull/7407)] ## Upgrade considerations When [upgrading](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/upgrade/) to Tempo 3.1, be aware of these considerations and breaking changes. Refer to [Upgrade to Tempo 3.1](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/upgrade/#upgrade-to-tempo-31) for full migration steps. | Condition | Action | | --- | --- | | You write vParquet3 blocks | Change block version before upgrading. Refer to [vParquet3 deprecation enforced](#vparquet3-deprecation-enforced). | | You use Redis cache | Review renamed YAML keys and removed Sentinel support. Refer to [Redis client rewrite](#redis-client-rewrite). | | You depend on a fixed Trace-by-ID shard count | Set `blocks_per_shard: 0` to fall back to `query_shards`. Refer to [Default value changes](#default-value-changes). | | You depend on gRPC streaming responses larger than 1 MiB | Set `max_grpc_streaming_packet_size` explicitly. Refer to [Default value changes](#default-value-changes). | ### Default block format is now vParquet5 Tempo 3.1 writes new blocks in vParquet5. Existing vParquet4 blocks are still readable, and no data migration is required. To keep writing vParquet4, set the block version explicitly: ```yaml storage: trace: block: version: vParquet4 ``` Refer to [Default block format is now vParquet5](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/upgrade/#default-block-format-is-now-vparquet5) for upgrade steps and [Apache Parquet block format](/docs/tempo/<TEMPO_VERSION>/configuration/parquet/) for format details. [[PR 7775](https://github.com/grafana/tempo/pull/7775)] ### vParquet3 deprecation enforced vParquet3 was deprecated in Tempo 2.10 and 3.0, but nothing in code prevented its use. Tempo 3.1 enforces the deprecation: Tempo refuses to start if configured to write vParquet3 blocks, and existing vParquet3 blocks are no longer compacted. Reads of existing vParquet3 blocks are unaffected. If your storage configuration specifies `vParquet3`, change the block version to `vParquet5` (default) or `vParquet4` before upgrading. [[PR 7858](https://github.com/grafana/tempo/pull/7858)] Refer to [vParquet3 deprecation enforced](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/upgrade/#vparquet3-deprecation-enforced) for upgrade steps and [Apache Parquet block format](/docs/tempo/<TEMPO_VERSION>/configuration/parquet/) for format details. ### Redis client rewrite The experimental Redis cache client has been completely rewritten: - Redis Cluster is now the default routing mode, - Redis Sentinel support is removed, - several YAML keys are renamed, and - the TLS block is replaced with a dskit-style block that fails closed on invalid configuration. If you don't use the Redis cache, no action is needed. Refer to [Redis cache configuration changes](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/upgrade/#redis-cache-configuration-changes). [[PR 7337](https://github.com/grafana/tempo/pull/7337)] ### Default value changes - Query sharding: Trace-by-ID lookups now shard by block count through a new `blocks_per_shard` option, which defaults to `30` and takes precedence over the older `query_shards` setting. To keep the previous fixed-shard-count behavior, set `blocks_per_shard: 0` to fall back to `query_shards`. [[PR 7105](https://github.com/grafana/tempo/pull/7105), [documentation](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/upgrade/#trace-by-id-query-sharding-now-scales-with-block-count)] - Metrics query validation: A metrics query request whose start and end timestamps are identical is now properly rejected instead of silently returning an empty or incorrect result. [[PR 7602](https://github.com/grafana/tempo/pull/7602)] - gRPC streaming packet size: The query frontend's default `max_grpc_streaming_packet_size` drops from 2 MiB to 1 MiB. If you depend on larger streamed gRPC responses, set `max_grpc_streaming_packet_size` explicitly to restore the previous value. [[PR 7615](https://github.com/grafana/tempo/pull/7615), [documentation](/docs/tempo/<TEMPO_VERSION>/troubleshooting/querying/response-too-large/)] - Memcached idle connections: Idle Memcached connections are no longer closed after 2 minutes by default, and the default `max_idle_conns` is raised from 16 to 100. This avoids a burst of new connection dials, and the tail-latency spike that comes with it, at the start of every read burst. To restore idle-connection reaping, set `min_idle_conns_headroom_percentage` to `0`. [[PR 7671](https://github.com/grafana/tempo/pull/7671), [documentation](/docs/tempo/<TEMPO_VERSION>/set-up-for-tracing/setup-tempo/upgrade/#memcached-cache-connection-defaults)] ### Build and platform changes - Tempo 3.1 upgrades to Go 1.27.1. Refer to [Security fixes](#security-fixes) for the CVEs this addresses. ## Security fixes - Updated Go to 1.27.1 and bumped vendored `golang.org/x/net` and `golang.org/x/text` to address [CVE-2026-39822](https://nvd.nist.gov/vuln/detail/CVE-2026-39822), [CVE-2026-42504](https://nvd.nist.gov/vuln/detail/CVE-2026-42504), [CVE-2026-27145](https://nvd.nist.gov/vuln/detail/CVE-2026-27145), [CVE-2026-42505](https://nvd.nist.gov/vuln/detail/CVE-2026-42505), [CVE-2026-42507](https://nvd.nist.gov/vuln/detail/CVE-2026-42507), [CVE-2026-46600](https://nvd.nist.gov/vuln/detail/CVE-2026-46600), and [CVE-2026-56852](https://nvd.nist.gov/vuln/detail/CVE-2026-56852). [[PR 7641](https://github.com/grafana/tempo/pull/7641)] - Fixed a cross-tenant escalation path in redaction job submission. The tenant is now sourced exclusively from the authenticated request context instead of a client-supplied field. [[GHSA-jw7f-3cgp-hgrr](https://github.com/advisories/GHSA-jw7f-3cgp-hgrr), [PR 7153](https://github.com/grafana/tempo/pull/7153)] ## Bug fixes For a complete list, refer to the [Tempo CHANGELOG](https://github.com/grafana/tempo/releases). - Prevent metrics-generator crashes when span-name processing prunes DRAIN clusters. [[PR 7787](https://github.com/grafana/tempo/pull/7787)] - Fix classic histogram `_sum` series (for example, `traces_spanmetrics_latency_sum`) under-reporting via `increase()`/`rate()` for series with high churn. The `_sum` series was missing the zero-sample seed that `_count` and `_bucket` already had. [[PR 7811](https://github.com/grafana/tempo/pull/7811)] - Write the live-store per-block query-range response cache atomically, and log and ignore cache read errors instead of failing the query. [[PR 7155](https://github.com/grafana/tempo/pull/7155)] - Fix a data race in the query frontend when dispatching request batches to queriers with `max_batch_size > 1`. [[PR 7664](https://github.com/grafana/tempo/pull/7664)] - Reject TraceQL queries larger than `max_query_expression_size_bytes` before parsing them. [[PR 7245](https://github.com/grafana/tempo/pull/7245), [documentation](/docs/tempo/<TEMPO_VERSION>/configuration/#cap-the-maximum-query-length)] - Fix `=nil` queries missing some values in TraceQL. [[PR 7345](https://github.com/grafana/tempo/pull/7345)] - Fix tag-value autocomplete ignoring query filters when the incomplete matcher targets an intrinsic (for example, `{ resource.service.name = "foo" && name = }`). [[PR 7660](https://github.com/grafana/tempo/pull/7660)] - Propagate the `limit` and `maxStaleValues` tag-value query parameters from the query frontend down to queriers and live-store, bounding per-block scans instead of applying the limit only at the frontend. [[PR 7609](https://github.com/grafana/tempo/pull/7609)] - Fix a panic in TraceQL metrics-math queries (for example, `(A) / (B)`) when a sub-query's `by()` clause uses the maximum number of group-by attributes. [[PR 7831](https://github.com/grafana/tempo/pull/7831)] - Cache empty tag-value results per block so a block with no matching values isn't re-scanned on every query. [[PR 7617](https://github.com/grafana/tempo/pull/7617)] - Stop silently dropping a block's tag values when its disk-cache entry is unreadable; the block is re-searched instead of skipped. [[PR 7610](https://github.com/grafana/tempo/pull/7610)] - Fix a rare cache collision between instant and range metrics queries. [[PR 7290](https://github.com/grafana/tempo/pull/7290)] - Fix double path prefix in the compactor and `DeleteVersioned` on Azure and S3 backends. [[PR 7271](https://github.com/grafana/tempo/pull/7271)] - Preserve the no-compact flag when copying vParquet5 blocks so a freshly flushed block isn't compacted or polled before it's complete. [[PR 7786](https://github.com/grafana/tempo/pull/7786)] - Bound per-trace slice preallocation during distributor rebatching. [[PR 7288](https://github.com/grafana/tempo/pull/7288)] - Return a retryable status for transient errors writing to Kafka. [[PR 7506](https://github.com/grafana/tempo/pull/7506)] - Guard the `TempoDistributorKafkaProduceFailing` alert on a non-zero produce rate so it doesn't page at `+Inf%` when no records have been produced yet. [[PR 7715](https://github.com/grafana/tempo/pull/7715)] - Fix per-tenant override for `retry_info_enabled` silently overriding the cluster default when unset. [[PR 7662](https://github.com/grafana/tempo/pull/7662)] - Fix the version reported by `--version`, the build-info metric, and `/api/status/buildinfo`; it's now read from a VERSION file instead of the most recently created git tag, which could belong to a different release. [[PR 7469](https://github.com/grafana/tempo/pull/7469)] - Fix the packaged configuration for deb/rpm installs. Also adds CI validation and an errors-only flag for `-config.verify`. [[PR 7830](https://github.com/grafana/tempo/pull/7830)] --- ### Sources Tempo Shared Alloy Remote Write Tempo (`docs/sources/tempo/shared/alloy-remote-write-tempo.md`) --- headless: true description: Set up Alloy to remote-write traces to Tempo. labels: products: - enterprise - oss - alloy --- [//]: # "This file describes how to configure Alloy to remote-write to Tempo." [//]: # "This shared file is included in these locations:" [//]: # "/tempo/docs/sources/tempo/set-up-for-tracing/validate/set-up-test-app.md" [//]: # "/tempo/docs/sources/tempo/set-up-for-tracing/instrument-send/set-up-collector/grafana-alloy.md" [//]: # "This file is used in the following versions:" [//]: # "next" [//]: # "latest" [//]: # "If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included." [//]: # "Any links should be fully qualified and not relative." This section uses a [Grafana Alloy Helm chart](/docs/alloy/<ALLOY_VERSION>/set-up/install/kubernetes/) deployment to send traces to Tempo. To do this, you need to create a configuration that can be used by Alloy to receive and export traces in OTLP `protobuf` format. 1. Create a new `values.yaml` file which we'll use as part of the Alloy install. 1. Edit the `values.yaml` file and add the following configuration to it: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Update the distributor endpoint to match your deployment. The service name and namespace depend on how you deployed Tempo: - Tanka: `distributor.<NAMESPACE>.svc.cluster.local:4317` - Helm: `<RELEASE-NAME>-distributor.<NAMESPACE>.svc.cluster.local:4317` 1. Deploy Alloy using Helm: ```bash helm install -f values.yaml grafana-alloy grafana/alloy ``` If you deploy Alloy into a specific namespace, create the namespace first and specify it to Helm by appending `--namespace=<grafana-alloy-namespace>` to the end of the command. --- ### Sources Tempo Shared Azure Metrics Generator (`docs/sources/tempo/shared/azure-metrics-generator.md`) --- headless: true description: Azure metrics generator configuration for Helm charts. labels: products: - enterprise - oss --- [//]: # "This file documents the Azure metrics generator configuration for Tempo when using Helm charts." [//]: # "This shared file is included in these locations:" [//]: # "/tempo/docs/sources/tempo/configuration/hosted-storage/azure.md" [//]: # "/tempo/docs/sources/tempo/metrics-from-traces/metrics-queries/configure-traceql-metrics.md" [//]: # "/helm-charts/tempo-distributed/get-started-helm-charts/_index.md" [//]: # [//]: # "If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included." [//]: # "Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/." The metrics-generator consumes trace data from Kafka and writes derived metrics using the Prometheus remote write protocol. If the metrics-generator needs access to Azure blob storage, list it in the `env var` expansion configuration so the `STORAGE_ACCOUNT_ACCESS_KEY` has the secret value. You can use this configuration example with Helm charts, like `tempo-distributed`. Replace any values in all caps with the values for your Helm deployment. ```yaml generator: extraArgs: - "-config.expand-env=true" extraEnv: - name: <STORAGE_ACCOUNT_ACCESS_KEY> valueFrom: secretKeyRef: name: <TEMPO-TRACES-STG-KEY> key: <TEMPO-TRACES-KEY> ``` --- ### Sources Tempo Shared Best Practices Traces (`docs/sources/tempo/shared/best-practices-traces.md`) --- headless: true description: Shared file for tracing best practices. labels: products: - enterprise - oss --- [//]: # 'This file documents the best practices for tracing for Tempo.' [//]: # 'This shared file is included in these locations:' [//]: # '/grafana/docs/sources/datasources/tempo/tracing-best-practices.md' [//]: # '/website/docs/grafana-cloud/send-data/traces/tracing-best-practices.md' [//]: # '/tempo/docs/sources/tempo/getting-started/best-practices.md' [//]: # [//]: # 'If you make changes to this file, verify that the meaning and content are not changed in any place where the file is included.' [//]: # 'Any links should be fully qualified and not relative: /docs/grafana/ instead of ../grafana/.' ## Span and resource attributes [Traces](https://grafana.com/docs/tempo/<TEMPO_VERSION>/introduction/) are built from spans, which denote units of work such as a call to, or from, an upstream service. Spans are constructed primarily of span and resource attributes. Spans also have a hierarchy, where parent spans can have children or siblings. In the screenshot below, the left side of the screen (1) shows the list of results for the query. The right side (2) lists each span that makes up the selected trace. A **span attribute** is a key/value pair that provides context for its span. For example, if the span deals with calling another service via HTTP, an attribute could include the HTTP URL (maybe as the span attribute key `http.url`) and the HTTP status code returned (as the span attribute `http.status_code`). Span attributes can consist of varying, non-null types. Unlike a span attribute, a **resource attribute** is a key/value pair that describes the context of how the span was collected. Generally, these attributes describe the process that created the span. This could be a set of resource attributes concerning a Kubernetes cluster, in which case you may see resource attributes, for example: `k8s.namespace`, `k8s.container_name`, and `k8s.cluster`. These can also include information on the libraries that were used to instrument the spans for a trace, or any other infrastructure information. For more information, read the [Attribute and Resource](https://opentelemetry.io/docs/specs/otel/overview/) sections in the OpenTelemetry specification. ### Naming conventions for span and resource attributes The OpenTelemetry project defines a number of semantic conventions for attributes, which can help you to determine which attributes are most important to include in your spans. These conventions provide a common vocabulary for describing different types of entities, which can help to ensure that your data is consistent and meaningful. When naming attributes, use consistent, nested namespaces to ensures that attribute keys are obvious to anyone observing the spans of a trace and that common attributes can be shared by spans. Using our example from above, the `http` prefix of the attribute is the namespace, and `url` and `status_code` are keys within that namespace. Attributes can also be nested, for example `http.url.protocol` might be `HTTP` or `HTTPS`, whereas `http.url.path` might be `/api/v1/query`. For more details around semantic naming conventions, refer to the [Recommendations for OpenTelemetry Authors](https://opentelemetry.io/docs/specs/otel/common/attribute-naming/#recommendations-for-opentelemetry-authors) and [OpenTelemetry Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/README.md) documentation. Some third-party libraries provide auto-instrumentation that generate span and span attributes when included in a source base. For more information about instrumenting your app for tracing, refer to the [Instrument for distributed tracing](/docs/tempo/latest/getting-started/instrumentation/) documentation. ### How many attributes should spans have? The decision of how many attributes to include in your spans is up to you. There is no hard and fast rule. Keep the number of attributes to a minimum, as each attribute adds overhead to the tracing system. In Grafana Cloud, this results in higher tracing costs. Only include attributes that are relevant to the operation that the span represents. For example, if you are tracing an HTTP request, you might include attributes such as the request method, the URL, and the response status code. If you are unsure whether or not to include an attribute, it is always better to err on the side of caution and leave it out. You can always add additional attributes later if you need them. In general, consider the following guidelines: - Don't include metrics or logs as attributes in your spans. - Don't use redundant attributes. - When determining which attributes to add, consider an application's service flow, and execution in the context of only the current span. The OpenTelemetry project doesn't specify a maximum number of attributes that a span can have. However, the default limits for the number of attributes per span is [128 entries](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#attribute-limits), so you will have to adjust that. There are also default limits on attribute value and name character lengths. ## Determining where to add spans When instrumenting, determine the smallest piece of work that you need to observe in a trace to be of value to ensure that you don’t over (or under) instrument. Creating a new span for any work that has a relatively significant duration allows the observation of a trace to immediately show where significant amounts of time are spent during the processing of a request into your application or system. For example, adding a span for a call to another services (either instrumented or not) may take an unknown amount of time to complete, and therefore being able to separate this work shows when services are taking longer than expected. Adding a span for a piece of work that might call many other functions in a loop is a good signal of how long that loop is taking (you might add a span attribute that counts how many time the loop runs to determine if the duration is acceptable). However, adding a span for each method or function call in that loop might not, as it might produce hundreds or thousands of worthless spans. ## Span length While there are some (high) default limits to the length that a span (and by definition, the traces they belong to) can be, these can be adjusted by [these configurations](https://grafana.com/docs/tempo/<TEMPO_VERSION>/configuration/#ingestion-limits). Traces that include a large number of spans and/or long-running spans can have an impact on the time taken to query them once stored. Cloud Traces users should contact Grafana Support to modify overrides. For long-running spans and traces, the best way to see this impact on requests is to send a few test cases and see what the performance looks like and to evaluate the trace size. From there, you can modify the configuration for Tempo or determine ways to re-architect how the trace is produced. You can consider breaking up the spans in several ways: - Decompose the query - For example, if a complex SQL query involves multiple operations (for example, uses joins, subqueries, or unions), consider creating separate spans for each significant operation. - Improve granulation of long-running spans - For long-running operations, you could create a new span for every predetermined interval of execution time. {{< admonition type="note" >}} This requires time-based tracking in your application's code and is more complex to implement. {{< /admonition >}} - Use span linking - Should data flow hit bottlenecks where further operations on that data might be batched at a later time, the use of [span links](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/overview.md#links-between-spans) can help keep traces constrained to an acceptable time range, while sharing context with other traces that work on the same data. This can also improve the readability of traces. --- --- METRICS --- - Files Extracted: 101 - Estimated Token Budget: ~186180 tokens - Canonical Reference: https://codewiki.google/github.com/grafana/tempo