jaeger Documentation

1. Project Overview & Quickstart (jaegertracing/jaeger) Copy Chapter

File: README.md Copy Chapter

[

Jaeger - a Distributed Tracing System

💥💥💥 Jaeger v2 is out! Read the blog post and try it out.

Quick Start Copy Chapter

Get Jaeger running in seconds with Docker:

# Run Jaeger all-in-one (includes UI, collector, query, and in-memory storage)
docker run --rm --name jaeger \
  -p 16686:16686 \
  -p 4317:4317 \
  -p 4318:4318 \
  jaegertracing/jaeger:latest

# Access the UI at http://localhost:16686
# Send traces via OTLP: gRPC on port 4317, HTTP on port 4318

For production deployments and more options, see the Getting Started Guide.

Architecture Copy Chapter

graph TD
    SDK["OpenTelemetry SDK"] --> |HTTP or gRPC| COLLECTOR
    COLLECTOR["Jaeger Collector"] --> STORE[Storage]
    COLLECTOR --> |gRPC| PLUGIN[Storage Plugin]
    COLLECTOR --> |gRPC/sampling| SDK
    PLUGIN --> STORE
    QUERY[Jaeger Query Service] --> STORE
    QUERY --> |gRPC| PLUGIN
    UI[Jaeger UI] --> |HTTP| QUERY
    subgraph Application Host
        subgraph User Application
            SDK
        end
    end

Jaeger is a distributed tracing platform created by Uber Technologies and donated to Cloud Native Computing Foundation.

See Jaeger documentation for getting started, operational details, and other information.

Jaeger is hosted by the Cloud Native Computing Foundation (CNCF) as the 7th top-level project, graduated in October 2019. See the CNCF Jaeger incubation announcement and Jaeger graduation announcement.

Get Involved Copy Chapter

Jaeger is an open source project with open governance. We welcome contributions from the community, and we would love your help to improve and extend the project. Here are some ideas for how to get involved. Many of them do not even require any coding.

Version Compatibility Guarantees Copy Chapter

Since Jaeger uses many components from the OpenTelemetry Collector we try to maintain configuration compatibility between Jaeger releases. Occasionally, configuration options in Jaeger (or in Jaeger v1 CLI flags) can be deprecated due to usability improvements, new functionality, or changes in our dependencies.
In such situations, developers introducing the deprecation are required to follow these guidelines.

In short, for a deprecated configuration option, you should expect to see the following message in the documentation or release notes:

(deprecated, will be removed after yyyy-mm-dd or in release vX.Y.Z, whichever is later)

A grace period of at least 3 months or two minor version bumps (whichever is later) from the first release
containing the deprecation notice will be provided before the deprecated configuration option can be deleted.

For example, consider a scenario where v2.0.0 is released on 01-Sep-2024 containing a deprecation notice for a configuration option.
This configuration option will remain in a deprecated state until the later of 01-Dec-2024 or v2.2.0 where it can be removed on or after either of those events.
It may remain deprecated for longer than the aforementioned grace period.

Go Version Compatibility Guarantees Copy Chapter

The Jaeger project attempts to track the currently supported versions of Go, as defined by the Go team.
Removing support for an unsupported Go version is not considered a breaking change.

Starting with the release of Go 1.21, support for Go versions will be updated as follows:

1. Soon after the release of a new Go minor version N, updates will be made to the build and tests steps to accommodate the latest Go minor version.
2. Soon after the release of a new Go minor version N, support for Go version N-1 will be removed and version N will become the minimum required version.

Note: All importable code has been moved to internal packages, so there is no need to maintain backward compatibility with older compilers (previously version N-1 was used).

Storage Backend Version Support Policy Copy Chapter

This policy covers storage backend versions only while they are reasonable for the project to test, maintain, and recommend for production use. Policy coverage is distinct from versions that may remain technically compatible with Jaeger.

For backends with a published upstream lifecycle, Jaeger bases its policy on the backend project's or vendor's supported-version or end-of-life policy, with the exact coverage defined below. Once upstream support ends, Jaeger may remove CI coverage, stop proactively fixing backend-specific issues for that version, and require users to upgrade before troubleshooting.

For backends without a clear upstream lifecycle, Jaeger supports the current stable major version and the immediately preceding major version. Older versions are best-effort only and may be removed when they block development, security fixes, dependency updates, or CI reliability.

Backend-specific policy:

• Elasticsearch: follow Elastic's Product and Version End of Life Policy.
• OpenSearch: follow the OpenSearch Release Schedule and Maintenance Policy. In practice, this means the current major version and the previous major version while they remain maintained.
• Cassandra: follow Apache Cassandra's maintained release lines, as published on the Cassandra download page, with the Jaeger support target being the current major version and the previous major version.
• ClickHouse: follow ClickHouse's release and backport policy and production guidance for stable and LTS releases. Jaeger treats LTS releases as its support lines and covers the current and previous LTS releases, which ClickHouse supports simultaneously for at least 12 months. Monthly stable releases may remain compatible or appear in CI but are not covered by this policy.

Support means backend versions are in scope for proactive compatibility fixes. CI coverage is not the definitive support matrix: it may test only a subset of supported versions and may retain older versions on a best-effort basis. CI coverage alone does not make a backend version supported. Jaeger does not provide support for the backend product itself, vendor-specific distributions, or versions outside the upstream maintenance window.

As of July 6, 2026, this policy means:

This table is informational and should be refreshed when upstream policies or release lines change. Versions listed as not covered may still appear in CI on a best-effort basis; their presence in CI does not extend the support policy.

Related Repositories Copy Chapter

Components Copy Section

• UI
• Data model

Documentation Copy Section

• Published: https://www.jaegertracing.io/docs/
• Source: https://github.com/jaegertracing/documentation

Building From Source Copy Chapter

See CONTRIBUTING.

Contributing Copy Chapter

See CONTRIBUTING.

Thanks to all the people who already contributed!

Maintainers Copy Section

Rules for becoming a maintainer are defined in the GOVERNANCE document.
The official maintainers of the Jaeger project are listed in the MAINTAINERS file.
Please use @jaegertracing/jaeger-maintainers to tag them on issues / PRs.

Some repositories under jaegertracing org have additional maintainers.

Project Status Meetings Copy Chapter

The Jaeger maintainers and contributors meet regularly on a video call. Everyone is welcome to join, including end users. For meeting details, see https://www.jaegertracing.io/get-in-touch/.

Roadmap Copy Chapter

See https://www.jaegertracing.io/docs/roadmap/

Get in Touch Copy Chapter

Have questions, suggestions, bug reports? Reach the project community via these channels:

• Slack chat room #jaeger (need to join CNCF Slack for the first time)
• jaeger-tracing mail group
• GitHub issues and discussions

Security Copy Chapter

Third-party security audits of Jaeger are available in https://github.com/jaegertracing/security-audits. Please see Issue #1718 for the summary of available security mechanisms in Jaeger.

Adopters Copy Chapter

Jaeger as a product consists of multiple components. We want to support different types of users,
whether they are only using our instrumentation libraries or full end to end Jaeger installation,
whether it runs in production or you use it to troubleshoot issues in development.

Please see ADOPTERS.md for some of the organizations using Jaeger today.
If you would like to add your organization to the list, please comment on our
survey issue.

Sponsors Copy Chapter

The Jaeger project owes its success in open source largely to the Cloud Native Computing Foundation (CNCF), our primary supporter. We deeply appreciate their vital support. Furthermore, we are grateful to Uber for their initial, project-launching donation, and for the continuous contributions of software and infrastructure from 1Password, Codecov.io, Dosu, GitHub, Google Analytics, Netlify, and Oracle Cloud Infrastructure. Thank you for your generous support.

License Copy Chapter

Copyright (c) The Jaeger Authors. Apache 2.0 License.

File: cmd/tracegen/README.md Copy Chapter

tracegen

tracegen is a utility that can generate a steady flow of simple traces useful for performance tuning.
Traces are produced concurrently from one or more worker goroutines. Run with -h to see all cli flags.

The binary is available from the Releases page, as well as a Docker image:

$ docker run jaegertracing/jaeger-tracegen -service abcd -traces 10

The generator can be configured to export traces in different formats, via -exporter flag.
By default, the exporters send data to localhost. If running in a container, this refers
to the networking namespace of the container itself, so to export to another container,
the exporters need to be provided with appropriate location.
OTLP exporter accepts configuration via environment variables.
For more information about configuring OTLP exporter, see OpenTelemetry Protocol Exporter.

See example in the included docker-compose file.

File: cmd/jaeger/docs/migration/all-in-one-metrics.md Copy Chapter

ALL-IN-ONE METRICS

Combined Metrics Copy Section

Equivalent Metrics Copy Section

File: cmd/jaeger/docs/migration/badger-metrics.md Copy Chapter

BADGER METRICS

Combined Metrics Copy Section

Equivalent Metrics Copy Section

File: cmd/jaeger/docs/migration/cassandra-metrics.md Copy Chapter

CASSANDRA METRICS

Combined Metrics Copy Section

Equivalent Metrics Copy Section

File: cmd/jaeger/docs/migration/elasticsearch-metrics.md Copy Chapter

ELASTICSEARCH METRICS

Combined Metrics Copy Section

Equivalent Metrics Copy Section

File: cmd/jaeger/docs/migration/opensearch-metrics.md Copy Chapter

OPENSEARCH METRICS

Combined Metrics Copy Section

Equivalent Metrics Copy Section

File: cmd/jaeger/internal/exporters/storageexporter/README.md Copy Chapter

jaeger_storage_exporter

This module implements exporter.Traces and writes spans into Jaeger native spanstore.SpanWriter, that it obtains from the jaeger_storage extension. This is primarily needed to wire a memory storage into the exporter pipeline (used for all-in-one), but the design of the exporter is such that it can do this for any V1 storage implementation.

Configuration Copy Chapter

Basic Configuration Copy Section

exporters:
  jaeger_storage_exporter:
    trace_storage: memstore

extensions:
  jaeger_storage:
    memory:
      memstore:
        max_traces: 100000

Retry Configuration Copy Section

The exporter supports configurable retry behavior for failed exports. By default, retry is disabled. When enabled, the following default settings are used:

• enabled: false (disabled by default)
• initial_interval: 5s
• randomization_factor: 0.5
• multiplier: 1.5
• max_interval: 30s
• max_elapsed_time: 5m

To enable and customize the retry behavior:

exporters:
  jaeger_storage_exporter:
    trace_storage: memstore
    retry_on_failure:
      enabled: true  # Explicitly enable retries
      initial_interval: 10s
      max_interval: 1m
      max_elapsed_time: 10m
    queue:
      enabled: true
      num_consumers: 10
      queue_size: 1000

File: cmd/jaeger/internal/extension/remotestorage/README.md Copy Chapter

Remote Storage Backend Extension

The Remote Storage Backend Extension exposes any Jaeger-supported storage backend through a gRPC API by
implementing the gRPC Storage API. The gRPC API currently consists of the following services:

• TraceReader
• DependencyReader

Overview Copy Chapter

This extension allows you to make Jaeger storage backends accessible remotely over gRPC.
For example, you can expose the in-memory storage backend like this:

extensions:
  jaeger_storage:
    backends:
      memory-storage:
        memory:
          max_traces: 100000
  remote_storage:
    endpoint: localhost:17271
    storage: memory-storage

File: cmd/jaeger/internal/extension/remotesampling/README.md Copy Chapter

Remote Sampling

Placeholder

/* Detailed source-code truncated for AI context efficiency. */

2. Official Technical Reference & Guides (jaegertracing/documentation) Copy Chapter

File: README.md Copy Chapter

Jaeger website

This repo houses all the assets used to build the Jaeger website, available at https://jaegertracing.io.

The site is built with Hugo and hosted by Netlify.

Contributing to the site Copy Chapter

We strongly encourage you to contribute to this site! For more information, see the contributing guide.

Publishing the site Copy Chapter

The site is published automatically by Netlify whenever changes are merged to the main branch. The site cannot be published in an ad-hoc way (e.g. through a make command or script in the repo).

Publishing new Jaeger version Copy Chapter

Please refer to RELEASE.md for instructions on how to release new version of documentation.

License Copy Chapter

Apache 2.0 License.

File: content/announcements/_index.md Copy Chapter

title: Announcements
toc_hide: true
cascade:
type: docs
params:
hide_feedback: true

redirects: [{ from: '*', to: '?' }] Copy Chapter

File: content/announcements/kubecon-na.md Copy Chapter

title: KubeCon North America 2025 date: 2025-10-01 expiryDate: 2025-11-13 Copy Chapter

Join us in Atlanta, Georgia for KubeCon +
CloudNativeCon North America 2025, November 10 - 13, 2025

File: content/announcements/ukraine.md Copy Chapter

title: We stand with Ukraine weight: -1 cSpell:ignore: ukraine body_class: ukraine-banner Copy Chapter

We stand with our friends and colleagues in Ukraine. To assist
in their time of need, visit Support Ukraine.

File: content/docs/v1/1.76/architecture/_index.md Copy Chapter

title: Architecture
weight: 3
children:

• title: APIs
  url: apis
• title: Sampling
  url: sampling

Terminology Copy Chapter

Jaeger represents tracing data in a data model inspired by the OpenTracing Specification. The data model is logically very similar to OpenTelemetry Traces, with some naming differences:

Span Copy Section

A span represents a logical unit of work that has an operation name, the start time of the operation, and the duration. Spans may be nested and ordered to model causal relationships.

Trace Copy Section

A trace represents the data or execution path through the system. It can be thought of as a directed acyclic graph of spans.

Baggage Copy Section

Baggage is arbitrary user-defined metadata (key-value pairs) that can be attached to distributed context and propagated by the tracing SDKs. See W3C Baggage for more information.

Architecture Copy Chapter

Jaeger can be deployed either as an all-in-one binary, where all Jaeger backend components
run in a single process, or as a scalable distributed system. There are two main deployment options discussed below.

Direct to storage Copy Section

In this deployment the collectors receive the data from traced applications and write it directly to storage. The storage must be able to handle both average and peak traffic. Collectors use an in-memory queue to smooth short-term traffic peaks, but a sustained traffic spike may result in dropped data if the storage is not able to keep up.

Collectors are able to centrally serve sampling configuration to the SDKs, known as remote sampling mode. They can also enable automatic sampling configuration calculation, known as adaptive sampling.

Via Kafka Copy Section

To prevent data loss between collectors and storage, Kafka can be used as an intermediary, persistent queue. An additional component, jaeger-ingester, needs to be deployed to read data from Kafka and save to the database. Multiple jaeger-ingesters can be deployed to scale up ingestion; they will automatically partition the load across them.

With OpenTelemetry Collector Copy Section

You do not need to use OpenTelemetry Collector, because jaeger-collector can receive OpenTelemetry data directly from the OpenTelemetry SDKs (using OTLP exporters). However, if you already use the OpenTelemetry Collectors, such as for gathering other types of telemetry or for pre-processing / enriching the tracing data, it can be placed between the SDKs and jaeger-collector's. The OpenTelemetry Collectors can be run as an application sidecar, as a host agent / daemon, or as a central cluster.

The OpenTelemetry Collector supports Jaeger's Remote Sampling protocol and can either serve static configurations from config files directly, or proxy the requests to the Jaeger backend (e.g., when using adaptive sampling).

OpenTelemetry Collector as a sidecar / host agent

Benefits:

• The SDK configuration is simplified as both trace export endpoint and sampling config endpoint can point to a local host and not worry about discovering where those services run remotely.
• Collector may provide data enrichment by adding environment information, like k8s pod name.
• Resource usage for data enrichment can be distributed across all application hosts.

Downsides:

• An extra layer of marshaling/unmarshaling the data.

OpenTelemetry Collector as a remote cluster

Benefits:

• Sharding capabilities, e.g., when using tail-based sampling.

Downsides:

• An extra layer of marshaling/unmarshaling the data.

Components Copy Chapter

This section details the constituent parts of Jaeger and how they relate to each other. It is arranged by the order in which spans from your application interact with them.

Tracing SDKs Copy Section

In order to generate tracing data, the applications must be instrumented with tracing SDK such as OpenTelemetry SDKs. An instrumented application creates spans when receiving new requests and attaches context information (trace id, span id, and baggage) to outgoing requests. Only the ids and baggage are propagated with requests; all other profiling data, like operation name, timing, tags and logs, is not propagated. Instead, it is exported out of process to the Jaeger backend asynchronously, in the background.

There are many ways to instrument an application:

• manually, using the tracing APIs directly,
• relying on instrumentation already created for a variety of existing open source frameworks,
• automatically, via byte code manipulation, monkey-patching, eBPF, and similar techniques.

Instrumentation typically should not depend on specific tracing SDKs, but only on abstract tracing APIs like the OpenTelemetry API. The tracing SDKs implement the tracing APIs and take care of data export.

The instrumentation is designed to be always on in production. To minimize overhead, the SDKs employ various sampling strategies. When a trace is sampled, the profiling span data is captured and transmitted to the Jaeger backend. When a trace is not sampled, no profiling data is collected at all, and the calls to the tracing API are short-circuited to incur a minimal amount of overhead. For more information, please refer to the Sampling page.

Collector Copy Section

jaeger-collector receives traces, runs them through a processing pipeline for validation and clean-up/enrichment, and stores them in a storage backend. Jaeger comes with built-in support for several storage backends (see Deployment), as well as extensible plugin framework for implementing custom storage plugins.

Query Copy Section

jaeger-query is a service that exposes the APIs for retrieving traces from storage and hosts a Web UI for searching and analyzing traces.

Ingester Copy Section

jaeger-ingester is a service that reads traces from Kafka and writes them to a storage backend. Effectively, it is a stripped-down version of the Jaeger collector that supports Kafka as the only input protocol.

File: content/docs/v1/1.76/architecture/apis.md Copy Chapter

title: APIs aliases: [../apis] hasparent: true Copy Chapter

Jaeger components implement various APIs for saving or retrieving trace data.

The following labels are used to describe API compatibility guarantees.

• stable - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. /api/v2 URL prefix or a different namespace in the IDL.
• internal - the APIs are intended for internal communications between Jaeger components and are not recommended for use by external components.
• deprecated - the APIs that are only maintained for legacy reasons and will be phased out in the future.

Since Jaeger v1.32, jaeger-collector and jaeger-query Service ports that serve gRPC endpoints enable gRPC reflection. Unfortunately, the internally used gogo/protobuf has a compatibility issue with the official golang/protobuf, and as a result only the list reflection command is currently working properly.

Span reporting APIs Copy Chapter

jaeger-collector is the component of the Jaeger backend that can receive spans. At this time it supports two sets of non-overlapping APIs.

OpenTelemetry Protocol (stable) Copy Section

Since v1.35, the Jaeger backend can receive trace data from the OpenTelemetry SDKs in their native OpenTelemetry Protocol (OTLP). It is no longer necessary to configure the OpenTelemetry SDKs with Jaeger exporters, nor deploy the OpenTelemetry Collector between the OpenTelemetry SDKs and the Jaeger backend.

The OTLP data is accepted in these formats: (1) binary gRPC, (2) Protobuf over HTTP, (3) JSON over HTTP. For more details on the OTLP receiver see the official documentation. Note that not all configuration options are supported in jaeger-collector (see --collector.otlp.* CLI Flags), and only tracing data is accepted, since Jaeger does not store other telemetry types.

Protobuf via gRPC (stable) Copy Section

Deprecated: we recommend the OpenTelemetry protocol.

Since Jaeger v1.11, the official protocol between user applications and jaeger-collectors is jaeger.api_v2.CollectorService gRPC endpoint defined in collector.proto IDL file. The same endpoint can be used to submit trace data from SDKs directly to jaeger-collector.

Thrift over HTTP (stable) Copy Section

Deprecated: we recommend the OpenTelemetry protocol.

The payload in jaeger.thrift format can be submitted in an HTTP POST request to the /api/traces endpoint, for example, https://jaeger-collector:14268/api/traces. The Batch struct needs to be encoded using Thrift's binary encoding, and the HTTP request should specify the content type header:

Content-Type: application/vnd.apache.thrift.binary

JSON over HTTP (n/a) Copy Section

There is no official Jaeger JSON format that can be accepted by jaeger-collector.
Jaeger does accept the OpenTelemetry protocol via JSON (see above).

Zipkin Formats (stable) Copy Section

jaeger-collector can also accept spans in several Zipkin data formats, namely JSON v1/v2 and Thrift. jaeger-collector needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:

• /api/v1/spans for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
• /api/v2/spans for submitting spans in Zipkin JSON v2.

Trace retrieval APIs Copy Chapter

Traces saved in the storage can be retrieved by calling jaeger-query Service.

gRPC/Protobuf (stable) Copy Section

The recommended way for programmatically retrieving traces and other data is via the jaeger.api_v2.QueryService gRPC endpoint defined in query.proto IDL file. In the default configuration this endpoint is accessible from jaeger-query:16685.

HTTP JSON (internal) Copy Section

Jaeger UI communicates with jaeger-query Service via JSON API. For example, a trace can be retrieved via a GET request to https://jaeger-query:16686/api/traces/{trace-id-hex-string}. This JSON API is intentionally undocumented and subject to change.

Remote Storage API (stable) Copy Chapter

When using the grpc storage type (a.k.a. remote storage), Jaeger components can use custom storage backends as long as those backends implement the gRPC Remote Storage API.

Remote Sampling Configuration (stable) Copy Chapter

This API supports Jaeger's Remote Sampling protocol, defined in the sampling.proto IDL file.

jaeger-collector implements this API. See Remote Sampling for details on how to configure the Collector with sampling strategies.

The following table lists different endpoints and formats that can be used to query for sampling strategies. The official HTTP/JSON endpoints use standard Protobuf-to-JSON mapping.

Examples

Run all-in-one in one terminal:

$ go run ./cmd/all-in-one \
  --sampling.strategies-file=cmd/all-in-one/sampling_strategies.json

Query the endpoint in another terminal:

$ curl "http://localhost:14268/api/sampling?service=foo"
{"strategyType":"PROBABILISTIC","probabilisticSampling":{"samplingRate":1}}

Service dependencies graph (internal) Copy Chapter

Can be retrieved fromjaeger-query Service at /api/dependencies endpoint. The GET request expects two parameters:

• endTs (number of milliseconds since epoch) - the end of the time interval
• lookback (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).

The returned JSON is a list of edges represented as tuples (caller, callee, count).

For programmatic access to the service graph, the recommended API is gRPC/Protobuf described above.

Service Performance Monitoring (internal) Copy Chapter

Please refer to the SPM Documentation

File: content/docs/v1/1.76/architecture/sampling.md Copy Chapter

title: Sampling aliases: [../sampling] hasparent: true Copy Chapter

Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: A -> B -> C. When service A receives a request that contains no tracing information, Jaeger tracer will start a new trace, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its spans will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.

Client Sampling Configuration Copy Chapter

{{< warning >}}
This section only applies to classic Jaeger SDKs, which are now deprecated.
We recommend using the OpenTelemetry SDKs.
{{< /warning >}}

When using configuration object to instantiate the tracer, the type of sampling can be selected via sampler.type and sampler.param properties. Jaeger libraries support the following samplers:

• Constant (sampler.type=const) sampler always makes the same decision for all traces. It either samples all traces (sampler.param=1) or none of them (sampler.param=0).
• Probabilistic (sampler.type=probabilistic) sampler makes a random sampling decision with the probability of sampling equal to the value of sampler.param property. For example, with sampler.param=0.1 approximately 1 in 10 traces will be sampled.
• Rate Limiting (sampler.type=ratelimiting) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, when sampler.param=2.0 it will sample requests with the rate of 2 traces per second.
• Remote (sampler.type=remote, which is also the default) sampler consults jaeger-collector for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a central configuration in Jaeger backend (see Remote Sampling), or even dynamically (see Adaptive Sampling).

Remote Sampling Copy Chapter

If your client SDKs are configured to use remote sampling configuration (see Remote Sampling API) then sampling rates can be centrally controlled via jaeger-collectors. In this setup a sampling strategy configuration is served to the client SDK that describes endpoints and their sampling probabilities. This configuration can be generated by jaeger-collector in two different ways: periodically loaded from a file or dynamically calculated based on traffic. The method of generation is controlled by the environment variable SAMPLING_CONFIG_TYPE which can be set to either file (default) or adaptive.

File-based Sampling Configuration Copy Section

jaeger-collectors can be instantiated with the --sampling.strategies-file option that points to a file containing sampling strategies to be served to Jaeger clients. The option's value can contain a path to a JSON file, which will be automatically reloaded if its contents change, or an HTTP URL from where the file will be periodically retrieved, with reload frequency controlled by the --sampling.strategies-reload-interval option.

If no configuration is provided, jaeger-collectors will return the default probabilistic sampling policy with probability 0.001 (0.1%) for all services.

Example strategies.json:

{
  "service_strategies": [
    {
      "service": "foo",
      "type": "probabilistic",
      "param": 0.8,
      "operation_strategies": [
        {
          "operation": "op1",
          "type": "probabilistic",
          "param": 0.2
        },
        {
          "operation": "op2",
          "type": "probabilistic",
          "param": 0.4
        }
      ]
    },
    {
      "service": "bar",
      "type": "ratelimiting",
      "param": 5
    }
  ],
  "default_strategy": {
    "type": "probabilistic",
    "param": 0.5,
    "operation_strategies": [
      {
        "operation": "/health",
        "type": "probabilistic",
        "param": 0.0
      },
      {
        "operation": "/metrics",
        "type": "probabilistic",
        "param": 0.0
      }
    ]
  }
}

service_strategies element defines service specific sampling strategies and operation_strategies defines operation specific sampling strategies. There are 2 types of strategies possible: probabilistic and ratelimiting which are described above (NOTE: ratelimiting is not supported for operation_strategies). default_strategy defines the catch-all sampling strategy that is propagated if the service is not included as part of service_strategies.

In the above example:

• All operations of service foo are sampled with probability 0.8 except for operations op1 and op2 which are probabilistically sampled with probabilities 0.2 and 0.4 respectively.
• All operations for service bar are rate-limited at 5 traces per second.
• Any other service will be sampled with probability 0.5 defined by the default_strategy.
• The default_strategy also includes shared per-operation strategies. In this example we disable tracing on /health and /metrics endpoints for all services by using probability 0. These per-operation strategies will apply to any new service not listed in the config, as well as to the foo and bar services unless they define their own strategies for these two operations.

Adaptive Sampling Copy Section

Since Jaeger v1.27.

Adaptive sampling works in jaeger-collector by observing the spans received from services and recalculating sampling probabilities for each service/endpoint combination to ensure that the volume of collected traces matches --sampling.target-samples-per-second. When a new service or endpoint is detected, it is initially sampled with --sampling.initial-sampling-probability until enough data is collected to calculate the rate appropriate for the traffic going through the endpoint.

Adaptive sampling requires a storage backend to store the observed traffic data and computed probabilities. At the moment memory (for all-in-one deployment), cassandra, badger, elasticsearch and opensearch are supported as sampling storage backends.

By default adaptive sampling will attempt to use the backend specified by SPAN_STORAGE_TYPE to store data. However, a second type of backend can also be specified by using SAMPLING_STORAGE_TYPE. For instance, SPAN_STORAGE_TYPE=elasticsearch SAMPLING_STORAGE_TYPE=cassandra ./jaeger-collector will run jaeger-collector in a mode where it attempts to store its span data in the configured elasticsearch cluster and its adaptive sampling data in the configured cassandra cluster. Note that this feature can not be used to store span and adaptive sampling data in two different backends of the same type.

Read this blog post for more details on adaptive sampling engine.

File: content/docs/v1/1.76/deployment/_index.md Copy Chapter

title: Deployment
weight: 4
children:

• title: Operator for Kubernetes
  navtitle: Kubernetes
  url: operator
• title: Frontend/UI
  url: frontend-ui
• title: CLI Flags
  url: cli
• title: Security
  url: security
• title: On Windows
  url: windows
• title: Service Performance Monitoring (SPM)
  navtitle: SPM
  url: spm

The main Jaeger backend components are released as Docker images on Docker Hub and Quay:

The images listed above are the primary release versions. Most components have additional images published:

• ${component}-debug images include Delve debugger
• ${component}-snapshot images are published from the tip of the main branch for every commit, allowing testing of unreleased versions
• ${component}-debug-snapshot snapshot images that include the Delve debugger

There are orchestration templates for running Jaeger with:

• Kubernetes: github.com/jaegertracing/jaeger-kubernetes,
• OpenShift: github.com/jaegertracing/jaeger-openshift.

Configuration Options Copy Chapter

Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):

• command line arguments,
• environment variables,
• configuration files in JSON, TOML, YAML, HCL, or Java properties formats.

To see the complete list of options, run the binary with the help command or refer to the CLI Flags page for more information. Options that are specific to a certain storage backend are only listed if the storage type is selected. For example, to see all available options in the Collector with Cassandra storage:

$ docker run --rm \
    -e SPAN_STORAGE_TYPE=cassandra \
    jaegertracing/jaeger-collector:{{< currentVersion >}} \
    help

In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:

All-in-one Copy Chapter

Jaeger all-in-one is a special distribution that combines three Jaeger components, collector, and query service/UI, in a single binary or container image. It is useful for single-node deployments where your trace volume is light enough to be handled by a single instance. By default, all-in-one starts with memory storage, meaning it will lose all data upon restart. All other span storage backends can also be used with all-in-one, but memory and badger are exclusive to all-in-one because they cannot be shared between instances.

All-in-one listens to the same ports as the components it contains (described below), with the exception of the admin port.

## make sure to expose only the ports you use in your deployment scenario!
docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14250:14250 \
  -p 14268:14268 \
  -p 14269:14269 \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 9411:9411 \
  jaegertracing/all-in-one:{{< currentVersion >}}

You can navigate to http://localhost:16686 to access the Jaeger UI.

Collector Copy Chapter

jaeger-collectors are stateless and thus many instances of jaeger-collector can be run in parallel.
jaeger-collectors require almost no configuration, except for storage location, such as
--cassandra.keyspace and --cassandra.servers options, or the location of Elasticsearch cluster,
via --es.server-urls, depending on which storage is specified. See the CLI Flags for all
command line options.

At default settings jaeger-collector exposes the following ports:

Ingester Copy Chapter

jaeger-ingester is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).

To view all exposed configuration options run the following command:

docker run \
  -e SPAN_STORAGE_TYPE=cassandra \
  jaegertracing/jaeger-ingester:{{< currentVersion >}}
  --help

Query Service & UI Copy Chapter

jaeger-query serves the API endpoints and a React/Javascript UI.
The service is stateless and is typically run behind a load balancer, such as NGINX.

At default settings the jaeger-query service exposes the following port(s):

Minimal deployment example (Elasticsearch backend): Copy Section

docker run -d --rm \
  -p 16685:16685 \
  -p 16686:16686 \
  -p 16687:16687 \
  -e SPAN_STORAGE_TYPE=elasticsearch \
  -e ES_SERVER_URLS=http://<ES_SERVER_IP>:<ES_SERVER_PORT> \
  jaegertracing/jaeger-query:{{< currentVersion >}}

Clock Skew Adjustment Copy Section

Jaeger backend combines trace data from applications that are usually running on different hosts. The hardware clocks on the hosts often experience relative drift, known as the clock skew effect. Clock skew can make it difficult to reason about traces, for example, when a server span may appear to start earlier than the client span, which should not be possible. jaeger-query service implements a clock skew adjustment algorithm (code) to correct for clock drift, using the knowledge about causal relationships between spans. All adjusted spans have a warning displayed in the UI that provides the exact clock skew delta applied to their timestamps.

Sometimes these adjustments themselves make the trace hard to understand. For example, when repositioning the server span within the bounds of its parent span, Jaeger does not know the exact relationship between the request and response latencies, so it assumes they are equal and places the child span in the middle of the parent span (see issue #961).

jaeger-query service supports a configuration flag --query.max-clock-skew-adjustment that controls how much clock skew adjustment should be allowed. Setting this parameter to zero (0s) disables clock skew adjustment completely. This setting applies to all traces retrieved from the given query service. There is an open ticket #197 to support toggling the adjustment on and off directly in the UI.

UI Base Path Copy Section

The base path for all jaeger-query HTTP routes can be set to a non-root value, e.g. /jaeger would cause all UI URLs to start with /jaeger. This can be useful when running jaeger-query behind a reverse proxy.

The base path can be configured via the --query.base-path command line parameter or the QUERY_BASE_PATH environment variable.

UI Customization and Embedding Copy Section

Please refer to the dedicated Frontend/UI page.

Remote Storage (component) Copy Chapter

jaeger-remote-storage implements the Remote Storage gRPC API and proxies it into one of the regular Jaeger backends. It can be useful in the situation when we want to run a full deployment of Jaeger components, e.g., separate collector and query services, but use a single-node storage backend like the memory store or Badger. Without the remote storage, the single-node backends can only be used with all-in-one since they cannot be shared between multiple processes.

At default settings the service listens on the following port(s):

Span Storage Backends Copy Chapter

Jaeger requires a persistent storage backend. Cassandra and Elasticsearch/OpenSearch are the primary supported distributed storage backends. Additional backends are discussed here.

The storage type can be passed via SPAN_STORAGE_TYPE environment variable. Valid values are cassandra, elasticsearch, kafka (only as a buffer), badger and memory.

As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the SPAN_STORAGE_TYPE environment variable. It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.

For large scale production deployment the Jaeger team recommends OpenSearch backend over Cassandra.

Memory Copy Section

The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
data will be lost once the process exits.

By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
integer value via --memory.max-traces.

Badger - local storage Copy Section

• Since Jaeger v1.9

Badger is an embedded local storage, only available
with all-in-one distribution. By default, it acts as ephemeral storage using a temporary file system.
This can be overridden by using the --badger.ephemeral=false option.

docker run \
  -e SPAN_STORAGE_TYPE=badger \
  -e BADGER_EPHEMERAL=false \
  -e BADGER_DIRECTORY_VALUE=/badger/data \
  -e BADGER_DIRECTORY_KEY=/badger/key \
  -v <storage_dir_on_host>:/badger \
  -p 16686:16686 \
  jaegertracing/all-in-one:{{< currentVersion >}}

• Upgrade Badger v1 to v3
• Badger file permissions as non-root service

Cassandra Copy Section

• Supported versions: 4.x, 5.x

Deploying Cassandra itself is out of scope for our documentation. One good
source of documentation is the Apache Cassandra Docs.

Configuration

Minimal

docker run \
  -e SPAN_STORAGE_TYPE=cassandra \
  -e CASSANDRA_SERVERS=<...> \
  jaegertracing/jaeger-collector:{{< currentVersion >}}

Note: White space characters are allowed in CASSANDRA_SERVERS. For Example: Servers can be passed as `CASSANDRA_SERVERS="1.2.3.4, 5.6.7.8" for better readability.

All options

To view the full list of configuration options, you can run the following command:

docker run \
  -e SPAN_STORAGE_TYPE=cassandra  \
  jaegertracing/jaeger-collector:{{< currentVersion >}} \
  --help

Schema script

A script is provided to initialize Cassandra keyspace and schema
using Cassandra's interactive shell cqlsh:

MODE=test sh ./internal/storage/v1/cassandra/schema/create.sh | cqlsh

Or using the published Docker image (make sure to provide the right IP address):

docker run \
  -e CQLSH_HOST={server IP address}  \
  jaegertracing/jaeger-cassandra-schema:{{< currentVersion >}}

For production deployment, pass MODE=prod DATACENTER={datacenter} arguments to the script,
where {datacenter} is the name used in the Cassandra configuration / network topology.

The script also allows overriding TTL, keyspace name, replication factor, etc.
Run the script without arguments to see the full list of recognized parameters.

Note: See README for more details on Cassandra schema management.

TLS support

Jaeger supports TLS client to node connections as long as you've configured
your Cassandra cluster correctly. After verifying with e.g. cqlsh, you can
configure the collector and query like this:

docker run \
  -e CASSANDRA_SERVERS=<...> \
  -e CASSANDRA_TLS=true \
  -e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
  -e CASSANDRA_TLS_KEY= \
  -e CASSANDRA_TLS_CERT= \
  -e CASSANDRA_TLS_CA= \
  jaegertracing/jaeger-collector:{{< currentVersion >}}

The schema tool also supports TLS. You need to make a custom cqlshrc file like
so:

# Creating schema in a cassandra cluster requiring client TLS certificates.
#
# Create a volume for the schema docker container containing four files:
# cqlshrc: this file
# ca-cert: the cert authority for your keys
# client-key: the keyfile for your client
# client-cert: the cert file matching client-key
#
# if there is any sort of DNS mismatch and you want to ignore server validation
# issues, then uncomment validate = false below.
#
# When running the container, map this volume to /root/.cassandra and set the
# environment variable CQLSH_SSL=--ssl
[ssl]
certfile = ~/.cassandra/ca-cert
userkey = ~/.cassandra/client-key
usercert = ~/.cassandra/client-cert
# validate = false

Compatible Backends

• ScyllaDB can be used as a drop-in replacement for Cassandra since it uses the same data model and query language.

Elasticsearch Copy Section

• Supported since Jaeger v0.6.0
• Supported ES versions: 7.x, 8.x (since Jaeger v1.52.0)

Elasticsearch version is automatically retrieved from root/ping endpoint.
Based on this version Jaeger uses compatible index mappings and Elasticsearch REST API.
The version can be explicitly provided via --es.version= flag.

Elasticsearch does not require initialization other than
installing and running Elasticsearch.
Once it is running, pass the correct configuration values to jaeger-collector and jaeger-query.

Configuration

Minimal

docker run \
  -e SPAN_STORAGE_TYPE=elasticsearch \
  -e ES_SERVER_URLS=<...> \
  jaegertracing/jaeger-collector:{{< currentVersion >}}

All options

To view the full list of configuration options, you can run the following command:

docker run \
  -e SPAN_STORAGE_TYPE=elasticsearch \
  jaegertracing/jaeger-collector:{{< currentVersion >}} \
  --help

Shards and Replicas for Elasticsearch indices

Shards and replicas are some configuration values to take special attention to, because this is decided upon
index creation. This article goes into
more information about choosing how many shards should be chosen for optimization.

Elasticsearch Rollover

Elasticsearch rollover is an index management strategy that optimizes use of resources allocated to indices.
For example, indices that do not contain any data still allocate shards, and conversely, a single index might contain significantly more data than the others.
Jaeger by default stores data in daily indices which might not optimally utilize resources. Rollover feature can be enabled by --es.use-aliases=true.

Rollover lets you configure when to roll over to a new index based on one or more of the following criteria:

• max_age - the maximum age of the index. It uses time units: d, h, m.
• max_docs - the maximum documents in the index.
• max_size - the maximum estimated size of primary shards (since Elasticsearch 6.x). It uses byte size units tb, gb, mb.

Rollover index management strategy is more complex than using the default daily indices and it requires an initialization job to prepare the storage and two cron jobs to manage indices.

To learn more about rollover index management in Jaeger refer to this
article.

For automated rollover, please refer to Elasticsearch ILM support.

Initialize

The following command prepares Elasticsearch for rollover deployment by creating index aliases, indices, and index templates:

docker run -it --rm --net=host jaegertracing/jaeger-es-rollover:latest init http://localhost:9200 # <1>

If you need to initialize archive storage, add -e ARCHIVE=true.

After the initialization Jaeger can be deployed with --es.use-aliases=true.

Rolling over to a new index

The next step is to periodically execute the rollover API which rolls the write alias to a new index based on supplied conditions. The command also adds a new index to the read alias to make new data available for search.

docker run -it --rm --net=host -e CONDITIONS='{"max_age": "2d"}' jaegertracing/jaeger-es-rollover:latest rollover  http://localhost:9200 # <1>

<1> The command rolls the alias over to a new index if the age of the current write index is older than 2 days. For more conditions see Elasticsearch docs.

The next step is to remove old indices from read aliases. It means that old data will not be available for search. This imitates the behavior of --es.max-span-age flag used in the default index-per-day deployment. This step could be optional and old indices could be simply removed by index cleaner in the next step.

docker run -it --rm --net=host -e UNIT=days -e UNIT_COUNT=7 jaegertracing/jaeger-es-rollover:latest lookback  http://localhost:9200 # <1>

<1> Removes indices older than 7 days from read alias.

Remove old data

The historical data can be removed with the jaeger-es-index-cleaner that is also used for daily indices.

docker run -it --rm --net=host -e ROLLOVER=true jaegertracing/jaeger-es-index-cleaner:latest 14 http://localhost:9200 # <1>

<1> Remove indices older than 14 days.

Elasticsearch ILM support

{{< warning >}}
Experimental feature added in release v1.22.0.

Supported Elasticsearch versions: 7.x
{{< /warning >}}
Elasticsearch ILM automatically manages indices according to performance, resiliency, and retention requirements.

For example:

• Rollover to a new index by size (bytes or number of documents) or age, archiving previous indices
• Delete stale indices to enforce data retention standards

Enabling ILM support

• Create an ILM policy in elasticsearch named jaeger-ilm-policy.

  For example, the following policy will rollover the "active" index when it is
  older than 1m and delete indices that are older than 2m.

  shell Copy

  curl -X PUT http://localhost:9200/_ilm/policy/jaeger-ilm-policy \
  -H 'Content-Type: application/json; charset=utf-8' \
  --data-binary @- << EOF
  {
    "policy": {
      "phases": {
        "hot": {
          "min_age": "0ms",
          "actions": {
            "rollover": {
              "max_age": "1m"
            },
            "set_priority": {
              "priority": 100
            }
          }
        },
        "delete": {
          "min_age": "2m",
          "actions": {
            "delete": {}
          }
        }
      }
    }
  }
  EOF

• Run elasticsearch initializer with ES_USE_ILM=true:

  shell Copy

  docker run -it --rm --net=host -e ES_USE_ILM=true jaegertracing/jaeger-es-rollover:latest init http://localhost:9200 # <1>

  <1> If you need to initialize archive storage, add -e ARCHIVE=true.

  {{< info >}}
  While initializing with ILM support, make sure that an ILM policy named jaeger-ilm-policy is created in Elasticsearch beforehand (see the previous step), otherwise the following error message will be shown:

  "ILM policy jaeger-ilm-policy doesn't exist in Elasticsearch. Please create it and rerun init"
  {{< /info >}}

  After the initialization, deploy Jaeger with --es.use-ilm=true and --es.use-aliases=true.

Upgrade Elasticsearch version

Elasticsearch defines wire and index compatibility versions. The index compatibility defines
the minimal version a node can read data from. For example Elasticsearch 8 can read indices
created by Elasticsearch 7, however it cannot read indices created by Elasticsearch 6 even
though they use the same index mappings. Therefore upgrade from Elasticsearch 7 to 8 does not require any
data migration. However, upgrade from Elasticsearch 6 to 8 has to be done through Elasticsearch 7 and wait
until indices created by ES 6.x are removed or explicitly reindexed.

Refer to the Elasticsearch documentation
for wire and index compatibility versions. Generally this information can be retrieved from root/ping REST endpoint.

Reindex

Manual reindexing can be used when upgrading from Elasticsearch 6 to 8 (through Elasticsearch 7)
without waiting until indices created by Elasticsearch 6 are removed.

1. Reindex all span indices to new indices with suffix -1:

   bash Copy

   curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
   {
     "source": {
       "index": "jaeger-span-*"
     },
     "dest": {
       "index": "jaeger-span"
     },
     "script": {
       "lang": "painless",
       "source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length())) + '-1'"
     }
   }

2. Delete indices with old mapping:

   bash Copy

   curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*,-\*-1

3. Create indices without -1 suffix:

   bash Copy

   curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
   {
     "source": {
       "index": "jaeger-span-*"
     },
     "dest": {
       "index": "jaeger-span"
     },
     "script": {
       "lang": "painless",
       "source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length() - 2))"
     }
   }

4. Remove suffixed indices:

   bash Copy

   curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*-1

Run the commands analogically for other Jaeger indices.

There might exist more effective migration procedure. Please share with the community any findings.

Kafka Copy Section

• Supported in Jaeger since 1.6.0
• Supported Kafka versions: 0.9+

Kafka can be used as an intermediary buffer between collector and an actual storage.
jaeger-collector is configured with SPAN_STORAGE_TYPE=kafka that makes it write all received spans
into a Kafka topic. jaeger-ingester is used to read from
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).

Writing to Kafka is particularly useful for building post-processing data pipelines.

Configuration

Minimal

docker run \
  -e SPAN_STORAGE_TYPE=kafka \
  -e KAFKA_PRODUCER_BROKERS=<...> \
  -e KAFKA_TOPIC=<...> \
  jaegertracing/jaeger-collector:{{< currentVersion >}}

All options

To view the full list of configuration options, you can run the following command:

docker run \
  -e SPAN_STORAGE_TYPE=kafka \
  jaegertracing/jaeger-collector:{{< currentVersion >}} \
  --help

Topic & partitions

Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to the Kafka quickstart documentation to learn how.

You can find more information about topics and partitions in general in the official documentation. This article provide more details about how to choose the number of partitions.

Remote Storage Copy Section

Jaeger supports a gRPC-based Remote Storage API that allows extending the Jaeger ecosystem with custom storage backends, not directly supported by the project. These storage backends can be deployed as a remote gRPC server (since Jaeger v1.30). Older deployment mode as sidecar plugin will not be supported starting from v1.58.

To use a remote storage as Jaeger storage backend, use grpc as the storage type and specify the remote gRPC server address. For more information, please refer to internal/storage/v1/grpc.

Example:

docker run \
  -e SPAN_STORAGE_TYPE=grpc \
  -e GRPC_STORAGE_SERVER=<...> \
  jaegertracing/all-in-one:{{< currentVersion >}}

Known remote storage backends:

• PostgreSQL by robbert229

Metrics Storage Backends Copy Chapter

Jaeger Query is capable of querying aggregated R.E.D metrics from a storage backend,
visualizing them on the Monitor tab. It should be emphasized that the
configured metrics storage type is for reading only and therefore, only applies
to the Jaeger Query component (and All In One, which contains Jaeger Query).

The storage type can be passed via METRICS_STORAGE_TYPE environment variable.
Valid values are: prometheus.

Prometheus Copy Section

Any PromQL-compatible backend is supported by Jaeger Query. A list of these have
been compiled by Julius Volz in:
https://promlabs.com/blog/2020/11/26/an-update-on-promql-compatibility-across-vendors

Configuration

Minimal

docker run \
  -e METRICS_STORAGE_TYPE=prometheus \
  jaegertracing/jaeger-query:{{< currentVersion >}}

All options

To view the full list of configuration options, you can run the following command:

docker run \
  -e METRICS_STORAGE_TYPE=prometheus \
  jaegertracing/jaeger-query:{{< currentVersion >}} \
  --help

TLS support

Jaeger supports TLS client to Prometheus server connections as long as you've configured
your Prometheus server correctly. You can
configure jaeger-query like so:

docker run \
  -e METRICS_STORAGE_TYPE=prometheus \
  -e PROMETHEUS_SERVER_URL=<...> \
  -e PROMETHEUS_TLS_ENABLED=true \
  -e PROMETHEUS_TLS_CA= \
  jaegertracing/jaeger-query:{{< currentVersion >}}

Aggregation Jobs for Service Dependencies Copy Chapter

Production deployments need an external process that aggregates data and creates dependency links between services. Project spark-dependencies is a Spark job which derives dependency links and writes them directly to the storage.

File: content/docs/v1/1.76/deployment/cli.md Copy Chapter

title: CLI flags aliases: [../cli] widescreen: true hasparent: true Copy Chapter

This is auto-generated documentation for CLI flags supported by Jaeger binaries.

• CLI flags for some binaries change depending on the SPAN_STORAGE_TYPE environment variable. Relevant variations are included below.
• Some binaries support commands (mostly informational), such as env, docs, version, and status. These commands are not included here.
• All parameters can be provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with the underscore _. For example, the value for the flag --cassandra.connections-per-host can be provided via the CASSANDRA_CONNECTIONS_PER_HOST environment variable.

{{< cli/tools-list >}}

File: content/docs/v1/1.76/deployment/frontend-ui.md Copy Chapter

title: Frontend/UI Configuration navtitle: Frontend/UI aliases: [../frontend-ui] hasparent: true weight: 7 Copy Chapter

Configuration Copy Chapter

Several aspects of the UI can be configured:

• The Dependencies section can be enabled / configured
• The Monitor tab (aka: Service Performance Monitoring) can be enabled / configured
• App analytics tracking can be enabled / configured (via Google Analytics or custom plugin)
• Additional menu options can be added to the global nav
• Search input limits can be configured

These options can be configured by a JSON configuration file. The --query.ui-config command line parameter of the query service must then be set to the path to the JSON file when the query service is started.

An example configuration file (see complete schema here):

{
  "dependencies": {
    "dagMaxNumServices": 200,
    "menuEnabled": true
  },
  "monitor": {
    "menuEnabled": true
  },
  "archiveEnabled": true,
  "tracking": {
    "gaID": "UA-000000-2",
    "trackErrors": true
  },
  "menu": [
    {
      "label": "About Jaeger",
      "items": [
        {
          "label": "GitHub",
          "url": "https://github.com/jaegertracing/jaeger"
        },
        {
          "label": "Docs",
          "url": "https://www.jaegertracing.io/docs/latest/"
        }
      ]
    }
  ],
  "search": {
    "maxLookback": {
      "label": "2 Days",
      "value": "2d"
    },
    "maxLimit": 1500
  },
  "linkPatterns": [{
    "type": "process",
    "key": "jaeger.version",
    "url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
    "text": "Information about Jaeger release #{jaeger.version}"
  },
  {
    "type": "tags",
    "key": "uniqueId",
    "url": "https://mykibana.com/uniqueId=#{uniqueId}&traceId=#{trace.traceID}",
    "text": "Redirect to kibana to view log"
  },
  {
    "type": "traces",
    "url": "https://my-logs.server?from=#{startTime | add -60000000 | epoch_micros_to_date_iso}&to=#{endTime | add 60000000 | epoch_micros_to_date_iso}'",
    "text": "Redirect to kibana to view log with formatted dates"
  }]
}

Dependencies Copy Section

dependencies.dagMaxNumServices defines the maximum number of services allowed before the DAG dependency view is disabled. Default: 200.

dependencies.menuEnabled enables (true) or disables (false) the dependencies menu button. Default: true.

Monitor Copy Section

monitor.menuEnabled enables (true) or disables (false) the Monitor menu button. Default: false.

Archive Support Copy Section

archiveEnabled enables (true) or disables (false) the archive traces button. Default: false. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.

App Analytics Tracking Copy Section

tracking.gaID defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-null value enables Google Analytics tracking. Default: null.

tracking.customWebAnalytics defines a factory function for a custom tracking plugin (only when using Javascript-form of UI configuration).

tracking.trackErrors enables (true) or disables (false) error tracking. Errors can only be tracked if a valid analytics tracker is configured. Default: true.

For additional details on app analytics see the tracking README in the UI repo.

Custom Menu Items Copy Section

menu allows additional links to be added to the global nav. The additional links are right-aligned.

In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:

{
  "label": "Some text here",
  "url": "https://example.com"
}

Links can either be members of the menu Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:

{
  "label": "Dropdown button",
  "items": [ ]
}

The items Array should contain one or more link configurations.

Search Input Limit Copy Section

The search.maxLimit configures the maximum results that the input let you search.

The search.maxLookback configures the maximum time before the present users can query for traces. The options in the Lookback dropdown greater than this value will not be shown.

Link Patterns Copy Section

The linkPatterns node can be used to create links from fields displayed in the Jaeger UI.

Both url and text can be defined as templates (i.e. using #{field-name}) where Jaeger UI will dynamically substitute values based on tags/logs/traces data.

For traces, the supported template fields are: duration, endTime, startTime, traceName and traceID.

Further, the trace template fields are available for substitution in process/logs/tags type when the trace template fields are prefixed with trace.. For example: trace.traceID, trace.startTime.

Formatting

In addition to interpolating fields into links, formatting functions can be used. The syntax is #{field | function} (eg #{endTime | epoch_micros_to_date_iso}'). Formatting functions can be chained together (eg #{startTime | add 60000000 | epoch_micros_to_date_iso}).

The available formatting functions and a description of their behavior are:

`epoch_micros_to_date_iso`

Format a date in microseconds since epoch to an ISO date time.

Example:
#{endTime | epoch_micros_to_date_iso}

`pad_start`

Pad the start of a string with a given character until the resulting string reaches the given length. Behavior is implemented by and thus matches javascript's String.padStart

Arguments:

• targetLength (integer) - Desired length of the string after padding
• padCharacter (string) - String to pad the start of the input string with

Example: #{traceID | pad_start 32 0}

`add`

Add a value to another number. Can be positive or negative.

Arguments:

• offset (integer) - Value to add to the input number

Example: #{startTime | add 1000000}

Embedded Mode Copy Chapter

Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of v0), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.

The embedded mode is induced and configured via URL query parameters.

To enter embedded mode, the uiEmbed=v0 query parameter and value must be added to the URL. For example, the following URL will show the trace with ID abc123 in embedded mode:

http://localhost:16686/trace/abc123?uiEmbed=v0

uiEmbed=v0 is required.

Further, for each supported page a button {{< rawhtml >}}{{< /rawhtml >}} is added that will open the non-embedded page in a new tab.

The following pages support embedded mode:

• Search Page
• Trace Page

Search Page Copy Section

To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with uiEmbed=v0.

For example:

http://localhost:16686/search?
    service=my-service&
    start=1543917759557000&
    end=1543921359557000&
    limit=20&
    lookback=1h&
    maxDuration&
    minDuration&
    uiEmbed=v0

Configuration options

The following query parameter can be used to configure the layout of the search page :

• uiSearchHideGraph=1 - disables the display of the scatter plot above the search results

http://localhost:16686/search?
    service=my-service&
    start=1543917759557000&
    end=1543921359557000&
    limit=20&
    lookback=1h&
    maxDuration&
    minDuration&
    uiEmbed=v0&
    uiSearchHideGraph=1

Trace Page Copy Section

To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with uiEmbed=v0.

For example:

http://localhost:16686/trace/{trace-id}?uiEmbed=v0

If we have navigated to this view from the search traces page we'll have a button to go back to the results page.

Configuration options

The following query parameters can be used to configure the layout of the trace page :

• uiTimelineCollapseTitle=1 causes the trace header to start out collapsed, which hides the summary and the minimap.

http://localhost:16686/trace/{trace-id}?
    uiEmbed=v0&
    uiTimelineCollapseTitle=1

• uiTimelineHideMinimap=1 removes the minimap, entirely, regardless of whether the trace header is expanded or not.

http://localhost:16686/trace/{trace-id}?
    uiEmbed=v0&
    uiTimelineHideMinimap=1

• uiTimelineHideSummary=1 - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.

http://localhost:16686/trace/{trace-id}?
    uiEmbed=v0&
    uiTimelineHideSummary=1

We can also combine the options:

http://localhost:16686/trace/{trace-id}?
    uiEmbed=v0&
    uiTimelineHideMinimap=1&
    uiTimelineHideSummary=1