## 1. Project Overview & Quickstart (jaegertracing/jaeger)
## File: README.md
[](https://stand-with-ukraine.pp.ua)
[
[![Unit Tests][ci-img]][ci]
[![Coverage Status][cov-img]][cov]
[![Project+Community stats][community-badge]][community-stats]
[![FOSSA Status][fossa-img]][fossa]
[![OpenSSF Scorecard][openssf-img]][openssf]
[![OpenSSF Best Practices][openssf-bp-img]][openssf-bp]
[![CLOMonitor][clomonitor-img]][clomonitor]
[![Artifact Hub][artifacthub-img]][artifacthub]
# Jaeger - a Distributed Tracing System
💥💥💥 Jaeger v2 is out! Read the [blog post](https://medium.com/jaegertracing/jaeger-v2-released-09a6033d1b10) and [try it out](https://www.jaegertracing.io/docs/latest/getting-started/).
## Quick Start
Get Jaeger running in seconds with Docker:
```bash
# Run Jaeger all-in-one (includes UI, collector, query, and in-memory storage)
docker run --rm --name jaeger \
-p 16686:16686 \
-p 4317:4317 \
-p 4318:4318 \
jaegertracing/jaeger:latest
# Access the UI at http://localhost:16686
# Send traces via OTLP: gRPC on port 4317, HTTP on port 4318
```
For production deployments and more options, see the [Getting Started Guide](https://www.jaegertracing.io/docs/latest/getting-started/).
## Architecture
```mermaid
graph TD
SDK["OpenTelemetry SDK"] --> |HTTP or gRPC| COLLECTOR
COLLECTOR["Jaeger Collector"] --> STORE[Storage]
COLLECTOR --> |gRPC| PLUGIN[Storage Plugin]
COLLECTOR --> |gRPC/sampling| SDK
PLUGIN --> STORE
QUERY[Jaeger Query Service] --> STORE
QUERY --> |gRPC| PLUGIN
UI[Jaeger UI] --> |HTTP| QUERY
subgraph Application Host
subgraph User Application
SDK
end
end
```
Jaeger is a distributed tracing platform created by [Uber Technologies](https://eng.uber.com/distributed-tracing/) and donated to [Cloud Native Computing Foundation](https://cncf.io).
See Jaeger [documentation][doc] for getting started, operational details, and other information.
Jaeger is hosted by the [Cloud Native Computing Foundation](https://cncf.io) (CNCF) as the 7th top-level project, graduated in October 2019. See the CNCF [Jaeger incubation announcement](https://www.cncf.io/blog/2017/09/13/cncf-hosts-jaeger/) and [Jaeger graduation announcement](https://www.cncf.io/announcement/2019/10/31/cloud-native-computing-foundation-announces-jaeger-graduation/).
## Get Involved
Jaeger is an open source project with open governance. We welcome contributions from the community, and we would love your help to improve and extend the project. Here are [some ideas](https://www.jaegertracing.io/get-involved/) for how to get involved. Many of them do not even require any coding.
## Version Compatibility Guarantees
Since Jaeger uses many components from the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector/) we try to maintain configuration compatibility between Jaeger releases. Occasionally, configuration options in Jaeger (or in Jaeger v1 CLI flags) can be deprecated due to usability improvements, new functionality, or changes in our dependencies.
In such situations, developers introducing the deprecation are required to follow [these guidelines](./CONTRIBUTING.md#deprecating-cli-flags).
In short, for a deprecated configuration option, you should expect to see the following message in the documentation or release notes:
```
(deprecated, will be removed after yyyy-mm-dd or in release vX.Y.Z, whichever is later)
```
A grace period of at least **3 months** or **two minor version bumps** (whichever is later) from the first release
containing the deprecation notice will be provided before the deprecated configuration option _can_ be deleted.
For example, consider a scenario where v2.0.0 is released on 01-Sep-2024 containing a deprecation notice for a configuration option.
This configuration option will remain in a deprecated state until the later of 01-Dec-2024 or v2.2.0 where it _can_ be removed on or after either of those events.
It may remain deprecated for longer than the aforementioned grace period.
## Go Version Compatibility Guarantees
The Jaeger project attempts to track the currently supported versions of Go, as [defined by the Go team](https://go.dev/doc/devel/release#policy).
Removing support for an unsupported Go version is not considered a breaking change.
Starting with the release of Go 1.21, support for Go versions will be updated as follows:
1. Soon after the release of a new Go minor version `N`, updates will be made to the build and tests steps to accommodate the latest Go minor version.
2. Soon after the release of a new Go minor version `N`, support for Go version `N-1` will be removed and version `N` will become the minimum required version.
Note: All importable code has been moved to internal packages, so there is no need to maintain backward compatibility with older compilers (previously version `N-1` was used).
## Storage Backend Version Support Policy
This policy covers storage backend versions only while they are reasonable for the project to test, maintain, and recommend for production use. Policy coverage is distinct from versions that may remain technically compatible with Jaeger.
For backends with a published upstream lifecycle, Jaeger bases its policy on the backend project's or vendor's supported-version or end-of-life policy, with the exact coverage defined below. Once upstream support ends, Jaeger may remove CI coverage, stop proactively fixing backend-specific issues for that version, and require users to upgrade before troubleshooting.
For backends without a clear upstream lifecycle, Jaeger supports the current stable major version and the immediately preceding major version. Older versions are best-effort only and may be removed when they block development, security fixes, dependency updates, or CI reliability.
Backend-specific policy:
* Elasticsearch: follow Elastic's [Product and Version End of Life Policy](https://www.elastic.co/support/eol).
* OpenSearch: follow the OpenSearch [Release Schedule and Maintenance Policy](https://opensearch.org/releases/). In practice, this means the current major version and the previous major version while they remain maintained.
* Cassandra: follow Apache Cassandra's maintained release lines, as published on the [Cassandra download page](https://cassandra.apache.org/_/download.html), with the Jaeger support target being the current major version and the previous major version.
* ClickHouse: follow ClickHouse's [release and backport policy](https://clickhouse.com/docs/development/backports) and production guidance for [stable and LTS releases](https://clickhouse.com/docs/faq/operations/production). Jaeger treats LTS releases as its support lines and covers the current and previous LTS releases, which ClickHouse supports simultaneously for at least 12 months. Monthly stable releases may remain compatible or appear in CI but are not covered by this policy.
Support means backend versions are in scope for proactive compatibility fixes. CI coverage is not the definitive support matrix: it may test only a subset of supported versions and may retain older versions on a best-effort basis. CI coverage alone does not make a backend version supported. Jaeger does not provide support for the backend product itself, vendor-specific distributions, or versions outside the upstream maintenance window.
As of July 6, 2026, this policy means:
| Backend | Jaeger policy basis | Versions covered by policy | Versions not covered |
| --- | --- | --- | --- |
| Elasticsearch | Elastic EOL policy | `9.x`; `8.19.x` until Elastic's published 8.x maintenance window ends | `8.18.x` and earlier |
| OpenSearch | OpenSearch maintenance policy | `3.x`; `2.x`, with `2.19.x` as the currently maintained 2.x line | `1.x` and earlier |
| Cassandra | Current major plus previous major, using maintained Apache release lines | `5.0.x`; maintained `4.x` lines, currently `4.1.x` and `4.0.x` | `3.x` and earlier |
| ClickHouse | Current and previous ClickHouse LTS releases | `26.3 LTS` and `25.8 LTS` | Monthly stable releases and older LTS branches |
This table is informational and should be refreshed when upstream policies or release lines change. Versions listed as not covered may still appear in CI on a best-effort basis; their presence in CI does not extend the support policy.
## Related Repositories
### Components
* [UI](https://github.com/jaegertracing/jaeger-ui)
* [Data model](https://github.com/jaegertracing/jaeger-idl)
### Documentation
* Published: https://www.jaegertracing.io/docs/
* Source: https://github.com/jaegertracing/documentation
## Building From Source
See [CONTRIBUTING](./CONTRIBUTING.md).
## Contributing
See [CONTRIBUTING](./CONTRIBUTING.md).
Thanks to all the people who already contributed!
### Maintainers
Rules for becoming a maintainer are defined in the [GOVERNANCE](./GOVERNANCE.md) document.
The official maintainers of the Jaeger project are listed in the [MAINTAINERS](./MAINTAINERS.md) file.
Please use `@jaegertracing/jaeger-maintainers` to tag them on issues / PRs.
Some repositories under [jaegertracing](https://github.com/jaegertracing) org have additional maintainers.
## Project Status Meetings
The Jaeger maintainers and contributors meet regularly on a video call. Everyone is welcome to join, including end users. For meeting details, see https://www.jaegertracing.io/get-in-touch/.
## Roadmap
See https://www.jaegertracing.io/docs/roadmap/
## Get in Touch
Have questions, suggestions, bug reports? Reach the project community via these channels:
* [Slack chat room `#jaeger`][slack] (need to join [CNCF Slack][slack-join] for the first time)
* [`jaeger-tracing` mail group](https://groups.google.com/forum/#!forum/jaeger-tracing)
* GitHub [issues](https://github.com/jaegertracing/jaeger/issues) and [discussions](https://github.com/jaegertracing/jaeger/discussions)
## Security
Third-party security audits of Jaeger are available in https://github.com/jaegertracing/security-audits. Please see [Issue #1718](https://github.com/jaegertracing/jaeger/issues/1718) for the summary of available security mechanisms in Jaeger.
## Adopters
Jaeger as a product consists of multiple components. We want to support different types of users,
whether they are only using our instrumentation libraries or full end to end Jaeger installation,
whether it runs in production or you use it to troubleshoot issues in development.
Please see [ADOPTERS.md](./ADOPTERS.md) for some of the organizations using Jaeger today.
If you would like to add your organization to the list, please comment on our
[survey issue](https://github.com/jaegertracing/jaeger/issues/207).
## Sponsors
The Jaeger project owes its success in open source largely to the Cloud Native Computing Foundation (CNCF), our primary supporter. We deeply appreciate their vital support. Furthermore, we are grateful to Uber for their initial, project-launching donation, and for the continuous contributions of software and infrastructure from 1Password, Codecov.io, Dosu, GitHub, Google Analytics, Netlify, and Oracle Cloud Infrastructure. Thank you for your generous support.
## License
Copyright (c) The Jaeger Authors. [Apache 2.0 License](./LICENSE).
[doc]: https://jaegertracing.io/docs/
[ci-img]: https://github.com/jaegertracing/jaeger/actions/workflows/ci-unit-tests.yml/badge.svg?branch=main
[ci]: https://github.com/jaegertracing/jaeger/actions/workflows/ci-unit-tests.yml?query=branch%3Amain
[cov-img]: https://codecov.io/gh/jaegertracing/jaeger/branch/main/graph/badge.svg
[cov]: https://codecov.io/gh/jaegertracing/jaeger/branch/main/
[fossa-img]: https://app.fossa.com/api/projects/git%2Bgithub.com%2Fjaegertracing%2Fjaeger.svg?type=shield
[fossa]: https://app.fossa.io/projects/git%2Bgithub.com%2Fjaegertracing%2Fjaeger?ref=badge_shield
[openssf-img]: https://api.securityscorecards.dev/projects/github.com/jaegertracing/jaeger/badge
[openssf]: https://securityscorecards.dev/viewer/?uri=github.com/jaegertracing/jaeger
[openssf-bp-img]: https://www.bestpractices.dev/projects/1273/badge
[openssf-bp]: https://www.bestpractices.dev/projects/1273
[clomonitor-img]: https://img.shields.io/endpoint?url=https://clomonitor.io/api/projects/cncf/jaeger/badge
[clomonitor]: https://clomonitor.io/projects/cncf/jaeger
[artifacthub-img]: https://img.shields.io/endpoint?url=https://artifacthub.io/badge/repository/jaegertracing
[artifacthub]: https://artifacthub.io/packages/search?repo=jaegertracing
[community-badge]: https://img.shields.io/badge/Project+Community-stats-blue.svg
[community-stats]: https://all.devstats.cncf.io/d/54/project-health?orgId=1&var-repogroup_name=Jaeger
[hotrod-tutorial]: https://medium.com/jaegertracing/take-jaeger-for-a-hotrod-ride-233cf43e46c2
[slack]: https://cloud-native.slack.com/archives/CGG7NFUJ3
[slack-join]: https://slack.cncf.io
[slack-img]: https://img.shields.io/badge/slack-join%20chat%20%E2%86%92-brightgreen?logo=slack
---
## File: cmd/tracegen/README.md
# tracegen
`tracegen` is a utility that can generate a steady flow of simple traces useful for performance tuning.
Traces are produced concurrently from one or more worker goroutines. Run with `-h` to see all cli flags.
The binary is available from the Releases page, as well as a Docker image:
```sh
$ docker run jaegertracing/jaeger-tracegen -service abcd -traces 10
```
The generator can be configured to export traces in different formats, via `-exporter` flag.
By default, the exporters send data to `localhost`. If running in a container, this refers
to the networking namespace of the container itself, so to export to another container,
the exporters need to be provided with appropriate location.
OTLP exporter accepts configuration via environment variables.
For more information about configuring OTLP exporter, see [OpenTelemetry Protocol Exporter](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/protocol/exporter.md).
See example in the included [docker-compose](./docker-compose.yml) file.
---
## File: cmd/jaeger/docs/migration/all-in-one-metrics.md
# ALL-IN-ONE METRICS
### Combined Metrics
| V1 Metric | V1 Labels | V2 Metric | V2 Labels |
|-----------|---------------|-----------|---------------|
| jaeger_query_latency | operation, result | jaeger_query_latency | operation, result |
| jaeger_query_responses | operation | jaeger_query_responses | operation |
| jaeger_query_requests_total | operation, result | jaeger_query_requests_total | operation, result |
| go_gc_duration_seconds | N/A | N/A | N/A |
| go_goroutines | N/A | N/A | N/A |
| go_info | version | N/A | N/A |
| go_memstats_alloc_bytes | N/A | N/A | N/A |
| go_memstats_alloc_bytes_total | N/A | N/A | N/A |
| go_memstats_buck_hash_sys_bytes | N/A | N/A | N/A |
| go_memstats_frees_total | N/A | N/A | N/A |
| go_memstats_gc_sys_bytes | N/A | N/A | N/A |
| go_memstats_heap_alloc_bytes | N/A | N/A | N/A |
| go_memstats_heap_idle_bytes | N/A | N/A | N/A |
| go_memstats_heap_inuse_bytes | N/A | N/A | N/A |
| go_memstats_heap_objects | N/A | N/A | N/A |
| go_memstats_heap_released_bytes | N/A | N/A | N/A |
| go_memstats_heap_sys_bytes | N/A | N/A | N/A |
| go_memstats_last_gc_time_seconds | N/A | N/A | N/A |
| go_memstats_lookups_total | N/A | N/A | N/A |
| go_memstats_mallocs_total | N/A | N/A | N/A |
| go_memstats_mcache_inuse_bytes | N/A | N/A | N/A |
| go_memstats_mcache_sys_bytes | N/A | N/A | N/A |
| go_memstats_mspan_inuse_bytes | N/A | N/A | N/A |
| go_memstats_mspan_sys_bytes | N/A | N/A | N/A |
| go_memstats_next_gc_bytes | N/A | N/A | N/A |
| go_memstats_other_sys_bytes | N/A | N/A | N/A |
| go_memstats_stack_inuse_bytes | N/A | N/A | N/A |
| go_memstats_stack_sys_bytes | N/A | N/A | N/A |
| go_memstats_sys_bytes | N/A | N/A | N/A |
| go_threads | N/A | N/A | N/A |
| jaeger_build_info | build_date, revision, version | N/A | N/A |
| jaeger_collector_batch_size | host | N/A | N/A |
| jaeger_collector_http_request_duration | method, path, status | N/A | N/A |
| jaeger_collector_http_server_errors_total | source, status | N/A | N/A |
| jaeger_collector_http_server_requests_total | type | N/A | N/A |
| jaeger_collector_in_queue_latency | host | N/A | N/A |
| jaeger_collector_queue_capacity | host | N/A | N/A |
| jaeger_collector_queue_length | host | N/A | N/A |
| jaeger_collector_save_latency | host | N/A | N/A |
| jaeger_collector_spans_bytes | host | N/A | N/A |
| jaeger_collector_spans_dropped_total | host | N/A | N/A |
| jaeger_collector_spans_received_total | debug, format, svc, transport | N/A | N/A |
| jaeger_collector_spans_rejected_total | debug, format, svc, transport | N/A | N/A |
| jaeger_collector_spans_saved_by_svc_total | debug, result, svc | N/A | N/A |
| jaeger_collector_spans_serviceNames | host | N/A | N/A |
| jaeger_collector_traces_received_total | debug, format, sampler_type, svc, transport | N/A | N/A |
| jaeger_collector_traces_rejected_total | debug, format, sampler_type, svc, transport | N/A | N/A |
| jaeger_collector_traces_saved_by_svc_total | debug, result, sampler_type, svc | N/A | N/A |
| process_cpu_seconds_total | N/A | N/A | N/A |
| process_max_fds | N/A | N/A | N/A |
| process_open_fds | N/A | N/A | N/A |
| process_resident_memory_bytes | N/A | N/A | N/A |
| process_start_time_seconds | N/A | N/A | N/A |
| process_virtual_memory_bytes | N/A | N/A | N/A |
| process_virtual_memory_max_bytes | N/A | N/A | N/A |
| N/A | N/A | exporter_send_failed_spans | exporter, service_instance_id, service_name, service_version |
| N/A | N/A | exporter_sent_spans | exporter, service_instance_id, service_name, service_version |
| N/A | N/A | process_cpu_seconds | service_instance_id, service_name, service_version |
| N/A | N/A | process_memory_rss | service_instance_id, service_name, service_version |
| N/A | N/A | process_runtime_heap_alloc_bytes | service_instance_id, service_name, service_version |
| N/A | N/A | process_runtime_total_alloc_bytes | service_instance_id, service_name, service_version |
| N/A | N/A | process_runtime_total_sys_memory_bytes | service_instance_id, service_name, service_version |
| N/A | N/A | process_uptime | service_instance_id, service_name, service_version |
| N/A | N/A | processor_batch_batch_send_size | processor, service_instance_id, service_name, service_version |
| N/A | N/A | processor_batch_batch_send_size_bytes | processor, service_instance_id, service_name, service_version |
| N/A | N/A | processor_batch_metadata_cardinality | processor, service_instance_id, service_name, service_version |
| N/A | N/A | processor_batch_timeout_trigger_send | processor, service_instance_id, service_name, service_version |
| N/A | N/A | receiver_accepted_spans | receiver, service_instance_id, service_name, service_version, transport |
| N/A | N/A | receiver_refused_spans | receiver, service_instance_id, service_name, service_version, transport |
| N/A | N/A | rpc_server_duration | rpc_grpc_status_code, rpc_method, rpc_service, rpc_system, service_instance_id, service_name, service_version |
| N/A | N/A | rpc_server_request_size | rpc_method, rpc_service, rpc_system, service_instance_id, service_name, service_version |
| N/A | N/A | rpc_server_requests_per_rpc | rpc_grpc_status_code, rpc_method, rpc_service, rpc_system, service_instance_id, service_name, service_version |
| N/A | N/A | rpc_server_response_size | rpc_method, rpc_service, rpc_system, service_instance_id, service_name, service_version |
| N/A | N/A | rpc_server_responses_per_rpc | rpc_grpc_status_code, rpc_method, rpc_service, rpc_system, service_instance_id, service_name, service_version |
| N/A | N/A | target_info | service_instance_id, service_name, service_version |
### Equivalent Metrics
| V1 Metric | V1 Labels | V2 Metric | V2 Labels |
|-----------|---------------|-----------|---------------|
| jaeger_collector_spans_rejected_total | debug, format, svc, transport | receiver_refused_spans | receiver, service_instance_id, service_name, service_version, transport |
| jaeger_build_info | build_date, revision, version | target_info | service_instance_id, service_name, service_version |
---
## File: cmd/jaeger/docs/migration/badger-metrics.md
# BADGER METRICS
### Combined Metrics
| V1 Metric | V1 Parameters | V2 Metric | V2 Parameters |
| ------------------------------------------ | ------------- | ---------------------------------------- | ------------- |
| jaeger_badger_compaction_current_num_lsm | N/A | jaeger_badger_compaction_current_num_lsm | N/A |
| jaeger_badger_get_num_memtable | N/A | jaeger_badger_get_num_memtable | N/A |
| jaeger_badger_get_num_user | N/A | jaeger_badger_get_num_user | N/A |
| jaeger_badger_get_with_result_num_user | N/A | jaeger_badger_get_with_result_num_user | N/A |
| jaeger_badger_iterator_num_user | N/A | jaeger_badger_iterator_num_user | N/A |
| jaeger_badger_put_num_user | N/A | jaeger_badger_put_num_user | N/A |
| jaeger_badger_read_bytes_lsm | N/A | jaeger_badger_read_bytes_lsm | N/A |
| jaeger_badger_read_bytes_vlog | N/A | jaeger_badger_read_bytes_vlog | N/A |
| jaeger_badger_read_num_vlog | N/A | jaeger_badger_read_num_vlog | N/A |
| jaeger_badger_size_bytes_lsm | N/A | jaeger_badger_size_bytes_lsm | N/A |
| jaeger_badger_size_bytes_vlog | N/A | jaeger_badger_size_bytes_vlog | N/A |
| jaeger_badger_write_bytes_l0 | N/A | jaeger_badger_write_bytes_l0 | N/A |
| jaeger_badger_write_bytes_user | N/A | jaeger_badger_write_bytes_user | N/A |
| jaeger_badger_write_bytes_vlog | N/A | jaeger_badger_write_bytes_vlog | N/A |
| jaeger_badger_write_num_vlog | N/A | jaeger_badger_write_num_vlog | N/A |
| jaeger_badger_write_pending_num_memtable | N/A | jaeger_badger_write_pending_num_memtable | N/A |
| jaeger_badger_key_log_bytes_available | N/A | N/A | N/A |
| jaeger_badger_storage_maintenance_last_run | N/A | N/A | N/A |
| jaeger_badger_storage_valueloggc_last_run | N/A | N/A | N/A |
| jaeger_badger_value_log_bytes_available | N/A | N/A | N/A |
### Equivalent Metrics
| V1 Metric | V1 Parameters | V2 Metric | V2 Parameters |
| ------------------------------------- | ------------------------------ | ---------------------- | ----------------------------------------------------------------------- |
| jaeger_collector_spans_rejected_total | debug, format, svc, transport | receiver_refused_spans | receiver, service_instance_id, service_name, service_version, transport |
| jaeger_build_info | build_date, revision, version | target_info | service_instance_id, service_name, service_version |
---
## File: cmd/jaeger/docs/migration/cassandra-metrics.md
# CASSANDRA METRICS
### Combined Metrics
| V1 Metric | V1 Parameters | V2 Metric | V2 Parameters |
|-----------|---------------|-----------|---------------|
| jaeger_cassandra_attempts_total | table | jaeger_cassandra_attempts_total | table |
| jaeger_cassandra_errors_total | table | jaeger_cassandra_errors_total | table |
| jaeger_cassandra_inserts_total | table | jaeger_cassandra_inserts_total | table |
| jaeger_cassandra_latency_err | table | jaeger_cassandra_latency_err | table |
| jaeger_cassandra_latency_ok | table | jaeger_cassandra_latency_ok | table |
| jaeger_cassandra_read_attempts_total | table | jaeger_cassandra_read_attempts_total | table |
| jaeger_cassandra_read_errors_total | table | jaeger_cassandra_read_errors_total | table |
| jaeger_cassandra_read_inserts_total | table | jaeger_cassandra_read_inserts_total | table |
| jaeger_cassandra_read_latency_err | table | jaeger_cassandra_read_latency_err | table |
| jaeger_cassandra_read_latency_ok | table | jaeger_cassandra_read_latency_ok | table |
| jaeger_cassandra_tag_index_skipped_total | N/A | jaeger_cassandra_tag_index_skipped_total | N/A |
### Equivalent Metrics
| V1 Metric | V1 Parameters | V2 Metric | V2 Parameters |
|-----------|---------------|-----------|---------------|
| jaeger_collector_spans_rejected_total | debug, format, svc, transport | receiver_refused_spans | receiver, service_instance_id, service_name, service_version, transport |
| jaeger_build_info | build_date, revision, version | target_info | service_instance_id, service_name, service_version |
---
## File: cmd/jaeger/docs/migration/elasticsearch-metrics.md
# ELASTICSEARCH METRICS
### Combined Metrics
| V1 Metric | V1 Parameters | V2 Metric | V2 Parameters |
|-----------|---------------|-----------|---------------|
| jaeger_bulk_index_attempts_total | N/A | jaeger_bulk_index_attempts_total | N/A |
| jaeger_bulk_index_errors_total | N/A | jaeger_bulk_index_errors_total | N/A |
| jaeger_bulk_index_inserts_total | N/A | jaeger_bulk_index_inserts_total | N/A |
| jaeger_bulk_index_latency_err | N/A | jaeger_bulk_index_latency_err | N/A |
| jaeger_bulk_index_latency_ok | N/A | jaeger_bulk_index_latency_ok | N/A |
| jaeger_index_create_attempts_total | N/A | jaeger_index_create_attempts_total | N/A |
| jaeger_index_create_errors_total | N/A | jaeger_index_create_errors_total | N/A |
| jaeger_index_create_inserts_total | N/A | jaeger_index_create_inserts_total | N/A |
| jaeger_index_create_latency_err | N/A | jaeger_index_create_latency_err | N/A |
| jaeger_index_create_latency_ok | N/A | jaeger_index_create_latency_ok | N/A |
### Equivalent Metrics
| V1 Metric | V1 Parameters | V2 Metric | V2 Parameters |
|-----------|---------------|-----------|---------------|
| jaeger_collector_spans_rejected_total | debug, format, svc, transport | receiver_refused_spans | receiver, service_instance_id, service_name, service_version, transport |
| jaeger_build_info | build_date, revision, version | target_info | service_instance_id, service_name, service_version |
---
## File: cmd/jaeger/docs/migration/opensearch-metrics.md
# OPENSEARCH METRICS
### Combined Metrics
| V1 Metric | V1 Parameters | V2 Metric | V2 Parameters |
|-----------|---------------|-----------|---------------|
| jaeger_bulk_index_attempts_total | N/A | jaeger_bulk_index_attempts_total | N/A |
| jaeger_bulk_index_errors_total | N/A | jaeger_bulk_index_errors_total | N/A |
| jaeger_bulk_index_inserts_total | N/A | jaeger_bulk_index_inserts_total | N/A |
| jaeger_bulk_index_latency_err | N/A | jaeger_bulk_index_latency_err | N/A |
| jaeger_bulk_index_latency_ok | N/A | jaeger_bulk_index_latency_ok | N/A |
| jaeger_index_create_attempts_total | N/A | jaeger_index_create_attempts_total | N/A |
| jaeger_index_create_errors_total | N/A | jaeger_index_create_errors_total | N/A |
| jaeger_index_create_inserts_total | N/A | jaeger_index_create_inserts_total | N/A |
| jaeger_index_create_latency_err | N/A | jaeger_index_create_latency_err | N/A |
| jaeger_index_create_latency_ok | N/A | jaeger_index_create_latency_ok | N/A |
### Equivalent Metrics
| V1 Metric | V1 Parameters | V2 Metric | V2 Parameters |
|-----------|---------------|-----------|---------------|
| jaeger_collector_spans_rejected_total | debug, format, svc, transport | receiver_refused_spans | receiver, service_instance_id, service_name, service_version, transport |
| jaeger_build_info | build_date, revision, version | target_info | service_instance_id, service_name, service_version |
---
## File: cmd/jaeger/internal/exporters/storageexporter/README.md
# jaeger_storage_exporter
This module implements `exporter.Traces` and writes spans into Jaeger native `spanstore.SpanWriter`, that it obtains from the [jaeger_storage](../../extension/jaegerstorage/) extension. This is primarily needed to wire a memory storage into the exporter pipeline (used for all-in-one), but the design of the exporter is such that it can do this for any V1 storage implementation.
## Configuration
### Basic Configuration
```yaml
exporters:
jaeger_storage_exporter:
trace_storage: memstore
extensions:
jaeger_storage:
memory:
memstore:
max_traces: 100000
```
### Retry Configuration
The exporter supports configurable retry behavior for failed exports. By default, retry is disabled. When enabled, the following default settings are used:
- `enabled`: false (disabled by default)
- `initial_interval`: 5s
- `randomization_factor`: 0.5
- `multiplier`: 1.5
- `max_interval`: 30s
- `max_elapsed_time`: 5m
To enable and customize the retry behavior:
```yaml
exporters:
jaeger_storage_exporter:
trace_storage: memstore
retry_on_failure:
enabled: true # Explicitly enable retries
initial_interval: 10s
max_interval: 1m
max_elapsed_time: 10m
queue:
enabled: true
num_consumers: 10
queue_size: 1000
```
---
## File: cmd/jaeger/internal/extension/remotestorage/README.md
# Remote Storage Backend Extension
The Remote Storage Backend Extension exposes any Jaeger-supported storage backend through a gRPC API by
implementing the gRPC Storage API. The gRPC API currently consists of the following services:
* [TraceReader](https://github.com/jaegertracing/jaeger-idl/blob/main/proto/storage/v2/trace_storage.proto)
* [DependencyReader](https://github.com/jaegertracing/jaeger-idl/blob/main/proto/storage/v2/dependency_storage.proto)
## Overview
This extension allows you to make Jaeger storage backends accessible remotely over gRPC.
For example, you can expose the in-memory storage backend like this:
```yaml
extensions:
jaeger_storage:
backends:
memory-storage:
memory:
max_traces: 100000
remote_storage:
endpoint: localhost:17271
storage: memory-storage
```
---
## File: cmd/jaeger/internal/extension/remotesampling/README.md
# Remote Sampling
Placeholder
```
/* Detailed source-code truncated for AI context efficiency. */
```
## 2. Official Technical Reference & Guides (jaegertracing/documentation)
## File: README.md
[![Netlify Status][netlify-img]][netlify]
# Jaeger website
This repo houses all the assets used to build the Jaeger website, available at .
The site is built with [Hugo](https://gohugo.io/) and hosted by [Netlify](https://www.netlify.com/).
## Contributing to the site
We strongly encourage you to contribute to this site! For more information, see the [contributing](CONTRIBUTING.md) guide.
## Publishing the site
The site is published automatically by [Netlify](https://www.netlify.com/) whenever changes are merged to the `main` branch. The site cannot be published in an ad-hoc way (e.g. through a `make` command or script in the repo).
## Publishing new Jaeger version
Please refer to [RELEASE.md](./RELEASE.md) for instructions on how to release new version of documentation.
## License
[Apache 2.0 License](./LICENSE).
[netlify-img]: https://api.netlify.com/api/v1/badges/d2b1a1ea-f454-4ba8-990c-cc469c959556/deploy-status
[netlify]: https://app.netlify.com/sites/jaegertracing/deploys
[htmltest]: https://github.com/wjdp/htmltest
---
## File: content/announcements/_index.md
---
title: Announcements
toc_hide: true
cascade:
type: docs
params:
hide_feedback: true
redirects: [{ from: '*', to: '?' }]
---
---
## File: content/announcements/kubecon-na.md
---
title: KubeCon North America 2025
date: 2025-10-01
expiryDate: 2025-11-13
---
Join us in Atlanta, Georgia for [**KubeCon +
CloudNativeCon North America 2025**][blog], November 10 - 13, 2025
[blog]: https://opentelemetry.io/blog/2025/kubecon-na/
---
## File: content/announcements/ukraine.md
---
title: We stand with Ukraine
weight: -1
cSpell:ignore: ukraine
body_class: ukraine-banner
---
![Ukraine flag][] We stand with our friends and colleagues in Ukraine. To assist
in their time of need, visit [Support Ukraine][].
[Ukraine flag]: /img/ukraine_flag.svg
[Support Ukraine]: https://war.ukraine.ua/support-ukraine/
---
## File: content/docs/v1/1.76/architecture/_index.md
---
title: Architecture
weight: 3
children:
- title: APIs
url: apis
- title: Sampling
url: sampling
---
## Terminology
Jaeger represents tracing data in a data model inspired by the [OpenTracing Specification](https://github.com/opentracing/specification/blob/master/specification.md). The data model is logically very similar to [OpenTelemetry Traces](https://opentelemetry.io/docs/concepts/signals/traces/), with some naming differences:
| Jaeger | OpenTelemetry | Notes |
| -------------------- | --------------- | ----------------------------------------------------------------------- |
| Tags | Attributes | Both support typed values, but nested tags are not supported in Jaeger. |
| Span Logs | Span Events | Point-in-time events on the span recorded in a structured form. |
| Span References | Span Links | Jaeger's Span References have a required type (`child-of` or `follows-from`) and always refer to predecessor spans; OpenTelemetry's Span Links have no type, but allow attributes. |
| Process | Resource | A struct describing the entity that produces the telemetry. |
### Span
A **span** represents a logical unit of work that has an operation name, the start time of the operation, and the duration. Spans may be nested and ordered to model causal relationships.
[](/img/spans-traces.png)
### Trace
A **trace** represents the data or execution path through the system. It can be thought of as a directed acyclic graph of spans.
### Baggage
**Baggage** is arbitrary user-defined metadata (key-value pairs) that can be attached to distributed context and propagated by the tracing SDKs. See [W3C Baggage](https://www.w3.org/TR/baggage/) for more information.
## Architecture
Jaeger can be deployed either as an **all-in-one** binary, where all Jaeger backend components
run in a single process, or as a scalable distributed system. There are two main deployment options discussed below.
### Direct to storage
In this deployment the collectors receive the data from traced applications and write it directly to storage. The storage must be able to handle both average and peak traffic. Collectors use an in-memory queue to smooth short-term traffic peaks, but a sustained traffic spike may result in dropped data if the storage is not able to keep up.
Collectors are able to centrally serve sampling configuration to the SDKs, known as [remote sampling mode](./sampling/#remote-sampling). They can also enable automatic sampling configuration calculation, known as [adaptive sampling](./sampling/#adaptive-sampling).
[](/img/architecture-v1-2023.png)
### Via Kafka
To prevent data loss between collectors and storage, Kafka can be used as an intermediary, persistent queue. An additional component, **jaeger-ingester**, needs to be deployed to read data from Kafka and save to the database. Multiple **jaeger-ingester**s can be deployed to scale up ingestion; they will automatically partition the load across them.
[](/img/architecture-v2-2023.png)
### With OpenTelemetry Collector
You **do not need to use OpenTelemetry Collector**, because **jaeger-collector** can receive OpenTelemetry data directly from the OpenTelemetry SDKs (using OTLP exporters). However, if you already use the OpenTelemetry Collectors, such as for gathering other types of telemetry or for pre-processing / enriching the tracing data, it __can be placed between__ the SDKs and **jaeger-collector**'s. The OpenTelemetry Collectors can be run as an application sidecar, as a host agent / daemon, or as a central cluster.
The OpenTelemetry Collector supports Jaeger's Remote Sampling protocol and can either serve static configurations from config files directly, or proxy the requests to the Jaeger backend (e.g., when using adaptive sampling).
[](/img/architecture-otel.png)
#### OpenTelemetry Collector as a sidecar / host agent
Benefits:
* The SDK configuration is simplified as both trace export endpoint and sampling config endpoint can point to a local host and not worry about discovering where those services run remotely.
* Collector may provide data enrichment by adding environment information, like k8s pod name.
* Resource usage for data enrichment can be distributed across all application hosts.
Downsides:
* An extra layer of marshaling/unmarshaling the data.
#### OpenTelemetry Collector as a remote cluster
Benefits:
* Sharding capabilities, e.g., when using [tail-based sampling](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md).
Downsides:
* An extra layer of marshaling/unmarshaling the data.
## Components
This section details the constituent parts of Jaeger and how they relate to each other. It is arranged by the order in which spans from your application interact with them.
### Tracing SDKs
In order to generate tracing data, the applications must be instrumented with tracing SDK such as [OpenTelemetry SDKs](https://opentelemetry.io). An instrumented application creates spans when receiving new requests and attaches context information (trace id, span id, and baggage) to outgoing requests. Only the ids and baggage are propagated with requests; all other profiling data, like operation name, timing, tags and logs, is not propagated. Instead, it is exported out of process to the Jaeger backend asynchronously, in the background.
[](/img/context-prop-2023.png)
There are many ways to instrument an application:
* manually, using the tracing APIs directly,
* relying on instrumentation already created for a variety of existing open source frameworks,
* automatically, via byte code manipulation, monkey-patching, eBPF, and similar techniques.
Instrumentation typically should not depend on specific tracing SDKs, but only on abstract tracing APIs like the OpenTelemetry API. The tracing SDKs implement the tracing APIs and take care of data export.
The instrumentation is designed to be always on in production. To minimize overhead, the SDKs employ various sampling strategies. When a trace is sampled, the profiling span data is captured and transmitted to the Jaeger backend. When a trace is not sampled, no profiling data is collected at all, and the calls to the tracing API are short-circuited to incur a minimal amount of overhead. For more information, please refer to the [Sampling](./sampling/) page.
### Collector
**jaeger-collector** receives traces, runs them through a processing pipeline for validation and clean-up/enrichment, and stores them in a storage backend. Jaeger comes with built-in support for several storage backends (see [Deployment](../deployment/)), as well as extensible plugin framework for implementing custom storage plugins.
### Query
**jaeger-query** is a service that exposes the [APIs](./apis/) for retrieving traces from storage and hosts a Web UI for searching and analyzing traces.
### Ingester
**jaeger-ingester** is a service that reads traces from Kafka and writes them to a storage backend. Effectively, it is a stripped-down version of the Jaeger collector that supports Kafka as the only input protocol.
---
## File: content/docs/v1/1.76/architecture/apis.md
---
title: APIs
aliases: [../apis]
hasparent: true
---
Jaeger components implement various APIs for saving or retrieving trace data.
The following labels are used to describe API compatibility guarantees.
* **stable** - the API guarantees backwards compatibility. If breaking changes are going to be made in the future, they will result in a new API version, e.g. `/api/v2` URL prefix or a different namespace in the IDL.
* **internal** - the APIs are intended for internal communications between Jaeger components and are not recommended for use by external components.
* **deprecated** - the APIs that are only maintained for legacy reasons and will be phased out in the future.
Since Jaeger v1.32, **jaeger-collector** and **jaeger-query** Service ports that serve gRPC endpoints enable [gRPC reflection][grpc-reflection]. Unfortunately, the internally used `gogo/protobuf` has a [compatibility issue][gogo-reflection] with the official `golang/protobuf`, and as a result only the `list` reflection command is currently working properly.
## Span reporting APIs
**jaeger-collector** is the component of the Jaeger backend that can receive spans. At this time it supports two sets of non-overlapping APIs.
### OpenTelemetry Protocol (stable)
Since v1.35, the Jaeger backend can receive trace data from the OpenTelemetry SDKs in their native [OpenTelemetry Protocol (OTLP)][otlp]. It is no longer necessary to configure the OpenTelemetry SDKs with Jaeger exporters, nor deploy the OpenTelemetry Collector between the OpenTelemetry SDKs and the Jaeger backend.
The OTLP data is accepted in these formats: (1) binary gRPC, (2) Protobuf over HTTP, (3) JSON over HTTP. For more details on the OTLP receiver see the [official documentation][otlp-rcvr]. Note that not all configuration options are supported in **jaeger-collector** (see `--collector.otlp.*` [CLI Flags](../../deployment/cli/#jaeger-collector)), and only tracing data is accepted, since Jaeger does not store other telemetry types.
| Port | Protocol | Endpoint | Format
| ----- | ------- | ------------ | ----
| 4317 | gRPC | n/a | Protobuf
| 4318 | HTTP | `/v1/traces` | Protobuf or JSON
[otlp-rcvr]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver/README.md
[otlp]: https://github.com/open-telemetry/opentelemetry-proto/blob/main/docs/specification.md
### Protobuf via gRPC (stable)
**Deprecated**: we recommend the OpenTelemetry protocol.
Since Jaeger v1.11, the official protocol between user applications and **jaeger-collector**s is `jaeger.api_v2.CollectorService` gRPC endpoint defined in [collector.proto] IDL file. The same endpoint can be used to submit trace data from SDKs directly to **jaeger-collector**.
### Thrift over HTTP (stable)
**Deprecated**: we recommend the OpenTelemetry protocol.
The payload in [jaeger.thrift] format can be submitted in an HTTP POST request to the `/api/traces` endpoint, for example, `https://jaeger-collector:14268/api/traces`. The `Batch` struct needs to be encoded using Thrift's `binary` encoding, and the HTTP request should specify the content type header:
```http
Content-Type: application/vnd.apache.thrift.binary
```
### JSON over HTTP (n/a)
There is no official Jaeger JSON format that can be accepted by **jaeger-collector**.
Jaeger does accept the OpenTelemetry protocol via JSON (see [above](#opentelemetry-protocol-stable)).
### Zipkin Formats (stable)
**jaeger-collector** can also accept spans in several Zipkin data formats, namely JSON v1/v2 and Thrift. **jaeger-collector** needs to be configured to enable Zipkin HTTP server, e.g. on port 9411 used by Zipkin collectors. The server enables two endpoints that expect POST requests:
* `/api/v1/spans` for submitting spans in Zipkin JSON v1 or Zipkin Thrift format.
* `/api/v2/spans` for submitting spans in Zipkin JSON v2.
## Trace retrieval APIs
Traces saved in the storage can be retrieved by calling **jaeger-query** Service.
### gRPC/Protobuf (stable)
The recommended way for programmatically retrieving traces and other data is via the `jaeger.api_v2.QueryService` gRPC endpoint defined in [query.proto] IDL file. In the default configuration this endpoint is accessible from `jaeger-query:16685`.
### HTTP JSON (internal)
Jaeger UI communicates with **jaeger-query** Service via JSON API. For example, a trace can be retrieved via a GET request to `https://jaeger-query:16686/api/traces/{trace-id-hex-string}`. This JSON API is intentionally undocumented and subject to change.
## Remote Storage API (stable)
When using the `grpc` storage type (a.k.a. [remote storage](../../deployment/#remote-storage)), Jaeger components can use custom storage backends as long as those backends implement the gRPC [Remote Storage API][storage.proto].
## Remote Sampling Configuration (stable)
This API supports Jaeger's [Remote Sampling](../sampling/#remote-sampling) protocol, defined in the [sampling.proto] IDL file.
**jaeger-collector** implements this API. See [Remote Sampling](../sampling/#remote-sampling) for details on how to configure the Collector with sampling strategies.
The following table lists different endpoints and formats that can be used to query for sampling strategies. The official HTTP/JSON endpoints use standard [Protobuf-to-JSON mapping](https://developers.google.com/protocol-buffers/docs/proto3#json).
Component | Port | Endpoint | Format | Notes
--------- | ----- | ----------------- | --------- | -----
Collector | 14268 | `/api/sampling` | HTTP/JSON | Recommended for most SDKs
Collector | 14250 | [sampling.proto] | gRPC | For SDKs that want to use gRPC (e.g. OpenTelemetry Java SDK)
**Examples**
Run all-in-one in one terminal:
```shell
$ go run ./cmd/all-in-one \
--sampling.strategies-file=cmd/all-in-one/sampling_strategies.json
```
Query the endpoint in another terminal:
```shell
$ curl "http://localhost:14268/api/sampling?service=foo"
{"strategyType":"PROBABILISTIC","probabilisticSampling":{"samplingRate":1}}
```
## Service dependencies graph (internal)
Can be retrieved from**jaeger-query** Service at `/api/dependencies` endpoint. The GET request expects two parameters:
* `endTs` (number of milliseconds since epoch) - the end of the time interval
* `lookback` (in milliseconds) - the length the time interval (i.e. start-time + lookback = end-time).
The returned JSON is a list of edges represented as tuples `(caller, callee, count)`.
For programmatic access to the service graph, the recommended API is gRPC/Protobuf described above.
## Service Performance Monitoring (internal)
Please refer to the [SPM Documentation](../../deployment/spm/#api)
[jaeger-idl]: https://github.com/jaegertracing/jaeger-idl/
[jaeger.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/main/thrift/jaeger.thrift
[sampling.thrift]: https://github.com/jaegertracing/jaeger-idl/blob/main/thrift/sampling.thrift
[collector.proto]: https://github.com/jaegertracing/jaeger-idl/blob/main/proto/api_v2/collector.proto
[query.proto]: https://github.com/jaegertracing/jaeger-idl/blob/main/proto/api_v2/query.proto
[sampling.proto]: https://github.com/jaegertracing/jaeger-idl/blob/main/proto/api_v2/sampling.proto
[grpc-reflection]: https://github.com/grpc/grpc-go/blob/master/Documentation/server-reflection-tutorial.md#enable-server-reflection
[gogo-reflection]: https://jbrandhorst.com/post/gogoproto/#reflection
[storage.proto]: https://github.com/jaegertracing/jaeger/blob/v1.76.0/internal/storage/v1/grpc/proto/storage.proto
---
## File: content/docs/v1/1.76/architecture/sampling.md
---
title: Sampling
aliases: [../sampling]
hasparent: true
---
Jaeger libraries implement consistent upfront (or head-based) sampling. For example, assume we have a simple call graph where service A calls service B, and B calls service C: `A -> B -> C`. When service A receives a request that contains no tracing information, Jaeger tracer will start a new trace, assign it a random trace ID, and make a sampling decision based on the currently installed sampling strategy. The sampling decision will be propagated with the requests to B and to C, so those services will not be making the sampling decision again but instead will respect the decision made by the top service A. This approach guarantees that if a trace is sampled, all its spans will be recorded in the backend. If each service was making its own sampling decision we would rarely get complete traces in the backend.
## Client Sampling Configuration
{{< warning >}}
This section only applies to classic Jaeger SDKs, which are now deprecated.
We recommend using the [OpenTelemetry SDKs](https://opentelemetry.io).
{{< /warning >}}
When using configuration object to instantiate the tracer, the type of sampling can be selected via `sampler.type` and `sampler.param` properties. Jaeger libraries support the following samplers:
* **Constant** (`sampler.type=const`) sampler always makes the same decision for all traces. It either samples all traces (`sampler.param=1`) or none of them (`sampler.param=0`).
* **Probabilistic** (`sampler.type=probabilistic`) sampler makes a random sampling decision with the probability of sampling equal to the value of `sampler.param` property. For example, with `sampler.param=0.1` approximately 1 in 10 traces will be sampled.
* **Rate Limiting** (`sampler.type=ratelimiting`) sampler uses a leaky bucket rate limiter to ensure that traces are sampled with a certain constant rate. For example, when `sampler.param=2.0` it will sample requests with the rate of 2 traces per second.
* **Remote** (`sampler.type=remote`, which is also the default) sampler consults **jaeger-collector** for the appropriate sampling strategy to use in the current service. This allows controlling the sampling strategies in the services from a central configuration in Jaeger backend (see [Remote Sampling](#remote-sampling)), or even dynamically (see [Adaptive Sampling](#adaptive-sampling)).
## Remote Sampling
If your client SDKs are configured to use remote sampling configuration (see [Remote Sampling API][remote-sampling-api]) then sampling rates can be centrally controlled via **jaeger-collector**s. In this setup a sampling strategy configuration is served to the client SDK that describes endpoints and their sampling probabilities. This configuration can be generated by **jaeger-collector** in two different ways: [periodically loaded from a file](#file-based-sampling-configuration) or [dynamically calculated based on traffic](#adaptive-sampling). The method of generation is controlled by the environment variable `SAMPLING_CONFIG_TYPE` which can be set to either `file` (default) or `adaptive`.
[remote-sampling-api]: ../apis/#remote-sampling-configuration-stable
### File-based Sampling Configuration
**jaeger-collector**s can be instantiated with the `--sampling.strategies-file` option that points to a file containing sampling strategies to be served to Jaeger clients. The option's value can contain a path to a JSON file, which will be automatically reloaded if its contents change, or an HTTP URL from where the file will be periodically retrieved, with reload frequency controlled by the `--sampling.strategies-reload-interval` option.
If no configuration is provided, **jaeger-collector**s will return the default probabilistic sampling policy with probability 0.001 (0.1%) for all services.
Example `strategies.json`:
```json
{
"service_strategies": [
{
"service": "foo",
"type": "probabilistic",
"param": 0.8,
"operation_strategies": [
{
"operation": "op1",
"type": "probabilistic",
"param": 0.2
},
{
"operation": "op2",
"type": "probabilistic",
"param": 0.4
}
]
},
{
"service": "bar",
"type": "ratelimiting",
"param": 5
}
],
"default_strategy": {
"type": "probabilistic",
"param": 0.5,
"operation_strategies": [
{
"operation": "/health",
"type": "probabilistic",
"param": 0.0
},
{
"operation": "/metrics",
"type": "probabilistic",
"param": 0.0
}
]
}
}
```
`service_strategies` element defines service specific sampling strategies and `operation_strategies` defines operation specific sampling strategies. There are 2 types of strategies possible: `probabilistic` and `ratelimiting` which are described [above](#client-sampling-configuration) (NOTE: `ratelimiting` is not supported for `operation_strategies`). `default_strategy` defines the catch-all sampling strategy that is propagated if the service is not included as part of `service_strategies`.
In the above example:
* All operations of service `foo` are sampled with probability 0.8 except for operations `op1` and `op2` which are probabilistically sampled with probabilities 0.2 and 0.4 respectively.
* All operations for service `bar` are rate-limited at 5 traces per second.
* Any other service will be sampled with probability 0.5 defined by the `default_strategy`.
* The `default_strategy` also includes shared per-operation strategies. In this example we disable tracing on `/health` and `/metrics` endpoints for all services by using probability 0. These per-operation strategies will apply to any new service not listed in the config, as well as to the `foo` and `bar` services unless they define their own strategies for these two operations.
### Adaptive Sampling
Since Jaeger v1.27.
Adaptive sampling works in **jaeger-collector** by observing the spans received from services and recalculating sampling probabilities for each service/endpoint combination to ensure that the volume of collected traces matches `--sampling.target-samples-per-second`. When a new service or endpoint is detected, it is initially sampled with `--sampling.initial-sampling-probability` until enough data is collected to calculate the rate appropriate for the traffic going through the endpoint.
Adaptive sampling requires a storage backend to store the observed traffic data and computed probabilities. At the moment `memory` (for all-in-one deployment), `cassandra`, `badger`, `elasticsearch` and `opensearch` are supported as sampling storage backends.
By default adaptive sampling will attempt to use the backend specified by `SPAN_STORAGE_TYPE` to store data. However, a second type of backend can also be specified by using `SAMPLING_STORAGE_TYPE`. For instance, `SPAN_STORAGE_TYPE=elasticsearch SAMPLING_STORAGE_TYPE=cassandra ./jaeger-collector` will run **jaeger-collector** in a mode where it attempts to store its span data in the configured elasticsearch cluster and its adaptive sampling data in the configured cassandra cluster. Note that this feature can not be used to store span and adaptive sampling data in two different backends of the same type.
Read [this blog post](https://medium.com/jaegertracing/adaptive-sampling-in-jaeger-50f336f4334) for more details on adaptive sampling engine.
---
## File: content/docs/v1/1.76/deployment/_index.md
---
title: Deployment
weight: 4
children:
- title: Operator for Kubernetes
navtitle: Kubernetes
url: operator
- title: Frontend/UI
url: frontend-ui
- title: CLI Flags
url: cli
- title: Security
url: security
- title: On Windows
url: windows
- title: Service Performance Monitoring (SPM)
navtitle: SPM
url: spm
---
The main Jaeger backend components are released as Docker images on [Docker Hub](https://hub.docker.com/r/jaegertracing) and [Quay](https://quay.io/organization/jaegertracing):
Component | Docker Hub | Quay
--------------------- | -------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------
**jaeger-all-in-one** | [hub.docker.com/r/jaegertracing/all-in-one/](https://hub.docker.com/r/jaegertracing/all-in-one/) | [quay.io/repository/jaegertracing/all-in-one](https://quay.io/repository/jaegertracing/all-in-one)
**jaeger-collector** | [hub.docker.com/r/jaegertracing/jaeger-collector/](https://hub.docker.com/r/jaegertracing/jaeger-collector/) | [quay.io/repository/jaegertracing/jaeger-collector](https://quay.io/repository/jaegertracing/jaeger-collector)
**jaeger-query** | [hub.docker.com/r/jaegertracing/jaeger-query/](https://hub.docker.com/r/jaegertracing/jaeger-query/) | [quay.io/repository/jaegertracing/jaeger-query](https://quay.io/repository/jaegertracing/jaeger-query)
**jaeger-ingester** | [hub.docker.com/r/jaegertracing/jaeger-ingester/](https://hub.docker.com/r/jaegertracing/jaeger-ingester/) | [quay.io/repository/jaegertracing/jaeger-ingester](https://quay.io/repository/jaegertracing/jaeger-ingester)
**jaeger-remote-storage** | [hub.docker.com/r/jaegertracing/jaeger-remote-storage/](https://hub.docker.com/r/jaegertracing/jaeger-remote-storage/) | [quay.io/repository/jaegertracing/jaeger-remote-storage](https://quay.io/repository/jaegertracing/jaeger-remote-storage)
The images listed above are the primary release versions. Most components have additional images published:
* `${component}-debug` images include Delve debugger
* `${component}-snapshot` images are published from the tip of the main branch for every commit, allowing testing of unreleased versions
* `${component}-debug-snapshot` snapshot images that include the Delve debugger
There are orchestration templates for running Jaeger with:
* Kubernetes: [github.com/jaegertracing/jaeger-kubernetes](https://github.com/jaegertracing/jaeger-kubernetes),
* OpenShift: [github.com/jaegertracing/jaeger-openshift](https://github.com/jaegertracing/jaeger-openshift).
## Configuration Options
Jaeger binaries can be configured in a number of ways (in the order of decreasing priority):
* command line arguments,
* environment variables,
* configuration files in JSON, TOML, YAML, HCL, or Java properties formats.
To see the complete list of options, run the binary with the `help` command or refer to the [CLI Flags](./cli/) page for more information. Options that are specific to a certain storage backend are _only listed if the storage type is selected_. For example, to see all available options in the Collector with Cassandra storage:
```sh
$ docker run --rm \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
help
```
In order to provide configuration parameters via environment variables, find the respective command line option and convert its name to UPPER_SNAKE_CASE, for example:
Command line option | Environment variable
-----------------------------------|-------------------------------
`--cassandra.connections-per-host` | `CASSANDRA_CONNECTIONS_PER_HOST`
`--metrics-backend` | `METRICS_BACKEND`
## All-in-one
Jaeger all-in-one is a special distribution that combines three Jaeger components, [collector](#collector), and [query service/UI](#query-service--ui), in a single binary or container image. It is useful for single-node deployments where your trace volume is light enough to be handled by a single instance. By default, all-in-one starts with `memory` storage, meaning it will lose all data upon restart. All other [span storage backends](#span-storage-backends) can also be used with all-in-one, but `memory` and `badger` are exclusive to all-in-one because they cannot be shared between instances.
All-in-one listens to the same ports as the components it contains (described below), with the exception of the admin port.
Port | Protocol | Function
----- | ------- | ---
14269 | HTTP | admin port: health check at `/` and metrics at `/metrics`
```bash
## make sure to expose only the ports you use in your deployment scenario!
docker run -d --name jaeger \
-e COLLECTOR_OTLP_ENABLED=true \
-e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
-p 5775:5775/udp \
-p 6831:6831/udp \
-p 6832:6832/udp \
-p 5778:5778 \
-p 16686:16686 \
-p 14250:14250 \
-p 14268:14268 \
-p 14269:14269 \
-p 4317:4317 \
-p 4318:4318 \
-p 9411:9411 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
You can navigate to `http://localhost:16686` to access the Jaeger UI.
## Collector
**jaeger-collector**s are stateless and thus many instances of **jaeger-collector** can be run in parallel.
**jaeger-collector**s require almost no configuration, except for storage location, such as
`--cassandra.keyspace` and `--cassandra.servers` options, or the location of Elasticsearch cluster,
via `--es.server-urls`, depending on which storage is specified. See the [CLI Flags](./cli/) for all
command line options.
At default settings **jaeger-collector** exposes the following ports:
| Port | Protocol | Endpoint | Function
| ----- | ------- | -------- | ----
| 4317 | gRPC | n/a | Accepts traces in [OpenTelemetry OTLP format][otlp] (Protobuf).
| 4318 | HTTP | `/v1/traces` | Accepts traces in [OpenTelemetry OTLP format][otlp] (Protobuf and JSON).
| 14268 | HTTP | `/api/sampling` | Serves sampling policies (see [Remote Sampling](../architecture/sampling/#remote-sampling)).
| | | `/api/traces` | Accepts spans in [jaeger.thrift][jaeger-thrift] format with `binary` thrift protocol (`POST`).
| 14269 | HTTP | `/` | Admin port: health check (`GET`).
| | | `/metrics` | Prometheus-style metrics (`GET`).
| 9411 | HTTP | `/api/v1/spans` and `/api/v2/spans` | Accepts Zipkin spans in Thrift, JSON and Proto (disabled by default).
| 14250 | gRPC | n/a | Accepts spans in [model.proto][] Protobuf format.
## Ingester
**jaeger-ingester** is a service which reads span data from Kafka topic and writes it to another storage backend (Elasticsearch or Cassandra).
Port | Protocol | Function
----- | ------- | ---
14270 | HTTP | admin port: health check at `/` and metrics at `/metrics`
To view all exposed configuration options run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-ingester:{{< currentVersion >}}
--help
```
## Query Service & UI
**jaeger-query** serves the API endpoints and a React/Javascript UI.
The service is stateless and is typically run behind a load balancer, such as [**NGINX**](https://www.nginx.com/).
At default settings the **jaeger-query** service exposes the following port(s):
Port | Protocol | Function
----- | ------- | ---
16685 | gRPC | Protobuf/gRPC [QueryService](https://github.com/jaegertracing/jaeger-idl/blob/master/proto/api_v2/query.proto)
16686 | HTTP | `/api/*` endpoints and Jaeger UI at `/`
16687 | HTTP | admin port: health check at `/` and metrics at `/metrics`
### Minimal deployment example (Elasticsearch backend):
```sh
docker run -d --rm \
-p 16685:16685 \
-p 16686:16686 \
-p 16687:16687 \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=http://: \
jaegertracing/jaeger-query:{{< currentVersion >}}
```
### Clock Skew Adjustment
Jaeger backend combines trace data from applications that are usually running on different hosts. The hardware clocks on the hosts often experience relative drift, known as the [clock skew effect](https://en.wikipedia.org/wiki/Clock_skew). Clock skew can make it difficult to reason about traces, for example, when a server span may appear to start earlier than the client span, which should not be possible. **jaeger-query** service implements a clock skew adjustment algorithm ([code](https://github.com/jaegertracing/jaeger/blob/v1.76.0/model/adjuster/clockskew.go)) to correct for clock drift, using the knowledge about causal relationships between spans. All adjusted spans have a warning displayed in the UI that provides the exact clock skew delta applied to their timestamps.
Sometimes these adjustments themselves make the trace hard to understand. For example, when repositioning the server span within the bounds of its parent span, Jaeger does not know the exact relationship between the request and response latencies, so it assumes they are equal and places the child span in the middle of the parent span (see [issue #961](https://github.com/jaegertracing/jaeger/issues/961#issuecomment-453925244)).
**jaeger-query** service supports a configuration flag `--query.max-clock-skew-adjustment` that controls how much clock skew adjustment should be allowed. Setting this parameter to zero (`0s`) disables clock skew adjustment completely. This setting applies to all traces retrieved from the given query service. There is an open [ticket #197](https://github.com/jaegertracing/jaeger-ui/issues/197) to support toggling the adjustment on and off directly in the UI.
### UI Base Path
The base path for all **jaeger-query** HTTP routes can be set to a non-root value, e.g. `/jaeger` would cause all UI URLs to start with `/jaeger`. This can be useful when running **jaeger-query** behind a reverse proxy.
The base path can be configured via the `--query.base-path` command line parameter or the `QUERY_BASE_PATH` environment variable.
### UI Customization and Embedding
Please refer to the [dedicated Frontend/UI page](./frontend-ui/).
## Remote Storage (component)
**jaeger-remote-storage** implements the [Remote Storage gRPC API][storage.proto] and proxies it into one of the regular Jaeger backends. It can be useful in the situation when we want to run a full deployment of Jaeger components, e.g., separate collector and query services, but use a single-node storage backend like the memory store or Badger. Without the remote storage, the single-node backends can only be used with all-in-one since they cannot be shared between multiple processes.
At default settings the service listens on the following port(s):
Port | Protocol | Function
----- | ------- | ---
17271 | gRPC | [Remote Storage API][storage.proto]
17270 | HTTP | admin port: health check at `/` and metrics at `/metrics`
## Span Storage Backends
Jaeger requires a persistent storage backend. Cassandra and Elasticsearch/OpenSearch are the primary supported distributed storage backends. Additional backends are [discussed here](https://github.com/jaegertracing/jaeger/issues/638).
The storage type can be passed via `SPAN_STORAGE_TYPE` environment variable. Valid values are `cassandra`, `elasticsearch`, `kafka` (only as a buffer), `badger` and `memory`.
As of version 1.6.0, it's possible to use multiple storage types at the same time by providing a comma-separated list of valid types to the `SPAN_STORAGE_TYPE` environment variable. It's important to note that all listed storage types are used for writing, but only the first type in the list will be used for reading and archiving.
For large scale production deployment the Jaeger team [recommends OpenSearch backend over Cassandra](../faq/#what-is-the-recommended-storage-backend).
### Memory
The in-memory storage is not intended for production workloads. It's intended as a simple solution to get started quickly and
data will be lost once the process exits.
By default, there's no limit in the amount of traces stored in memory but a limit can be established by passing an
integer value via `--memory.max-traces`.
### Badger - local storage
* Since Jaeger v1.9
[Badger](https://github.com/dgraph-io/badger) is an embedded local storage, only available
with **all-in-one** distribution. By default, it acts as ephemeral storage using a temporary file system.
This can be overridden by using the `--badger.ephemeral=false` option.
```sh
docker run \
-e SPAN_STORAGE_TYPE=badger \
-e BADGER_EPHEMERAL=false \
-e BADGER_DIRECTORY_VALUE=/badger/data \
-e BADGER_DIRECTORY_KEY=/badger/key \
-v :/badger \
-p 16686:16686 \
jaegertracing/all-in-one:{{< currentVersion >}}
```
* [Upgrade Badger v1 to v3](https://github.com/jaegertracing/jaeger/blob/v1.76.0/internal/storage/v1/badger/docs/upgrade-v1-to-v3.md)
* [Badger file permissions as non-root service](https://github.com/jaegertracing/jaeger/blob/v1.76.0/internal/storage/v1/badger/docs/storage-file-non-root-permission.md)
### Cassandra
* Supported versions: 4.x, 5.x
Deploying Cassandra itself is out of scope for our documentation. One good
source of documentation is the [Apache Cassandra Docs](https://cassandra.apache.org/doc/latest/).
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
-e CASSANDRA_SERVERS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
Note: White space characters are allowed in `CASSANDRA_SERVERS`. For Example: Servers can be passed as `CASSANDRA_SERVERS="1.2.3.4, 5.6.7.8" for better readability.
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=cassandra \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Schema script
A script is provided to initialize Cassandra keyspace and schema
using Cassandra's interactive shell [`cqlsh`][cqlsh]:
```sh
MODE=test sh ./internal/storage/v1/cassandra/schema/create.sh | cqlsh
```
Or using the published Docker image (make sure to provide the right IP address):
```sh
docker run \
-e CQLSH_HOST={server IP address} \
jaegertracing/jaeger-cassandra-schema:{{< currentVersion >}}
```
For production deployment, pass `MODE=prod DATACENTER={datacenter}` arguments to the script,
where `{datacenter}` is the name used in the Cassandra configuration / network topology.
The script also allows overriding TTL, keyspace name, replication factor, etc.
Run the script without arguments to see the full list of recognized parameters.
**Note**: See [README](https://github.com/jaegertracing/jaeger/blob/v1.76.0/internal/storage/v1/cassandra/schema/README.md) for more details on Cassandra schema management.
#### TLS support
Jaeger supports TLS client to node connections as long as you've configured
your Cassandra cluster correctly. After verifying with e.g. `cqlsh`, you can
configure the collector and query like this:
```sh
docker run \
-e CASSANDRA_SERVERS=<...> \
-e CASSANDRA_TLS=true \
-e CASSANDRA_TLS_SERVER_NAME="CN-in-certificate" \
-e CASSANDRA_TLS_KEY= \
-e CASSANDRA_TLS_CERT= \
-e CASSANDRA_TLS_CA= \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
The schema tool also supports TLS. You need to make a custom cqlshrc file like
so:
```ini
# Creating schema in a cassandra cluster requiring client TLS certificates.
#
# Create a volume for the schema docker container containing four files:
# cqlshrc: this file
# ca-cert: the cert authority for your keys
# client-key: the keyfile for your client
# client-cert: the cert file matching client-key
#
# if there is any sort of DNS mismatch and you want to ignore server validation
# issues, then uncomment validate = false below.
#
# When running the container, map this volume to /root/.cassandra and set the
# environment variable CQLSH_SSL=--ssl
[ssl]
certfile = ~/.cassandra/ca-cert
userkey = ~/.cassandra/client-key
usercert = ~/.cassandra/client-cert
# validate = false
```
#### Compatible Backends
* ScyllaDB [can be used](https://github.com/jaegertracing/jaeger/blob/v1.76.0/docker-compose/scylladb/README.md) as a drop-in replacement for Cassandra since it uses the same data model and query language.
### Elasticsearch
* Supported since Jaeger v0.6.0
* Supported ES versions: 7.x, 8.x (since Jaeger v1.52.0)
Elasticsearch version is automatically retrieved from root/ping endpoint.
Based on this version Jaeger uses compatible index mappings and Elasticsearch REST API.
The version can be explicitly provided via `--es.version=` flag.
Elasticsearch does not require initialization other than
[installing and running Elasticsearch](https://www.elastic.co/downloads/elasticsearch).
Once it is running, pass the correct configuration values to **jaeger-collector** and **jaeger-query**.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
-e ES_SERVER_URLS=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=elasticsearch \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Shards and Replicas for Elasticsearch indices
Shards and replicas are some configuration values to take special attention to, because this is decided upon
index creation. [This article](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) goes into
more information about choosing how many shards should be chosen for optimization.
#### Elasticsearch Rollover
[Elasticsearch rollover](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html) is an index management strategy that optimizes use of resources allocated to indices.
For example, indices that do not contain any data still allocate shards, and conversely, a single index might contain significantly more data than the others.
Jaeger by default stores data in daily indices which might not optimally utilize resources. Rollover feature can be enabled by `--es.use-aliases=true`.
Rollover lets you configure when to roll over to a new index based on one or more of the following criteria:
* `max_age` - the maximum age of the index. It uses [time units](https://www.elastic.co/guide/en/elasticsearch/reference/master/common-options.html#time-units): `d`, `h`, `m`.
* `max_docs` - the maximum documents in the index.
* `max_size` - the maximum estimated size of primary shards (since Elasticsearch 6.x). It uses [byte size units](https://www.elastic.co/guide/en/elasticsearch/reference/master/common-options.html#byte-units) `tb`, `gb`, `mb`.
Rollover index management strategy is more complex than using the default daily indices and it requires an initialization job to prepare the storage and two cron jobs to manage indices.
To learn more about rollover index management in Jaeger refer to this
[article](https://medium.com/jaegertracing/using-elasticsearch-rollover-to-manage-indices-8b3d0c77915d).
For automated rollover, please refer to [Elasticsearch ILM support](#elasticsearch-ilm-support).
##### Initialize
The following command prepares Elasticsearch for rollover deployment by creating index aliases, indices, and index templates:
```sh
docker run -it --rm --net=host jaegertracing/jaeger-es-rollover:latest init http://localhost:9200 # <1>
```
If you need to initialize archive storage, add `-e ARCHIVE=true`.
After the initialization Jaeger can be deployed with `--es.use-aliases=true`.
##### Rolling over to a new index
The next step is to periodically execute the rollover API which rolls the write alias to a new index based on supplied conditions. The command also adds a new index to the read alias to make new data available for search.
```shell
docker run -it --rm --net=host -e CONDITIONS='{"max_age": "2d"}' jaegertracing/jaeger-es-rollover:latest rollover http://localhost:9200 # <1>
```
<1> The command rolls the alias over to a new index if the age of the current write index is older than 2 days. For more conditions see [Elasticsearch docs](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html).
The next step is to remove old indices from read aliases. It means that old data will not be available for search. This imitates the behavior of `--es.max-span-age` flag used in the default index-per-day deployment. This step could be optional and old indices could be simply removed by index cleaner in the next step.
```sh
docker run -it --rm --net=host -e UNIT=days -e UNIT_COUNT=7 jaegertracing/jaeger-es-rollover:latest lookback http://localhost:9200 # <1>
```
<1> Removes indices older than 7 days from read alias.
##### Remove old data
The historical data can be removed with the `jaeger-es-index-cleaner` that is also used for daily indices.
```shell
docker run -it --rm --net=host -e ROLLOVER=true jaegertracing/jaeger-es-index-cleaner:latest 14 http://localhost:9200 # <1>
```
<1> Remove indices older than 14 days.
#### Elasticsearch ILM support
{{< warning >}}
Experimental feature added in [release v1.22.0](https://github.com/jaegertracing/jaeger/releases/tag/v1.22.0).
Supported Elasticsearch versions: 7.x
{{< /warning >}}
[Elasticsearch ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html) automatically manages indices according to performance, resiliency, and retention requirements.
For example:
* Rollover to a new index by size (bytes or number of documents) or age, archiving previous indices
* Delete stale indices to enforce data retention standards
###### Enabling ILM support
* Create an ILM policy in elasticsearch named jaeger-ilm-policy.
For example, the following policy will rollover the "active" index when it is
older than 1m and delete indices that are older than 2m.
```shell
curl -X PUT http://localhost:9200/_ilm/policy/jaeger-ilm-policy \
-H 'Content-Type: application/json; charset=utf-8' \
--data-binary @- << EOF
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_age": "1m"
},
"set_priority": {
"priority": 100
}
}
},
"delete": {
"min_age": "2m",
"actions": {
"delete": {}
}
}
}
}
}
EOF
```
* Run elasticsearch initializer with `ES_USE_ILM=true`:
```shell
docker run -it --rm --net=host -e ES_USE_ILM=true jaegertracing/jaeger-es-rollover:latest init http://localhost:9200 # <1>
```
<1> If you need to initialize archive storage, add `-e ARCHIVE=true`.
{{< info >}}
While initializing with ILM support, make sure that an ILM policy named `jaeger-ilm-policy` is created in Elasticsearch beforehand (see the previous step), otherwise the following error message will be shown:
"ILM policy jaeger-ilm-policy doesn't exist in Elasticsearch. Please create it and rerun init"
{{< /info >}}
After the initialization, deploy Jaeger with `--es.use-ilm=true` and `--es.use-aliases=true`.
#### Upgrade Elasticsearch version
Elasticsearch defines wire and index compatibility versions. The index compatibility defines
the minimal version a node can read data from. For example Elasticsearch 8 can read indices
created by Elasticsearch 7, however it cannot read indices created by Elasticsearch 6 even
though they use the same index mappings. Therefore upgrade from Elasticsearch 7 to 8 does not require any
data migration. However, upgrade from Elasticsearch 6 to 8 has to be done through Elasticsearch 7 and wait
until indices created by ES 6.x are removed or explicitly reindexed.
Refer to the Elasticsearch [documentation](https://www.elastic.co/docs/deploy-manage/upgrade/deployment-or-cluster)
for wire and index compatibility versions. Generally this information can be retrieved from root/ping REST endpoint.
##### Reindex
Manual reindexing can be used when upgrading from Elasticsearch 6 to 8 (through Elasticsearch 7)
without waiting until indices created by Elasticsearch 6 are removed.
1. Reindex all span indices to new indices with suffix `-1`:
```bash
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
{
"source": {
"index": "jaeger-span-*"
},
"dest": {
"index": "jaeger-span"
},
"script": {
"lang": "painless",
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length())) + '-1'"
}
}
```
2. Delete indices with old mapping:
```bash
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*,-\*-1
```
3. Create indices without `-1` suffix:
```bash
curl -ivX POST -H "Content-Type: application/json" http://localhost:9200/_reindex -d @reindex.json
{
"source": {
"index": "jaeger-span-*"
},
"dest": {
"index": "jaeger-span"
},
"script": {
"lang": "painless",
"source": "ctx._index = 'jaeger-span-' + (ctx._index.substring('jaeger-span-'.length(), ctx._index.length() - 2))"
}
}
```
4. Remove suffixed indices:
```bash
curl -ivX DELETE -H "Content-Type: application/json" http://localhost:9200/jaeger-span-\*-1
```
Run the commands analogically for other Jaeger indices.
There might exist more effective migration procedure. Please share with the community any findings.
### Kafka
* Supported in Jaeger since 1.6.0
* Supported Kafka versions: 0.9+
Kafka can be used as an intermediary buffer between collector and an actual storage.
**jaeger-collector** is configured with `SPAN_STORAGE_TYPE=kafka` that makes it write all received spans
into a Kafka topic. [**jaeger-ingester**](#ingester) is used to read from
Kafka and store spans in another storage backend (Elasticsearch or Cassandra).
Writing to Kafka is particularly useful for building post-processing data pipelines.
#### Configuration
##### Minimal
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
-e KAFKA_PRODUCER_BROKERS=<...> \
-e KAFKA_TOPIC=<...> \
jaegertracing/jaeger-collector:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e SPAN_STORAGE_TYPE=kafka \
jaegertracing/jaeger-collector:{{< currentVersion >}} \
--help
```
#### Topic & partitions
Unless your Kafka cluster is configured to automatically create topics, you will need to create it ahead of time. You can refer to [the Kafka quickstart documentation](https://kafka.apache.org/documentation/#quickstart_createtopic) to learn how.
You can find more information about topics and partitions in general in the [official documentation](https://kafka.apache.org/documentation/#intro_topics). [This article](https://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/) provide more details about how to choose the number of partitions.
### Remote Storage
Jaeger supports a gRPC-based [Remote Storage API][storage.proto] that allows extending the Jaeger ecosystem with custom storage backends, not directly supported by the project. These storage backends can be deployed as a remote gRPC server (since Jaeger v1.30). Older deployment mode as sidecar plugin will not be supported starting from v1.58.
To use a remote storage as Jaeger storage backend, use `grpc` as the storage type and specify the remote gRPC server address. For more information, please refer to [internal/storage/v1/grpc](https://github.com/jaegertracing/jaeger/tree/v1.76.0/internal/storage/v1/grpc).
Example:
```sh
docker run \
-e SPAN_STORAGE_TYPE=grpc \
-e GRPC_STORAGE_SERVER=<...> \
jaegertracing/all-in-one:{{< currentVersion >}}
```
Known remote storage backends:
* [PostgreSQL by robbert229](https://github.com/robbert229/jaeger-postgresql)
## Metrics Storage Backends
Jaeger Query is capable of querying aggregated R.E.D metrics from a storage backend,
visualizing them on the [Monitor tab](./spm/). It should be emphasized that the
configured metrics storage type is for reading _only_ and therefore, only applies
to the Jaeger Query component (and All In One, which contains Jaeger Query).
The storage type can be passed via `METRICS_STORAGE_TYPE` environment variable.
Valid values are: `prometheus`.
### Prometheus
Any PromQL-compatible backend is supported by Jaeger Query. A list of these have
been compiled by Julius Volz in:
https://promlabs.com/blog/2020/11/26/an-update-on-promql-compatibility-across-vendors
#### Configuration
##### Minimal
```sh
docker run \
-e METRICS_STORAGE_TYPE=prometheus \
jaegertracing/jaeger-query:{{< currentVersion >}}
```
##### All options
To view the full list of configuration options, you can run the following command:
```sh
docker run \
-e METRICS_STORAGE_TYPE=prometheus \
jaegertracing/jaeger-query:{{< currentVersion >}} \
--help
```
#### TLS support
Jaeger supports TLS client to Prometheus server connections as long as you've [configured
your Prometheus server](https://prometheus.io/docs/guides/tls-encryption/) correctly. You can
configure **jaeger-query** like so:
```sh
docker run \
-e METRICS_STORAGE_TYPE=prometheus \
-e PROMETHEUS_SERVER_URL=<...> \
-e PROMETHEUS_TLS_ENABLED=true \
-e PROMETHEUS_TLS_CA= \
jaegertracing/jaeger-query:{{< currentVersion >}}
```
## Aggregation Jobs for Service Dependencies
Production deployments need an external process that aggregates data and creates dependency links between services. Project [spark-dependencies](https://github.com/jaegertracing/spark-dependencies) is a Spark job which derives dependency links and writes them directly to the storage.
[cqlsh]: https://cassandra.apache.org/doc/latest/cassandra/managing/tools/cqlsh.html
[zipkin-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/zipkincore.thrift
[jaeger-thrift]: https://github.com/jaegertracing/jaeger-idl/blob/master/thrift/jaeger.thrift
[model.proto]: https://github.com/jaegertracing/jaeger-idl/blob/main/proto/api_v2/model.proto
[thriftrw]: https://www.npmjs.com/package/thriftrw
[storage.proto]: https://github.com/jaegertracing/jaeger/blob/v1.76.0/internal/storage/v1/grpc/proto/storage.proto
[otlp]: https://github.com/open-telemetry/opentelemetry-proto/blob/main/docs/specification.md
---
## File: content/docs/v1/1.76/deployment/cli.md
---
title: CLI flags
aliases: [../cli]
widescreen: true
hasparent: true
---
This is auto-generated documentation for CLI flags supported by Jaeger binaries.
* CLI flags for some binaries change depending on the `SPAN_STORAGE_TYPE` environment variable. Relevant variations are included below.
* Some binaries support _commands_ (mostly informational), such as `env`, `docs`, `version`, and `status`. These commands are not included here.
* All parameters can be provided via environment variables, by changing all letters to upper-case and replacing all punctuation characters with the underscore `_`. For example, the value for the flag `--cassandra.connections-per-host` can be provided via the `CASSANDRA_CONNECTIONS_PER_HOST` environment variable.
{{< cli/tools-list >}}
---
## File: content/docs/v1/1.76/deployment/frontend-ui.md
---
title: Frontend/UI Configuration
navtitle: Frontend/UI
aliases: [../frontend-ui]
hasparent: true
weight: 7
---
## Configuration
Several aspects of the UI can be configured:
* The Dependencies section can be enabled / configured
* The [Monitor tab (aka: Service Performance Monitoring)](../spm/) can be enabled / configured
* App analytics tracking can be enabled / configured (via Google Analytics or custom plugin)
* Additional menu options can be added to the global nav
* Search input limits can be configured
These options can be configured by a JSON configuration file. The `--query.ui-config` command line parameter of the query service must then be set to the path to the JSON file when the query service is started.
An example configuration file (see [complete schema here](https://github.com/jaegertracing/jaeger-ui/blob/main/packages/jaeger-ui/src/types/config.ts)):
```json
{
"dependencies": {
"dagMaxNumServices": 200,
"menuEnabled": true
},
"monitor": {
"menuEnabled": true
},
"archiveEnabled": true,
"tracking": {
"gaID": "UA-000000-2",
"trackErrors": true
},
"menu": [
{
"label": "About Jaeger",
"items": [
{
"label": "GitHub",
"url": "https://github.com/jaegertracing/jaeger"
},
{
"label": "Docs",
"url": "https://www.jaegertracing.io/docs/latest/"
}
]
}
],
"search": {
"maxLookback": {
"label": "2 Days",
"value": "2d"
},
"maxLimit": 1500
},
"linkPatterns": [{
"type": "process",
"key": "jaeger.version",
"url": "https://github.com/jaegertracing/jaeger-client-java/releases/tag/#{jaeger.version}",
"text": "Information about Jaeger release #{jaeger.version}"
},
{
"type": "tags",
"key": "uniqueId",
"url": "https://mykibana.com/uniqueId=#{uniqueId}&traceId=#{trace.traceID}",
"text": "Redirect to kibana to view log"
},
{
"type": "traces",
"url": "https://my-logs.server?from=#{startTime | add -60000000 | epoch_micros_to_date_iso}&to=#{endTime | add 60000000 | epoch_micros_to_date_iso}'",
"text": "Redirect to kibana to view log with formatted dates"
}]
}
```
### Dependencies
`dependencies.dagMaxNumServices` defines the maximum number of services allowed before the DAG dependency view is disabled. Default: `200`.
`dependencies.menuEnabled` enables (`true`) or disables (`false`) the dependencies menu button. Default: `true`.
### Monitor
`monitor.menuEnabled` enables (`true`) or disables (`false`) the Monitor menu button. Default: `false`.
### Archive Support
`archiveEnabled` enables (`true`) or disables (`false`) the archive traces button. Default: `false`. It requires a configuration of an archive storage in Query service. Archived traces are only accessible directly by ID, they are not searchable.
### App Analytics Tracking
`tracking.gaID` defines the Google Analytics tracking ID. This is required for Google Analytics tracking, and setting it to a non-`null` value enables Google Analytics tracking. Default: `null`.
`tracking.customWebAnalytics` defines a factory function for a custom tracking plugin (only when using Javascript-form of UI configuration).
`tracking.trackErrors` enables (`true`) or disables (`false`) error tracking. Errors can only be tracked if a valid analytics tracker is configured. Default: `true`.
For additional details on app analytics see the [tracking README](https://github.com/jaegertracing/jaeger-ui/blob/master/packages/jaeger-ui/src/utils/tracking/README.md) in the UI repo.
### Custom Menu Items
`menu` allows additional links to be added to the global nav. The additional links are right-aligned.
In the sample JSON config above, the configured menu will have a dropdown labeled "About Jaeger" with sub-options for "GitHub" and "Docs". The format for a link in the top right menu is as follows:
```json
{
"label": "Some text here",
"url": "https://example.com"
}
```
Links can either be members of the `menu` Array, directly, or they can be grouped into a dropdown menu option. The format for a group of links is:
```json
{
"label": "Dropdown button",
"items": [ ]
}
```
The `items` Array should contain one or more link configurations.
### Search Input Limit
The `search.maxLimit` configures the maximum results that the input let you search.
The `search.maxLookback` configures the maximum time before the present users can query for traces. The options in the Lookback dropdown greater than this value will not be shown.
Field | Description
------|------------
label | The text displayed in the search form dropdown
value | The value submitted in the search query if the label is selected
### Link Patterns
The `linkPatterns` node can be used to create links from fields displayed in the Jaeger UI.
Field | Description
------|------------
type | The metadata section in which your link will be added: process, tags, logs, traces
key | The name of tag/process/log attribute which value will be displayed as a link, this field is not necessary for type `traces`.
url | The URL where the link should point to, it can be an external site or relative path in Jaeger UI
text | The text displayed in the tooltip for the link
Both `url` and `text` can be defined as templates (i.e. using `#{field-name}`) where Jaeger UI will dynamically substitute values based on tags/logs/traces data.
For traces, the supported template fields are: `duration`, `endTime`, `startTime`, `traceName` and `traceID`.
Further, the trace template fields are available for substitution in process/logs/tags type when the trace template fields are prefixed with `trace.`. For example: `trace.traceID`, `trace.startTime`.
#### Formatting
In addition to interpolating fields into links, formatting functions can be used. The syntax is `#{field | function}` (eg `#{endTime | epoch_micros_to_date_iso}'`). Formatting functions can be chained together (eg `#{startTime | add 60000000 | epoch_micros_to_date_iso}`).
The available formatting functions and a description of their behavior are:
##### `epoch_micros_to_date_iso`
Format a date in microseconds since epoch to an ISO date time.
Example:
`#{endTime | epoch_micros_to_date_iso}`
##### `pad_start`
Pad the start of a string with a given character until the resulting string reaches the given length. Behavior is implemented by and thus matches javascript's [String.padStart](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/padStart)
Arguments:
- `targetLength` (integer) - Desired length of the string after padding
- `padCharacter` (string) - String to pad the start of the input string with
Example: `#{traceID | pad_start 32 0}`
##### `add`
Add a value to another number. Can be positive or negative.
Arguments:
- `offset` (integer) - Value to add to the input number
Example: `#{startTime | add 1000000}`
## Embedded Mode
Starting with version 1.9, Jaeger UI provides an "embedded" layout mode which is intended to support integrating Jaeger UI into other applications. Currently (as of `v0`), the approach taken is to remove various UI elements from the page to make the UI better suited for space-constrained layouts.
The embedded mode is induced and configured via URL query parameters.
To enter embedded mode, the `uiEmbed=v0` query parameter and value must be added to the URL. For example, the following URL will show the trace with ID `abc123` in embedded mode:
```text
http://localhost:16686/trace/abc123?uiEmbed=v0
```
`uiEmbed=v0` is required.
Further, for each supported page a button {{< rawhtml >}}{{< /rawhtml >}} is added that will open the non-embedded page in a new tab.
The following pages support embedded mode:
* Search Page
* Trace Page
### Search Page
To integrate the Search Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```text
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0
```
[](/img/frontend-ui/embed-search-traces.png)
#### Configuration options
The following query parameter can be used to configure the layout of the search page :
* `uiSearchHideGraph=1` - disables the display of the scatter plot above the search results
```text
http://localhost:16686/search?
service=my-service&
start=1543917759557000&
end=1543921359557000&
limit=20&
lookback=1h&
maxDuration&
minDuration&
uiEmbed=v0&
uiSearchHideGraph=1
```
[](/img/frontend-ui/embed-search-traces-hide-graph.png)
### Trace Page
To integrate the Trace Page to our application we have to indicate to the Jaeger UI that we want to use the embed mode with `uiEmbed=v0`.
For example:
```sh
http://localhost:16686/trace/{trace-id}?uiEmbed=v0
```
[](/img/frontend-ui/embed-trace-view.png)
If we have navigated to this view from the search traces page we'll have a button to go back to the results page.
[](/img/frontend-ui/embed-trace-view-with-back-button.png)
#### Configuration options
The following query parameters can be used to configure the layout of the trace page :
* `uiTimelineCollapseTitle=1` causes the trace header to start out collapsed, which hides the summary and the minimap.
```text
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineCollapseTitle=1
```
[](/img/frontend-ui/embed-trace-view-with-collapse.png)
* `uiTimelineHideMinimap=1` removes the minimap, entirely, regardless of whether the trace header is expanded or not.
```text
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1
```
[](/img/frontend-ui/embed-trace-view-with-hide-minimap.png)
* `uiTimelineHideSummary=1` - removes the trace summary information (number of services, etc.) entirely, regardless of whether the trace header is expanded or not.
```text
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideSummary=1
```
[](/img/frontend-ui/embed-trace-view-with-hide-summary.png)
We can also combine the options:
```text
http://localhost:16686/trace/{trace-id}?
uiEmbed=v0&
uiTimelineHideMinimap=1&
uiTimelineHideSummary=1
```
[](/img/frontend-ui/embed-trace-view-with-hide-details-and-hide-minimap.png)