opentelemetry-collector-contrib Documentation

Technical Documentation: open-telemetry/opentelemetry-collector-contrib

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

OpenTelemetry Collector Contrib

This is a repository for OpenTelemetry Collector components that are not suitable for the core repository of the collector.

The official distributions, core and contrib, are available as part of the opentelemetry-collector-releases repository. Some of the components in this repository are part of the "core" distribution, such as the Jaeger and Prometheus components, but most of the components here are only available as part of the "contrib" distribution. Users of the OpenTelemetry Collector are also encouraged to build their own custom distributions with the OpenTelemetry Collector Builder, using the components they need from the core repository, the contrib repository, and possibly third-party or internal repositories.

Each component has its own support levels, as defined in the following sections. For each signal that a component supports, there's a stability level, setting the right expectations. It is possible then that a component will be Stable for traces but Alpha for metrics and Development for logs.

Stability levels Copy Chapter

Stability level for components in this repository follow the definitions from the OpenTelemetry Collector repository.

Gated features Copy Chapter

Some features are hidden behind feature gates before they are part of the main code path for the component. Note that the feature gates themselves might be at different lifecycle stages.

Support Copy Chapter

Each component is supported either by the community of OpenTelemetry Collector Contrib maintainers, as defined by the GitHub group @open-telemetry/collector-contrib-maintainers, or by specific vendors. See the individual README files for information about the specific components.

The OpenTelemetry Collector Contrib maintainers may at any time downgrade specific components if they are deemed unmaintained or if they pose a risk to the repository and/or binary distribution.

Even though the OpenTelemetry Collector Contrib maintainers are ultimately responsible for the components hosted here, actual support will likely be provided by individual contributors, typically a code owner for the specific component.

Contributing Copy Chapter

See CONTRIBUTING.md.

Maintainers Copy Section

• Alex Boten, Grafana Labs
• Andrzej Stencel, Elastic
• Antoine Toulme, Splunk
• Bogdan Drutu, Snowflake
• Braydon Kains, Google
• Christos Markou, Elastic
• Dmitrii Anoshin, Splunk
• Edmo Vamerlatti Costa, Elastic
• Evan Bradley, Dynatrace
• Pablo Baeyens, DataDog
• Sean Marciniak, Splunk
• Tyler Helmuth, Grafana Labs
• Yang Song, DataDog

For more information about the maintainer role, see the community repository.

Approvers Copy Section

• Arthur Silva Sens, Grafana Labs
• Constança Manteigas, Elastic
• Curtis Robert, Splunk
• David Ashpole, Google
• Israel Blancas, Coralogix
• Joshua MacDonald, Microsoft
• Matt Wear, Dash0
• Mike Goldsmith, Honeycomb
• Paulo Dias, Five9
• Paulo Janotti, Splunk
• Sam DeHaan, Grafana Labs
• Vihas Makwana, Elastic

For more information about the approver role, see the community repository.

Triagers Copy Section

• Benedikt Bongartz, Red Hat
• Bogdan Stancu, Adobe
• Dónal O'Sullivan, Elastic
• Douglas Camata, Coralogix
• Dylan Strohschein, Dynatrace
• George Krajcsovits, Grafana Labs
• James Moessis, Optiver
• Jared Tan, DaoCloud
• Khushi Jain, Elastic
• Murphy Chen, DaoCloud
• Ondrej Dubaj, Dynatrace
• Roger Coll, Elastic
• Vibhanshu Singh
• Actively seeking contributors to triage issues

For more information about the triager role, see the community repository.

Emeritus Copy Section

• Alolita Sharma, Triager
• Andrew Wilkins, Approver
• Anthony Mirabella, Approver
• Bryan Aguilar, Approver
• Daniel Jaglowski, Maintainer
• Florian Bacher, Triager
• Gabriel Aszalos, Triager
• Goutham Veeramachaneni, Triager
• Juraci Paixão Kröhling, Maintainer
• Przemek Maciolek, Approver
• Punya Biswal, Triager
• Ruslan Kovalov, Approver
• Steve Flanders, Triager
• Tigran Najaryan, Maintainer
• Ziqi Zhao, Approver

For more information about the emeritus role, see the community repository.

No Over-Representation Copy Section

A community member cannot be promoted to be a Collector contrib maintainer if, after their promotion, the resulting maintainers group has more than one-fourth (25%) of the members affiliated with the same employer. Job changes and similar events might result in over-representation, and no new maintainers from the same company can be promoted until representation is balanced again. In the event of confusion or concern, the OpenTelemetry Collector SIG will defer to the CNCF definition of "same employer".

PRs and Reviews Copy Chapter

When creating a PR please follow the process described
here.

New PRs will be automatically associated with the reviewers based on
CODEOWNERS. PRs will be also automatically assigned to one of the
maintainers or approvers for facilitation.

The facilitator is responsible for helping the PR author and reviewers to make progress
or if progress cannot be made for closing the PR.

If the reviewers do not have approval rights the facilitator is also responsible
for the official approval that is required for the PR to be merged and if the facilitator
is a maintainer they are responsible for merging the PR as well.

The facilitator is not required to perform a thorough review, but they are encouraged to
enforce Collector best practices and consistency across the codebase and component
behavior. The facilitators will typically rely on codeowner's detailed review of the code
when making the final approval decision.

Marking the PR with the ready to merge label should only happen (by triagers/approvers/maintainers)
once there is at least one approval from an approver, as per the description above.

Marking the PR with the waiting-for-approvers label can be done (by triagers/code owners/approvers/maintainers)
once the PR has been reviewed and approved by code owners, all review comments are addressed, CI is green,
and it is only waiting on a review from an approver or maintainer. Use this label instead of pinging
@open-telemetry/collector-contrib-approvers. Once an approver approves the PR, replace it with the ready to merge label.

Overview (`.chloggen/README.md`) Copy Section

Changelog folder Copy Section

This repo uses chloggen to manage its changelog files. You can find the source code for the tool here.

Here is a quick explanation of the config.yaml file for chloggen:

# The directory that stores individual changelog entries.
# Each entry is stored in a dedicated yaml file.
# - 'chloggen new' will copy the 'template_yaml' to this directory as a new entry file.
# - 'chloggen validate' will validate that all entry files are valid.
# - 'chloggen update' will read and delete all entry files in this directory, and update 'changelog_md'.
# Specify as relative path from root of repo.
# (Optional) Default: .chloggen
entries_dir: .chloggen

# This file is used as the input for individual changelog entries.
# Specify as relative path from root of repo.
# (Optional) Default: .chloggen/TEMPLATE.yaml
template_yaml: .chloggen/TEMPLATE.yaml

summary_template: .chloggen/summary.tmpl

# The CHANGELOG file or files to which 'chloggen update' will write new entries
# (Optional) Default filename: CHANGELOG.md
change_logs:
  user: CHANGELOG.md
  api: CHANGELOG-API.md

# The default change_log or change_logs to which an entry should be added.
# If 'change_logs' is specified in this file, and no value is specified for 'default_change_logs',
# then 'change_logs' MUST be specified in every entry file.
default_change_logs: [user]

Overview (`pkg/README.md`) Copy Section

`pkg` folder

The Go modules in this folder can help in building new components (receivers, processors, exporters, extensions...) and are meant for external consumption.
Carefully review the public API to ensure that we only expose what is deemed necessary, so that we can easily change the code without affecting external users.

Breaking changes can happen on any module on this folder under the opentelemetry-collector contribution guidelines.

If you are introducing a new module that will only be used by components in the opentelemetry-collector-contrib repository, consider placing it on the internal folder instead.
This will reduce our public API footprint and maintenance burden.

Overview (`testbed/README.md`) Copy Section

OpenTelemetry Collector Testbed

Testbed is a controlled environment and tools for conducting end-to-end tests for the Otel Collector,
including reproducible short-term benchmarks, correctness tests, long-running stability tests and
maximum load stress tests.

Usage Copy Chapter

For each type of tests that should have a summary report create a new directory and then a test suite function which utilizes *testing.M. This function should delegate all functionality to testbed.DoTestMain supplying a global instance of testbed.TestResultsSummary to it.

Each test case within the suite should create a testbed.TestCase and supply implementations of each of the various interfaces the NewTestCase function takes as parameters.

DataFlow Copy Chapter

testbed.TestCase uses LoadGenerator and MockBackend to further encapsulate pluggable components. LoadGenerator's reference implementation ProviderSender, created via testbed.NewLoadGenerator() further encapsulates DataProvider and DataSender in order to generate and send data. Any type satisfying the LoadGenerator interface can be used with testbed.NewLoadGeneratorTestCase() to exercise functionality beyond the single component limitation of ProviderSender. MockBackend further encapsulates DataReceiver and provides consume functionality.

For instance, if using the existing end-to-end test, the general dataflow can be (Note that MockBackend does not really have a consumer instance, only to make it intuitive, this diagram draws it a separate module):

Pluggable Test Components Copy Chapter

• DataProvider - Generates test data to send to receiver under test.
  • PerfTestDataProvider - Implementation of the DataProvider for use in performance tests. Tracing IDs are based on the incremented batch and data items counters.
  • GoldenDataProvider - Implementation of DataProvider for use in correctness tests. Provides data from the "Golden" dataset generated using pairwise combinatorial testing techniques.
• DataSender - Sends data to the collector instance under test.
  • OTLPTraceDataSender, OTLPMetricsDataSender, OTLPLogsDataSender - OTLP/gRPC and OTLP/HTTP variants live in the testbed/testbed package.
  • All other senders live in their own sub-package under testbed/datasenders/<component>datasender, e.g. jaegerdatasender.NewJaegerGRPCDataSender (jaeger receiver) or zipkindatasender.NewZipkinDataSender (zipkin receiver). See testbed/datasenders/ for the full set.
• DataReceiver - Receives data from the collector instance under test and stores it for use in test assertions.
  • OTLPDataReceiver, OTLPHTTPDataReceiver - live in the testbed/testbed package.
  • All other receivers live in their own sub-package under testbed/datareceivers/<component>datareceiver, e.g. jaegerdatareceiver.NewJaegerDataReceiver or zipkindatareceiver.NewZipkinDataReceiver. See testbed/datareceivers/ for the full set.
• OtelcolRunner - Configures, starts and stops one or more instances of otelcol which will be the subject of testing being executed.
  • ChildProcess - Implementation of OtelcolRunner runs a single otelcol as a child process on the same machine as the test executor.
  • InProcessCollector - Implementation of OtelcolRunner runs a single otelcol as a go routine within the same process as the test executor.
• TestCaseValidator - Validates and reports on test results.
  • PerfTestValidator - Implementation of TestCaseValidator for test suites using PerformanceResults for summarizing results.
  • CorrectnessTestValidator - Implementation of TestCaseValidator for test suites using CorrectnessResults for summarizing results.
• TestResultsSummary - Records itemized test case results plus a summary of one category of testing.
  • PerformanceResults - Implementation of TestResultsSummary with fields suitable for reporting performance test results.
  • CorrectnessResults - Implementation of TestResultsSummary with fields suitable for reporting data translation correctness test results.

Adding New Receiver and/or Exporters to the testbed Copy Chapter

New DataSender and DataReceiver implementations live in their own sub-package
under testbed/datasenders/<component>datasender and
testbed/datareceivers/<component>datareceiver. Connector helpers follow the
same pattern under testbed/dataconnectors/<component>dataconnector. Keeping
one component per package means each test binary only compiles the senders,
receivers, and connectors it actually exercises.

Generally, when designing a test for new exporter and receiver components, developers should mainly focus on designing and implementing the components with yellow background in the diagram above as the other components are implemented by the testbed framework:

• DataSender - This part should provide below interfaces for testing purpose:

  • Start() - Start sender and connect to the configured endpoint. Must be called before sending data.
  • Flush() - Send any accumulated data.
  • GetCollectorPort() - Return the port to which this sender will send data.
  • GenConfigYAMLStr() - Generate a config string to place in receiver part of collector config so that it can receive data from this sender.
  • ProtocolName() - Return protocol name to use in collector config pipeline.

• DataReceiver - This part should provide below interfaces for testing purpose:

  • Start() - Start receiver.
  • Stop() - Stop receiver.
  • GenConfigYAMLStr() - Generate a config string to place in exporter part of collector config so that it can send data to this receiver.
  • ProtocolName() - Return protocol name to use in collector config pipeline.

• Testing - This part may vary from what kind of testing developers would like to do. In existing implementation, we can refer to End-to-End testing, Metrics testing, Traces testing, Correctness Traces testing, and Correctness Metrics testing. For instance, if developers would like to design a trace test for a new exporter and receiver:

  • go Copy

    func TestTrace10kSPS(t *testing.T) {
        tests := []struct {
            name         string
            sender       testbed.DataSender
            receiver     testbed.DataReceiver
            resourceSpec testbed.ResourceSpec
        }{
            {
                "NewExporterOrReceiver",
                xxxdatasender.NewXXXDataSender(testbed.DefaultHost, testutil.GetAvailablePort(t)),
                xxxdatareceiver.NewXXXDataReceiver(testutil.GetAvailablePort(t)),
                testbed.ResourceSpec{
                    ExpectedMaxCPU: XX,
                    ExpectedMaxRAM: XX,
                },
            },
            ...
        }
        processors := []ProcessorNameAndConfigBody{
            {
                Name: "batch",
                Body: `
      batch:
    `,
            },
        }
        for _, test := range tests {
            t.Run(test.name, func(t *testing.T) {
                Scenario10kItemsPerSecond(
                    t,
                    test.sender,
                    test.receiver,
                    test.resourceSpec,
                    performanceResultsSummary,
                    processors,
                )
            })
        }
    }

Running the tests Copy Chapter

To run the tests, use the e2e-test Makefile target, which will compile the Collector and run the end-to-end test suites against it.

Run all tests Copy Section

Run the following at the root of the repo:

make e2e-test

Run a particular test suite Copy Section

Run the following at the root of the repo:

TESTS_DIR=correctnesstests/metrics make e2e-test

Advanced usage Copy Section

A Makefile is also located at testbed/Makefile that offers targets to directly run certain test suites. Note that these targets will not compile the Collector before running.

Overview (`cmd/codecovgen/README.md`) Copy Section

codecovgen

This tool generates the component list in the .codecov.yml file.
Each Go module gets its own component, which allows to have code coverage results on a per-component basis.

Overview (`cmd/githubgen/README.md`) Copy Section

⚠️ This executable has moved to opentelemetry-go-build-tools.

See https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/37294.

Overview (`cmd/golden/README.md`) Copy Section

Golden tester

Overview (`cmd/opampsupervisor/README.md`) Copy Section

OpAMP Supervisor for the OpenTelemetry Collector

This is an implementation of an OpAMP Supervisor that runs a Collector instance using configuration provided from an OpAMP server. This implementation
is following a design specified here.
The design is still undergoing changes, and as such this implementation may change as well.

Binary and container images for the Supervisor are available under tags starting with cmd/opampsupervisor here.

More information Copy Chapter

If you'd like to learn more about OpAMP, see the
OpAMP specification.
For production-ready client and server implementations in Go, see the
OpAMP Go repository.

If you have questions or would like to participate, we'd love to hear from you.
There is a regular SIG meeting, see the
meeting notes
for more information. You can also get in touch with us in the
#otel-opamp channel on
the CNCF Slack workspace.

Installing the Supervisor Copy Chapter

Binary Copy Section

Binaries are available as downloadable assets from OpenTelemetry Collector releases with tags using the prefix cmd/opampsupervisor. You can download the file containing the binary for your OS and architecture, and install it on your machine manually.

Linux/macOS

Replace <VERSION> with your desired Supervisor version, and <OS> and <ARCH> with your platform-specific information:

curl --proto '=https' --tlsv1.2 -fL -o opampsupervisor \
"https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fopampsupervisor%2Fv<VERSION>/opampsupervisor_<VERSION>_<OS>_<ARCH>"
chmod +x opampsupervisor

Windows

Replace <VERSION> with your desired Supervisor version:

Invoke-WebRequest -Uri "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fopampsupervisor%2Fv<VERSION>/opampsupervisor_<VERSION>_windows_amd64.exe" -OutFile "opampsupervisor.exe"
Unblock-File -Path "opampsupervisor.exe"

Container image Copy Section

Container images are available for download from GitHub Container registry:

docker pull ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-opampsupervisor

Using the Supervisor Copy Chapter

To use the Supervisor, you will need four things:

1. A Supervisor binary, which can be obtained through the installation step above.
2. A Collector binary that you would like to control through the Supervisor.
3. A Supervisor config file. See examples here.
4. A running OpAMP server.

To test the Supervisor with an example server, download the
opamp-go repository, and run the
OpAMP example server in the internal/examples/server directory.

git clone [email protected]:open-telemetry/opamp-go.git
cd opamp-go/internal/examples/server
go run .

Visit localhost:4321 to verify that the server is running.

In a separate directory, create a Supervisor configuration file named supervisor.yaml. Copy one of the existing examples or create a new configuration. The agent::executable should be set to the path of your Collector binary:

agent:
  executable: <COLLECTOR_BINARY>

Start the Supervisor:

./opampsupervisor --config=supervisor.yaml

The supervisor's own log encoding is configured under telemetry.logs.encoding.
Use console for plain-text logs with ISO8601 timestamps. Leave the field
unset (or set it to json) to keep the default JSON output.

telemetry:
  logs:
    encoding: console

The supervisor can also configure resource detection for its own telemetry
using the experimental telemetry.resource.detection/development field.

telemetry:
  resource:
    detection/development:
      detectors:
        - host: {}
        - process: {}

Supported detector entries are container, host, process, and service.

Do not set the OTEL_EXPERIMENTAL_CONFIG_FILE environment variable for the
Supervisor process. xotelconf.NewSDK uses that file to configure the SDK and
it supersedes the configuration passed by the Supervisor. If the variable is
set (for example to control the Collector's resource config), it can also
override the Supervisor's own resource detection and telemetry resource
settings. The Supervisor logs a warning at startup when the variable is set
while telemetry.resource.detection/development is configured.

Visit localhost:4321 again to verify that your Collector appears in the Agents list.

Collector crash log snippets Copy Chapter

The Supervisor can include the tail of the Collector's logs in health and remote config failure messages when the Collector process exits unexpectedly. This is disabled by default because Collector logs may contain sensitive data. To enable it, set agent::collector_crash_log_snippet_kib to the maximum number of log KiB to include. A value of 0 disables the feature. 4 is a reasonable starting point. The maximum value is 1024 (1 MiB).

agent:
  collector_crash_log_snippet_kib: 4

Persistent data storage Copy Chapter

The supervisor persists some data to disk in order to mantain state between restarts. The directory where this data is stored may be specified via the supervisor configuration:

storage:
  directory: "/path/to/storage/dir"

By default, the supervisor will use /var/lib/otelcol/supervisor on posix systems, and %ProgramData%/Otelcol/Supervisor on Windows.

This directory will be created on supervisor startup if it does not exist.

Healthcheck Copy Chapter

The Supervisor can be configured to expose a healthcheck endpoint that can be used to determine whether the Supervisor is running and healthy. This can be configured in the Supervisor configuration file:

healthcheck:
  endpoint: localhost:13133

At the moment this healtcheck checks that the following conditions are met:

• The Supervisor is able to persist state to disk.
• The Supervisor is able to generate the agent's configuration.

For more details on the healthcheck configuration, see the see the full list of ServerConfig options.

Note that the healthceck endpoint is not enabled by default. To enable it, you must explicitly set at least the endpoint field in the configuration.

Startup Fallback Configuration Copy Chapter

The Supervisor supports a startup fallback configuration mechanism that provides resilience when the OpAMP server is unreachable at startup and there's no previous configuration state persisted in disk. This is useful for ensuring the Collector can start with a known-good configuration during network outages or server maintenance. When the Supervisor successfully connects to the OpAMP server, the regular configuration (indicated by agent::config_files) is restored and any potential remote configuration received from the OpAMP server is applied.

If, when the Supervisor starts, there's previous configuration state persisted in disk, the Supervisor will use it to start the Collector.

Configuration Options Copy Section

The following options can be configured under the agent section:

agent:
  startup_fallback_configs:
  - /path/to/fallback_config.yaml

Important Notes Copy Section

• The startup fallback configuration is a set of standalone configuration files. It is intentionally not merged with the agent::config_files setting. This ensures predictable behavior without dependencies on other configuration files.

• The OpAMP extension and "own telemetry" configuration are automatically added to the fallback configuration. This maintains the Collector's own telemetry flowing and the communication between Supervisor and Collector.

• The Supervisor will continue attempting to reconnect to the OpAMP server even while using the fallback configuration.

Example Copy Section

See examples/supervisor_fallback.yaml for a complete example configuration with fallback enabled.

Automatic Remote Configuration Rollback Copy Chapter

The Supervisor supports automatic rollback when a remote configuration received from
the OpAMP backend causes the Collector to fail to start correctly. This feature is
disabled by default and can be enabled in the Supervisor configuration file:

agent:
  automatic_config_rollback: true

When enabled, the Supervisor will watch the health of the Collector after applying remote
configurations received from the OpAMP backend and it will cache the last working one in
disk, besides the last received. If a remote configurations causes the Collector
to fail to start correctly, the Supervisor will automatically swap to the last working one
and restart the Collector. During this process, a new "APPLIED" status will be reported for
the last working configuration.

Status Copy Chapter

The OpenTelemetry OpAMP Supervisor is intended to be the reference
implementation of an OpAMP Supervisor, and as such will support all OpAMP
capabilities. Additionally, it follows a design document for the features it
intends to support.

For a list of open issues related to the Supervisor, see these issues.

Key:

✅: Fully implemented

⚠️: Implemented with caveats

OpAMP capabilities Copy Section

Supervisor specification features Copy Section

Overview (`cmd/telemetrygen/README.md`) Copy Section

Telemetry generator for OpenTelemetry

This utility simulates a client generating traces, metrics, and logs. It is useful for testing and demonstration purposes.

Installing Copy Chapter

To install the latest version run the following command:

go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest

Check the go install reference to install specific versions.

Docker build Copy Section

You can build locally the Docker image with:

make docker-telemetrygen

Using github actions, we also push a docker image on commit to main or on release to this Github docker registry.

Running Copy Chapter

First, you'll need an OpenTelemetry Collector to receive the telemetry data. Follow the project's instructions for a detailed setting up guide. The following configuration file should be sufficient:

config.yaml:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [debug]
    metrics:
      receivers: [otlp]
      exporters: [debug]
    traces:
      receivers: [otlp]
      exporters: [debug]

Starting OpenTelemetry collector via docker:

docker run -p 4317:4317 -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.149.0

Other options for running the collector are documented here https://opentelemetry.io/docs/collector/getting-started/

Export timeout Copy Chapter

The maximum time to wait for signals to reach the destination can be configured with the following flag:

--timeout=10s #Defaults to 10s

Batching signals Copy Chapter

All telemetry signals have batching capability configurable with the following flags:

--batch=false #Defaults to true, no flag needed for batching

--batch-size=1000 #Defaults to 100

Once the OpenTelemetry Collector instance is up and running, run telemetrygen for your desired telemetry:

Traces Copy Section

telemetrygen traces --otlp-insecure --duration 5s

Or, to generate a specific number of traces:

telemetrygen traces --otlp-insecure --traces 1

Or, to generate traces continuously (use --duration inf with --rate for throttling):

telemetrygen traces --otlp-insecure --duration inf --rate 0.1

To send traces in secure connection, see examples/secure-tracing

Check telemetrygen traces --help for all the options.

Logs Copy Section

telemetrygen logs --otlp-insecure --duration 5s

Or, to generate a specific number of logs:

telemetrygen logs --otlp-insecure --logs 1

Or, to generate logs continuously:

telemetrygen logs --otlp-insecure --duration inf --rate 0.1

To send logs in secure connection, see examples/secure-tracing

Check telemetrygen logs --help for all the options.

Metrics Copy Section

telemetrygen metrics --otlp-insecure --duration 5s

Or, to generate a specific number of metrics:

telemetrygen metrics --otlp-insecure --metrics 1

Or, to generate metrics continuously:

telemetrygen metrics --otlp-insecure --duration inf --rate 0.1

To send metrics in secure connection, see examples/secure-tracing

Check telemetrygen metrics --help for all the options.

Attributes Copy Chapter

Custom attributes can be added at two different levels, using two different flags:

Because service.name is a resource attribute, --otlp-attributes service.name="my-service" overrides the --service flag.

Both flags accept the same value formats: key="value", key=true, key=false, key=, and key=["value1", "value2", ...] for slices. All items of a slice must have the same type. Both flags may be repeated to set multiple attributes.

String values must keep their double quotes when they reach telemetrygen, so escape them (or wrap the whole argument in single quotes) to prevent the shell from removing them:

telemetrygen traces --otlp-insecure --traces 1 \
  --otlp-attributes service.name=\"my-service\" \
  --otlp-attributes host.name=\"host-1\" \
  --telemetry-attributes http.request.method=\"GET\" \
  --telemetry-attributes http.response.status_code=200

This generates a resource carrying service.name and host.name, and spans that each carry http.request.method and http.response.status_code.

Overview (`config/configdbauth/README.md`) Copy Section

Database Authentication Configuration

This package provides the configuration for wiring a connection-oriented
component to a dbauth provider that supplies a credential at the moment the
component opens a connection — for example, a database receiver that needs a
username and password (or a short-lived token) to dial its database.

It is the configuration half of the db_auth framework. The provider
interface it resolves (Provider, Credential, Request) lives in a
separate, dependency-light module,
extension/dbauth, so a provider extension can
implement the interface without taking on this package's confmap dependency

How a provider is declared and selected Copy Chapter

A credential provider is a Collector extension that also implements
dbauth.Provider. It is declared once (and listed in service.extensions) with
its provider-wide configuration, then each component references it by component
ID inside its db_auth block — the value of the block is the provider's
component ID:

extensions:
  aws_iam:                       # declared once, with provider-wide config
    region: us-east-2
  aws_iam/west:                  # a second instance for a different region
    region: us-west-2

receivers:
  postgresql/this:
    endpoint: this-db:5432
    username: monitor
    db_auth: aws_iam             # reference the extension by ID
  postgresql/another:
    endpoint: another-db:5432
    username: reader
    db_auth: aws_iam/west        # reference a differently-configured instance

service:
  extensions: [aws_iam, aws_iam/west]

The extension holds all provider-wide configuration (here, the AWS region). To
vary that configuration across components, declare multiple extension instances
(aws_iam, aws_iam/west) and point each component at the one it needs. There
is no inline override in the db_auth block — the value is only a reference.

The per-connection inputs a provider needs — the endpoint to connect to, the
database user — are not configured on the extension or repeated in the
db_auth block. The consuming component already knows them (its own
endpoint and username) and passes them to the provider with each
GetCredential call as a dbauth.Request. So one declared extension serves many
components that differ only by endpoint/user, with no per-database extension
instance and nothing repeated.

This means:

• A provider is declared once per configuration (one extension instance in
  the collector build); any receiver can reference it by ID, with no per-receiver
  provider list to maintain, and new providers require no receiver code changes.
• Databases that share provider config reference one instance; a database that
  needs a different provider-wide value references a separate declared instance.
• Resolution mirrors config/configauth: the component finds the provider in
  host.GetExtensions() at Start(), not via an init() registry.

The credential and the request Copy Chapter

The provider interface, Credential, and Request are defined in
extension/dbauth:

type Provider interface {
    GetCredential(ctx context.Context, req Request) (*Credential, error)
}

type Request struct {
    Endpoint string // host:port the credential is for
    Username string // the consumer's configured username
}

type Credential struct {
    Username *string    // nil: use the consumer's configured username
    Secret   string     // opaque: a password or a minted token
    NotAfter *time.Time // nil: no expiry applies; advisory only
}

Request carries the per-connection inputs the consumer supplies on every call;
a provider uses the fields it needs (AWS IAM mints a token scoped to Endpoint
for Username) and ignores the rest. Provider-wide configuration (such as the
AWS region) lives on the extension's own config, not in the request.

The Credential carries credential material and nothing about how to apply
it — the consuming component decides where the secret goes (for a SQL driver,
usually the password slot of a connection string). The pointer fields are
load-bearing:

• Username == nil means the provider supplies no username and the consumer
  should fall back to its own configured username. A non-nil pointer (including a
  pointer to the empty string) means the provider generated the username and the
  consumer must use it. Dynamic providers such as Vault mint both a username and a
  secret per lease, which is why this is a pointer rather than a plain string.
• NotAfter == nil means no expiry applies (a static password). When set, it is
  an advisory hint for observability — the provider owns refresh-before-expiry, so
  consumers need not act on it.

Genericity Copy Chapter

The same Credential and the same Provider interface cover every connection
credential shape without an interface change. How each maps onto the struct:

Overview (`connector/countconnector/README.md`) Copy Section

Count Connector

Supported Pipeline Types Copy Chapter

The count connector can be used to count spans, span events, metrics, data points, and log records.

Configuration Copy Chapter

If you are not already familiar with connectors, you may find it helpful to first visit the Connectors README.

Default Configuration Copy Section

The count connector may be used without any configuration settings. The following table describes the
default behavior of the connector.

For example, in the following configuration the connector will count spans and span events from the traces/in
pipeline and emit metrics called trace.span.count and trace.span.event.count onto the metrics/out pipeline.

receivers:
  foo:
exporters:
  bar:
connectors:
  count:

service:
  pipelines:
    traces/in:
      receivers: [foo]
      exporters: [count]
    metrics/out:
      receivers: [count]
      exporters: [bar]

Custom Counts Copy Section

Optionally, emit custom counts by defining metrics under one or more of the following sections:

• spans
• spanevents
• metrics
• datapoints
• logs

Optionally, specify a description for the metric.

Note: If any custom metrics are defined for a data type, the default metric will not be emitted.

Conditions

Conditions may be specified for custom metrics. If specified, data that matches any one
of the conditions will be counted. i.e. Conditions are ORed together.

receivers:
  foo:
exporters:
  bar:
connectors:
  count:
    spanevents:
      my.prod.event.count:
        description: The number of span events from my prod environment.
        conditions:
          - 'spanevent.attributes["env"] == "prod"'
          - 'spanevent.name == "prodevent"'

Attributes

spans, spanevents, datapoints, and logs may be counted according to attributes. In such cases, attribute precedence follows this order:
span(logRecord, DataPoint, profile) attributes > scope attributes > resource attributes.

If attributes are specified for custom metrics, a separate count will be generated for each unique
set of attribute values. Each count will be emitted as a data point on the same metric.

Optionally, include a default_value for an attribute, to count data that does not contain the attribute. The default_value value can be of type string, integer, or float.

receivers:
  foo:
exporters:
  bar:
connectors:
  count:
    logs:
      my.log.count:
        description: The number of logs from each environment.
        attributes:
          - key: env
            default_value: unspecified_environment

Example Usage Copy Section

Count spans and span events, only exporting the count metrics.

receivers:
  foo:
exporters:
  bar:
connectors:
  count:
service:
  pipelines:
    traces:
      receivers: [foo]
      exporters: [count]
    metrics:
      receivers: [count]
      exporters: [bar]

Count spans and span events, exporting both the original traces and the count metrics.

receivers:
  foo:
exporters:
  bar/traces_backend:
  bar/metrics_backend:
connectors:
  count:
service:
  pipelines:
    traces:
      receivers: [foo]
      exporters: [bar/traces_backend, count]
    metrics:
      receivers: [count]
      exporters: [bar/metrics_backend]

Count spans, span events, metrics, data points, and log records, exporting count metrics to a separate backend.

receivers:
  foo/traces:
  foo/metrics:
  foo/logs:
exporters:
  bar/all_types:
  bar/counts_only:
connectors:
  count:
service:
  pipelines:
    traces:
      receivers: [foo/traces]
      exporters: [bar/all_types, count]
    metrics:
      receivers: [foo/metrics]
      exporters: [bar/all_types, count]
    logs:
      receivers: [foo/logs]
      exporters: [bar/all_types, count]
    metrics/counts:
      receivers: [count]
      exporters: [bar/counts_only]

Count logs with a severity of ERROR or higher.

receivers:
  foo:
exporters:
  bar:
connectors:
  count:
    logs:
      my.error.log.count:
        description: Error+ logs.
        conditions:
          - `severity_number >= SEVERITY_NUMBER_ERROR`
service:
  pipelines:
    logs:
      receivers: [foo]
      exporters: [count]
    metrics:
      receivers: [count]
      exporters: [bar]

Count logs with a severity of ERROR or higher. Maintain a separate count for each environment.

receivers:
  foo:
exporters:
  bar:
connectors:
  count:
    logs:
      my.error.log.count:
        description: Error+ logs.
        conditions:
          - `severity_number >= SEVERITY_NUMBER_ERROR`
        attributes:
          - key: env
service:
  pipelines:
    logs:
      receivers: [foo]
      exporters: [count]
    metrics:
      receivers: [count]
      exporters: [bar]

Count all spans and span events (default behavior). Count metrics and data points based on the env attribute.

receivers:
  foo/traces:
  foo/metrics:
  foo/logs:
exporters:
  bar/all_types:
  bar/counts_only:
connectors:
  count:
    metrics:
      my.prod.metric.count:
        conditions:
         - `attributes["env"] == "prod"
      my.test.metric.count:
        conditions:
         - `attributes["env"] == "test"
    datapoints:
      my.prod.datapoint.count:
        conditions:
         - `attributes["env"] == "prod"
      my.test.datapoint.count:
        conditions:
         - `attributes["env"] == "test"
service:
  pipelines:
    traces:
      receivers: [foo/traces]
      exporters: [bar/all_types, count]
    metrics:
      receivers: [foo/metrics]
      exporters: [bar/all_types, count]
    metrics/counts:
      receivers: [count]
      exporters: [bar/counts_only]

Overview (`connector/datadogconnector/README.md`) Copy Section

Datadog Connector

Supported Pipeline Types Copy Chapter

Description Copy Chapter

The Datadog Connector is a connector component that derives APM statistics, in the form of metrics, from service traces, for display in the Datadog APM product. This component is required for trace-emitting services and their statistics to appear in Datadog APM.

The Datadog connector can also forward the traces passed into it into another trace pipeline. Notably, if you plan to sample your traces with the tailsamplingprocessor or the probabilisticsamplerprocessor, you should place the Datadog connector upstream to ensure that the metrics are computed before sampling, ensuring their accuracy. An example is given below.

Usage Copy Chapter

# ...
processors:
  # ...
  probabilistic_sampler:
    sampling_percentage: 20

connectors:
  # add the "datadog" connector definition and further configurations
  datadog/connector:

exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
    sending_queue:
      batch:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [datadog/connector]

    traces/2: # this pipeline uses sampling
      receivers: [datadog/connector]
      processors: [probabilistic_sampler]
      exporters: [datadog]

    metrics:
      receivers: [datadog/connector]
      exporters: [datadog]

In this example configuration, incoming traces are received through OTLP, and processed by the Datadog connector in the traces pipeline. The traces are then forwarded to the traces/2 pipeline, where a sample of them is exported to Datadog. In parallel, the APM stats computed from the full stream of traces are sent to the metrics pipeline, where they are exported to Datadog as well.

Configurations Copy Chapter

/* Detailed source-code truncated for AI context efficiency. */

Warnings Copy Chapter

• Statefulness: The datadog connector aggregates trace metrics into buckets defined by bucket_interval before flushing them, keeping this aggregation state in memory on a single collector instance. All spans that should be aggregated together must be routed to the same collector instance, or the resulting trace metrics will be incomplete or inconsistent.

Overview (`connector/exceptionsconnector/README.md`) Copy Section

Exceptions Connector

Supported Pipeline Types Copy Chapter

Overview Copy Chapter

Generate metrics and logs from recorded application exceptions associated with spans.

Each metric and log will have at least the following dimensions:

• Service name
• Span name
• Span kind
• Status code

With the provided default config, each metric and log will also have the following dimensions:

• Exception message
• Exception type

Each log will additionally have the following attributes:

• Exception stacktrace
• Span attributes. If you want to filter out some attributes (like only copying HTTP attributes starting with http.) use the transform processor.

Configurations Copy Chapter

If you are not already familiar with connectors, you may find it helpful to first visit the Connectors README.

The following settings can be optionally configured:

• dimensions: the list of dimensions to add together with the default dimensions defined above.

  Each additional dimension is defined with a name which is looked up in the span's collection of attributes or resource attributes.

  The provided default config includes exception.type and exception.message as additional dimensions.

• exemplars: Use to configure how to attach exemplars to metrics.

  • enabled (default: false): enabling will add spans as Exemplars.

Examples Copy Chapter

The following is a simple example usage of the exceptions connector.

receivers:
  nop:

exporters:
  nop:

connectors:
  exceptions:

service:
  pipelines:
    traces:
      receivers: [nop]
      exporters: [exceptions]
    metrics:
      receivers: [exceptions]
      exporters: [nop]
    logs:
      receivers: [exceptions]
      exporters: [nop]

The following is a more complex example usage of the exceptions connector using Prometheus and Loki as exporters.

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  prometheus_remote_write:
    endpoint: http://prometheus:9090/api/v1/write
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

connectors:
  exceptions:

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [exceptions]
    metrics:
      receivers: [exceptions]
      exporters: [prometheus_remote_write]
    logs:
      receivers: [exceptions]
      exporters: [loki]

The full list of settings exposed for this connector are documented in exceptionsconnector/config.go.

More Examples Copy Section

For more example configuration covering various other use cases, please visit the testdata directory.

Overview (`connector/failoverconnector/README.md`) Copy Section

Failover Connector

Supported Pipeline Types Copy Chapter

Allows for health based routing between trace, metric, and log pipelines depending on the health of target downstream exporters.

Configuration Copy Chapter

If you are not already familiar with connectors, you may find it helpful to first visit the Connectors README.

The following settings are available:

• priority_levels (required): list of pipeline level priorities in a 1 - n configuration, multiple pipelines can sit at a single priority level.

• retry_interval (optional): the frequency at which the pipeline levels will attempt to reestablish connection with all higher priority levels. Default value is 10 minutes. (See Example below for further explanation)

• sending_queue (optional): queues and batches data in the connector before it is routed to a priority level. Supports the common sending_queue settings and is enabled by default.
  We recommend configuring the queue here instead of on the exporters in the priority pipelines, and disabling sending_queue on those exporters.
  Moving the queue to the connector avoids data being stuck in the queue of an unhealthy exporter, where it cannot be failed over to another priority level.

• condition: Conditional statements that decide when the connector should failover. Currently supports below types

  • error:
    • contains: ["network failure", "connection refused"] (required list of non-empty, case-insensitive substrings)

  Only errors whose message contains any configured string trigger failover. Non-matching errors are returned
  to the upstream caller without failing over, and they do not mark the pipeline level unhealthy during
  retry sampling either. When condition is omitted, all errors trigger failover.

The connector intakes a list of priority_levels each of which can contain multiple pipelines.
If any pipeline at a stable level fails, the level is considered unhealthy and the connector will move down one priority level and route all data to the new level (assuming it is stable).

The connector will periodically try to reestablish a stable connection with the higher priority levels. retry_interval will be the frequency at which the connector will try to iterate through all unhealthy higher priority levels.

Configuration Example:

connectors:
  failover:
    priority_levels:
      - [traces/first, traces/also_first]
      - [traces/second]
      - [traces/third]
    retry_interval: 10s

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [failover]
    traces/first:
      receivers: [failover]
      exporters: [otlp_grpc/first]
    traces/second:
      receivers: [failover]
      exporters: [otlp_grpc/second]
    traces/third:
      receivers: [failover]
      exporters: [otlp_grpc/third]
    traces/also_first:
      receivers: [failover]
      exporters: [otlp_grpc/fourth]

Overview (`connector/grafanacloudconnector/README.md`) Copy Section

Grafana Cloud Connector

Supported Pipeline Types Copy Chapter

Deprecation Notice: The component type has been renamed from grafanacloud to grafana_cloud
to follow the OpenTelemetry snake_case naming convention.
The old name grafanacloud still works but is deprecated and will be removed in a future release.
Please update your configuration to use grafana_cloud.

Overview Copy Chapter

The Grafana Cloud Connector (grafanacloudconnector) is a connector component that can analyze telemetry in pipelines to generate usage metrics for the following Grafana Cloud products:

• Application Observability

Usage Copy Chapter

connectors:
  grafana_cloud/connector:

extensions:
  basicauth/grafana_cloud:
    client_auth:
      username: "${env:GRAFANA_CLOUD_INSTANCE_ID}"
      password: "${env:GRAFANA_CLOUD_API_KEY}"

exporters:
  otlphttp/grafana_cloud:
    endpoint: "${env:GRAFANA_CLOUD_OTLP_ENDPOINT}"
    auth:
      authenticator: basicauth/grafana_cloud

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/grafana_cloud, grafana_cloud/connector]

    metrics:
      receivers: [otlp, grafana_cloud/connector]
      exporters: [otlphttp/grafana_cloud]

In this example configuration, usage metrics for Grafana Cloud Application Observability are derived from incoming spans.

Configuration Copy Chapter

connectors:
  grafanacloud/connector:
    ## @param host_identifiers List of span attributes used to identify the host running the workloads.
    ## The first matching attribute is used.
    # host_identifiers: ["k8s.node.name", "host.id", "k8s.node.uid"]
    ## @param metrics_flush_interval Interval at which host metrics are flushed.
    ## Valid values are between 15s and 5m.
    # metrics_flush_interval: 60s

Example configuration for the component Copy Section

connectors:
  grafana_cloud:
    host_identifiers: ["mycompany.myHostAttribute", "k8s.node.name", "host.id", "k8s.node.uid"]
    metrics_flush_interval: 60s

This connector will generate a host info metric based on the first host_identifiers resource attribute found on spans. The rest are skipped. Valid flush intervals are between 15s and 5m.

Overview (`connector/metricsaslogsconnector/README.md`) Copy Section

Metrics as Logs Connector

Supported Pipeline Types Copy Chapter

Deprecation Notice: The component type has been renamed from metricsaslogs to metrics_as_logs
to follow the OpenTelemetry snake_case naming convention.
The old name metricsaslogs still works but is deprecated and will be removed in a future release.
Please update your configuration to use metrics_as_logs.

This connector converts OpenTelemetry metrics into logs, creating one log entry per metric data point. Each metric data point is transformed into a structured log record with configurable JSON body format.

Current Limitations Copy Chapter

⚠️ Current implementation discards the following metric features:

• Metric exemplars
• Advanced metadata

These features may be added in future iterations.

Configuration Copy Chapter

The following settings can be optionally configured:

• include_resource_attributes (default = true): Whether to include resource attributes in the generated logs
• include_scope_info (default = true): Whether to include instrumentation scope information in the generated logs

Log Body Format Copy Chapter

The connector always generates log bodies in the following JSON format:

{"metric_name": "$NAME", "value": "$VALUE"}

Where:

• $NAME is the actual metric name
• $VALUE is the metric value (simple values for gauge/sum, complex JSON for histogram/summary)

Example Usage Copy Chapter

Basic Configuration Copy Section

connectors:
  metrics_as_logs:

service:
  pipelines:
    logs:
      receivers: [metrics_as_logs]
      processors: []
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: []
      exporters: [metrics_as_logs]

Advanced Configuration Copy Section

connectors:
  metrics_as_logs:
    include_resource_attributes: false
    include_scope_info: false

Example Metric Conversions Copy Section

For a gauge metric cpu_usage with value 85.2:

{
  "body": {"metric_name": "cpu_usage", "value": "85.2"},
  "attributes": {
    "metric.name": "cpu_usage",
    "metric.type": "Gauge",
    "metric.description": "CPU usage percentage",
    "metric.unit": "%"
  }
}

For a histogram metric request_duration:

{
  "body": {
    "metric_name": "request_duration", 
    "value": "{\"count\":100,\"sum\":1.5,\"bucket_counts\":[10,50,40],\"explicit_bounds\":[0.1,0.5,1.0]}"
  },
  "attributes": {
    "metric.name": "request_duration",
    "metric.type": "Histogram",
    "metric.description": "Request duration in seconds",
    "metric.unit": "s",
    "metric.aggregation_temporality": "Delta"
  }
}

Output Structure Copy Chapter

Each metric data point is converted to a log record with:

• Body: Fixed JSON format: {"metric_name": "$NAME", "value": "$VALUE"}
• Timestamp: Metric data point timestamp
• Observed Timestamp: Metric data point start timestamp (if available)
• Attributes:
  • Original metric data point attributes (labels)
  • metric.name: The metric name
  • metric.type: The metric type (Gauge, Sum, Histogram, etc.)
  • metric.description: Metric description (if available)
  • metric.unit: Metric unit (if available)
  • Additional type-specific attributes:
    • For Sum metrics: metric.is_monotonic, metric.aggregation_temporality
    • For Histogram/ExponentialHistogram: metric.aggregation_temporality
  • Resource attributes (if include_resource_attributes is true)
  • Instrumentation scope information (if include_scope_info is true)

Supported Metric Types Copy Chapter

All OpenTelemetry metric types are supported:

• Gauge: Point-in-time measurements
• Sum: Cumulative or delta measurements
• Histogram: Distribution of measurements with buckets
• Exponential Histogram: Distribution with exponentially sized buckets
• Summary: Distribution with quantile values

Value Encoding Copy Chapter

The value field in the JSON body contains:

• For simple metrics (Gauge, Sum): numeric value as string
• For complex metrics (Histogram, etc.): JSON-encoded object as string

All JSON special characters in metric names and values are properly escaped.

Overview (`connector/otlpjsonconnector/README.md`) Copy Section

OTLP JSON Connector

Supported Pipeline Types Copy Chapter

Deprecation Notice: The component type has been renamed from otlpjson to otlp_json
to follow the OpenTelemetry snake_case naming convention.
The old name otlpjson still works but is deprecated and will be removed in a future release.
Please update your configuration to use otlp_json.

Allows to extract otlpjson data from incoming Logs and specifically the Body field.
The data is written in
Protobuf JSON
encoding
using OpenTelemetry
protocol.

Configuration Copy Chapter

Configuration Example:

receivers:
  file_log:
    include:
      - /var/log/foo.log

exporters:
  debug:

connectors:
  # Deprecated (still works):
  # otlpjson:
  # New name:
  otlp_json:

service:
  pipelines:
    logs/raw:
      receivers: [file_log]
      exporters: [otlpjson]
    metrics/otlp:
      receivers: [otlpjson]
      exporters: [debug]
    logs/otlp:
      receivers: [otlpjson]
      exporters: [debug]
    traces/otlp:
      receivers: [otlpjson]
      exporters: [debug]

Overview (`connector/roundrobinconnector/README.md`) Copy Section

Round-Robin Connector

Supported Pipeline Types Copy Chapter

Deprecation Notice: The component type has been renamed from roundrobin to round_robin
to follow the OpenTelemetry snake_case naming convention.
The old name roundrobin still works but is deprecated and will be removed in a future release.
Please update your configuration to use round_robin.

The round_robin connector can fork pipelines of the same type and equally split the load between them.

Configuration Copy Chapter

If you are not already familiar with connectors, you may find it helpful to first visit the Connectors README.

The roundrobin connector does not have any configuration settings.

receivers:
  otlp:
exporters:
  prometheus_remote_write/1:
  prometheus_remote_write/2:
connectors:
  round_robin:

Preprocess data, then export using multiple exporter instances to scale the throughput if the exporter
does not support scale well (e.g. prometheus_remote_write).

receivers:
  otlp:
processors:
  resource_detection:
exporters:
  prometheus_remote_write/1:
  prometheus_remote_write/2:
connectors:
  round_robin:
service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [resource_detection]
      exporters: [round_robin]
    metrics/1:
      receivers: [round_robin]
      exporters: [prometheus_remote_write/1]
    metrics/2:
      receivers: [round_robin]
      exporters: [prometheus_remote_write/2]

Overview (`connector/routingconnector/README.md`) Copy Section

Routing Connector

Supported Pipeline Types Copy Chapter

Routes logs, metrics or traces based on resource attributes to specific pipelines using OpenTelemetry Transformation Language (OTTL) statements as routing conditions.

Configuration Copy Chapter

If you are not already familiar with connectors, you may find it helpful to first visit the Connectors README.

The following settings are available:

• table (required): the routing table for this connector.
• table.condition: the routing condition provided as the OTTL condition. Required if table.statement is not provided. Use context-qualified paths (e.g., resource.attributes["key"], span.attributes["key"]) to automatically infer the context (see Context Inference).
• table.statement: the routing condition provided as the OTTL statement. Required if table.condition is not provided. Generally condition is preferred since it is more terse. May not be used with the deprecated request context.
• table.context (optional): the OTTL Context in which the condition/statement will be evaluated. Deprecated: request — use otelcol.client.metadata or otelcol.grpc.metadata paths instead (see Limitations). In most cases this field should be omitted; the context is inferred automatically from context-qualified paths. If specified, it takes precedence over inference.
• table.action (optional, default: move): determines what happens to the data when the routing condition is met. Valid values are move and copy.
  • move: Matched data is moved to the target pipeline(s) and removed from subsequent route evaluation. This is the default behavior.
  • copy: Matched data is copied to the target pipeline(s) but remains available for evaluation by subsequent routes. This allows the same data to be routed to multiple pipelines.
• table.pipelines (required): the list of pipelines to use when the routing condition is met.
• default_pipelines (optional): contains the list of pipelines to use when a record does not meet any of specified conditions.
• error_mode (optional): determines how errors returned from OTTL statements are handled. Valid values are propagate, ignore and silent. If ignore or silent is used and a statement's condition has an error then the payload will be routed to the default pipelines. When silent is used the error is not logged. If not supplied, ignore is used.

Context Inference Copy Section

The routing connector supports OTTL context inference, allowing you to write clearer and more maintainable routing conditions using context-qualified paths. This is the recommended approach for specifying routing conditions.

- condition: resource.attributes["env"] == "prod"
  pipelines: [logs/prod]
- condition: span.attributes["http.method"] == "GET"
  pipelines: [traces/http]
- condition: log.severity_text == "ERROR"
  pipelines: [logs/errors]

This approach makes it immediately clear which attributes you're accessing without needing a separate context field.

Supported contexts Copy Section

The otelcol.client.metadata and otelcol.grpc.metadata paths provide access to incoming HTTP and gRPC request metadata respectively, and are valid in all signal contexts.

Limitations Copy Section

• Deprecated: The request context is deprecated. Use otelcol.client.metadata["key"] (HTTP/client metadata) or otelcol.grpc.metadata["key"] (gRPC metadata) paths instead. These are supported in all signal contexts. A warning is logged when the request context is used. The request context only supports the condition field with a very limited grammar: request["key"] == "value" or request["key"] != "value".
• When using context inference without an explicit context field, the inferred context must be compatible with the pipeline signal type (e.g., span context can only be used in traces pipelines).

Supported [OTTL] functions Copy Section

• Standard OTTL Converter Functions
• delete_key
• delete_matching_keys

Additional Settings Copy Chapter

The full list of settings exposed for this connector are documented in config.go with detailed sample configuration files:

• logs
• metrics
• traces

Examples Copy Chapter

[!NOTE]
The examples below use context-qualified paths, which is the recommended configuration style. The explicit context field is still supported for backward compatibility but is no longer the primary documentation style. See Context Inference for details.

Route logs based on tenant Copy Section

receivers:
    otlp:

exporters:
  file/other:
    path: ./other.log
  file/acme:
    path: ./acme.log
  file/ecorp:
    path: ./ecorp.log

connectors:
  routing:
    default_pipelines: [logs/other]
    table:
      - context: resource
        condition: otelcol.client.metadata["X-Tenant"][0] == "acme"
        pipelines: [logs/acme]
      - context: resource
        condition: otelcol.client.metadata["X-Tenant"][0] == "ecorp"
        pipelines: [logs/ecorp]

service:
  pipelines:
    logs/in:
      receivers: [otlp]
      exporters: [routing]
    logs/acme:
      receivers: [routing]
      exporters: [file/acme]
    logs/ecorp:
      receivers: [routing]
      exporters: [file/ecorp]
    logs/other:
      receivers: [routing]
      exporters: [file/other]

Route logs based on region Copy Section

receivers:
    otlp:

exporters:
  file/other:
    path: ./other.log
  file/east:
    path: ./east.log
  file/west:
    path: ./west.log

connectors:
  routing:
    default_pipelines: [logs/other]
    table:
      - condition: log.attributes["region"] == "east"
        pipelines: [logs/east]
      - condition: log.attributes["region"] == "west"
        pipelines: [logs/west]

service:
  pipelines:
    logs/in:
      receivers: [otlp]
      exporters: [routing]
    logs/east:
      receivers: [routing]
      exporters: [file/east]
    logs/west:
      receivers: [routing]
      exporters: [file/west]
    logs/other:
      receivers: [routing]
      exporters: [file/other]

Route low-severity logs to cheap storage, remainder by service name Copy Section

receivers:
    otlp:

exporters:
  file/cheap:
    path: ./cheap.log
  file/service1:
    path: ./service1-important.log
  file/service2:
    path: ./service2-important.log

connectors:
  routing:
    table:
      - condition: log.severity_number < SEVERITY_NUMBER_ERROR
        pipelines: [logs/cheap]
      - condition: resource.attributes["service.name"] == "service1"
        pipelines: [logs/service1]
      - condition: resource.attributes["service.name"] == "service2"
        pipelines: [logs/service2]

service:
  pipelines:
    logs/in:
      receivers: [otlp]
      exporters: [routing]
    logs/cheap:
      receivers: [routing]
      exporters: [file/cheap]
    logs/service1:
      receivers: [routing]
      exporters: [file/service1]
    logs/service2:
      receivers: [routing]
      exporters: [file/service2]

Route low-severity logs to cheap storage, remainder by tenant Copy Section

receivers:
    otlp:

exporters:
  file/cheap:
    path: ./cheap.log
  file/acme:
    path: ./acme.log
  file/ecorp:
    path: ./ecorp.log

connectors:
  routing:
    table:
      - condition: log.severity_number < SEVERITY_NUMBER_ERROR
        pipelines: [logs/cheap]
      - context: resource
        condition: otelcol.client.metadata["X-Tenant"][0] == "acme"
        pipelines: [logs/acme]
      - context: resource
        condition: otelcol.client.metadata["X-Tenant"][0] == "ecorp"
        pipelines: [logs/ecorp]

service:
  pipelines:
    logs/in:
      receivers: [otlp]
      exporters: [routing]
    logs/cheap:
      receivers: [routing]
      exporters: [file/cheap]
    logs/acme:
      receivers: [routing]
      exporters: [file/acme]
    logs/ecorp:
      receivers: [routing]
      exporters: [file/ecorp]

Route all logs to an archive, while also routing errors using `action: copy` Copy Section

Conditions with no OTTL paths (such as the literal "true") cannot be inferred, so an explicit context field is required.

receivers:
    otlp:

exporters:
  file/archive:
    path: ./archive.log
  file/errors:
    path: ./errors.log
  file/other:
    path: ./other.log

connectors:
  routing:
    default_pipelines: [logs/other]
    table:
      - context: resource
        condition: "true"
        action: copy
        pipelines: [logs/archive]
      - condition: log.severity_number >= SEVERITY_NUMBER_ERROR
        pipelines: [logs/errors]

service:
  pipelines:
    logs/in:
      receivers: [otlp]
      exporters: [routing]
    logs/archive:
      receivers: [routing]
      exporters: [file/archive]
    logs/errors:
      receivers: [routing]
      exporters: [file/errors]
    logs/other:
      receivers: [routing]
      exporters: [file/other]

In this example:

• All logs are first copied to the archive pipeline (using action: copy), which means the original data remains available for subsequent route evaluation.
• Error logs are then moved to the errors pipeline (using the default action: move).
• Any remaining logs (non-errors) go to the default pipeline.

Route traces to multiple pipelines using `action: copy` Copy Section

receivers:
    otlp:

exporters:
  file/prod:
    path: ./prod.json
  file/high-latency:
    path: ./high-latency.json
  file/other:
    path: ./other.json

connectors:
  routing:
    default_pipelines: [traces/other]
    table:
      - condition: resource.attributes["env"] == "prod"
        action: copy
        pipelines: [traces/prod]
      - condition: span.attributes["http.duration_ms"] > 1000
        pipelines: [traces/high-latency]

service:
  pipelines:
    traces/in:
      receivers: [otlp]
      exporters: [routing]
    traces/prod:
      receivers: [routing]
      exporters: [file/prod]
    traces/high-latency:
      receivers: [routing]
      exporters: [file/high-latency]
    traces/other:
      receivers: [routing]
      exporters: [file/other]

In this example:

• Production traces are copied to the prod pipeline. Since action: copy is used, the traces remain available for subsequent evaluation.
• High-latency spans (>1000ms) are then moved to the high-latency pipeline. A production trace with high latency will appear in both the prod and high-latency pipelines.
• Remaining traces go to the default pipeline.

`match_once` Copy Chapter

The match_once field was deprecated as of v0.116.0 and removed in v0.120.0.

The following examples demonstrate some strategies for migrating a configuration from match_once.

Example without `default_pipelines` Copy Section

If not using default_pipelines, you may be able to split the router into multiple parallel routers.
In the following example, the "env" and "region" are not directly related.

routing:
  match_once: false
  table:
    - condition: attributes["env"] == "prod"
       pipelines: [ logs/prod ]
    - condition: attributes["env"] == "dev"
       pipelines: [ logs/dev ]
    - condition: attributes["region"] == "east"
       pipelines: [ logs/east ]
    - condition: attributes["region"] == "west"
       pipelines: [ logs/west ]

service:
  pipelines:
    logs/in::exporters: [routing]
    logs/prod::receivers: [routing]
    logs/dev::receivers: [routing]
    logs/east::receivers: [routing]
    logs/west::receivers: [routing]

Therefore, the same behavior can be achieved using separate routers. Listing both routers in the pipeline configuration will
result in each receiving an independent handle to the data. The same data can then match routes in both routers.

routing/env:
  table:
    - condition: resource.attributes["env"] == "prod"
       pipelines: [ logs/prod ]
    - condition: resource.attributes["env"] == "dev"
       pipelines: [ logs/dev ]
routing/region:
  table:
    - condition: resource.attributes["region"] == "east"
       pipelines: [ logs/east ]
    - condition: resource.attributes["region"] == "west"
       pipelines: [ logs/west ]

service:
  pipelines:
    logs/in::exporters: [routing/env, routing/region]
    logs/prod::receivers: [routing/env]
    logs/dev::receivers: [routing/env]
    logs/east::receivers: [routing/region]
    logs/west::receivers: [routing/region]

Example with `default_pipelines` Copy Section

The following example demonstrates strategies for migrating from match_once: true while using default_pipelines.

routing:
  match_once: true
  default_pipelines: [ logs/default ]
  table:
    - condition: resource.attributes["env"] == "prod"
      pipelines: [ logs/prod ]
    - condition: resource.attributes["env"] == "dev"
      pipelines: [ logs/dev ]
    - condition: resource.attributes["region"] == "east"
      pipelines: [ logs/east ]
    - condition: resource.attributes["region"] == "west"
      pipelines: [ logs/west ]

service:
  pipelines:
    logs/in::exporters: [routing]
    logs/default::receivers: [routing]
    logs/prod::receivers: [routing]
    logs/dev::receivers: [routing]
    logs/east::receivers: [routing]
    logs/west::receivers: [routing]

If the number of routes are limited, you may be able to articulate a route for each combination of conditions. This avoids the need to change any pipelines.

routing:
  default_pipelines: [ logs/default ]
  table:
    - condition: resource.attributes["env"] == "prod" and resource.attributes["region"] == "east"
       pipelines: [ logs/prod, logs/east ]
    - condition: resource.attributes["env"] == "prod" and resource.attributes["region"] == "west"
       pipelines: [ logs/prod, logs/west ]
    - condition: resource.attributes["env"] == "dev" and resource.attributes["region"] == "east"
       pipelines: [ logs/dev, logs/east ]
    - condition: resource.attributes["env"] == "dev" and resource.attributes["region"] == "west"
       pipelines: [ logs/dev, logs/west ]

service:
  pipelines:
    logs/in::exporters: [routing]
    logs/default::receivers: [routing]
    logs/prod::receivers: [routing]
    logs/dev::receivers: [routing]
    logs/east::receivers: [routing]
    logs/west::receivers: [routing]

A more general solution is to use a layered approach. In this design, the first layer is a single router that sorts data according to whether it matches
any route or no route. This allows the second layer to work without default_pipelines. The downside to this approach is that the set of conditions
in the first and second layers must be kept in sync.

# First layer separates logs that match no routes
routing:
  default_pipelines: [ logs/default ]
  table: # all routes forward to second layer
    - condition: resource.attributes["env"] == "prod"
       pipelines: [ logs/env, logs/region ] 
    - condition: resource.attributes["env"] == "dev"
       pipelines: [ logs/env, logs/region ]
    - condition: resource.attributes["region"] == "east"
       pipelines: [ logs/env, logs/region ]
    - condition: resource.attributes["region"] == "west"
       pipelines: [ logs/env, logs/region ]

# Second layer routes logs based on environment and region
routing/env:
  table:
    - condition: resource.attributes["env"] == "prod"
       pipelines: [ logs/prod ]
    - condition: resource.attributes["env"] == "dev"
       pipelines: [ logs/dev ]
routing/region:
  table:
    - condition: resource.attributes["region"] == "east"
       pipelines: [ logs/east ]
    - condition: resource.attributes["region"] == "west"
       pipelines: [ logs/west ]

service:
  pipelines:
    logs/in::exporters: [routing]
    logs/prod::receivers: [routing/env]
    logs/dev::receivers: [routing/env]
    logs/east::receivers: [routing/region]
    logs/west::receivers: [routing/region]

Overview (`connector/servicegraphconnector/README.md`) Copy Section

Service Graph Connector

Supported Pipeline Types Copy Chapter

Note: The component type has been renamed from servicegraph to service_graph.
The old name still works but is deprecated and will be removed in a future release.
Please update your configuration to use service_graph.

Overview Copy Chapter

The service graphs connector builds a map representing the interrelationships between various services in a system.
The connector will analyse trace data and generate metrics describing the relationship between the services.
These metrics can be used by data visualization apps (e.g. Grafana) to draw a service graph.

Service graphs are useful for a number of use-cases:

• Infer the topology of a distributed system. As distributed systems grow, they become more complex. Service graphs can help you understand the structure of the system.
• Provide a high level overview of the health of your system.
  Service graphs show error rates, latencies, among other relevant data.
• Provide an historic view of a system’s topology.
  Distributed systems change very frequently,
  and service graphs offer a way of seeing how these systems have evolved over time.

This component is based on Grafana Tempo's service graph processor.

How it works Copy Chapter

Service graphs work by inspecting traces and looking for spans with parent-children relationship that represent a request.
The connector uses the OpenTelemetry semantic conventions to detect a myriad of requests.
It currently supports the following requests:

• A direct request between two services where the outgoing and the incoming span must have span.kind client and server respectively.
• A request across a messaging system where the outgoing and the incoming span must have span.kind producer and consumer respectively.
• A database request; in this case the connector looks for spans containing attributes span.kind=client as well as db.name.

Every span that can be paired up to form a request is kept in an in-memory store,
until its corresponding pair span is received or the maximum waiting time has passed.
When either of these conditions are reached, the request is recorded and removed from the local store.

Each emitted metrics series have the client and server label corresponding with the service doing the request and the service receiving the request.

traces_service_graph_request_total{client="app", server="db", connection_type="database"} 20

TLDR: The connector will try to find spans belonging to requests as seen from the client and the server and will create a metric representing an edge in the graph.

Metrics Copy Chapter

The following metrics are emitted by the connector:

Duration is measured both from the client and the server sides.

Possible values for connection_type: unset, messaging_system, or database.

Additional labels can be included using the dimensions configuration option. Those labels will have a prefix to mark where they originate (client or server span kinds).
The client_ prefix relates to the dimensions coming from spans with SPAN_KIND_CLIENT, and the server_ prefix relates to the
dimensions coming from spans with SPAN_KIND_SERVER.

Since the service graph connector has to process both sides of an edge,
it needs to process all spans of a trace to function properly.
If spans of a trace are spread out over multiple instances, spans are not paired up reliably.
A possible solution to this problem is using the load balancing exporter
in a layer on front of collector instances running this connector.

Visualization Copy Chapter

Service graph metrics are natively supported by Grafana since v9.0.4.
To run it, configure a Tempo data source's 'Service Graphs' by linking to the Prometheus backend where metrics are being sent:

apiVersion: 1
datasources:
  # Prometheus backend where metrics are sent
  - name: Prometheus
    type: prometheus
    uid: prometheus
    url: <prometheus-url>
    jsonData:
        httpMethod: GET
    version: 1
  - name: Tempo
    type: tempo
    uid: tempo
    url: <tempo-url>
    jsonData:
      httpMethod: GET
      serviceMap:
        datasourceUid: 'prometheus'
    version: 1

Configuration Copy Chapter

The following settings are required:

• latency_histogram_buckets: the list of durations defining the latency histogram buckets. Make sure use either latency_histogram_buckets or exponential_histogram_max_size.
  • Default: [2ms, 4ms, 6ms, 8ms, 10ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1s, 1400ms, 2s, 5s, 10s, 15s]
• exponential_histogram_max_size: (no default) the maximum number of buckets per positive or negative number range.
• dimensions: the list of dimensions to add together with the default dimensions defined above.

The following settings can be optionally configured:

• store: defines the config for the in-memory store used to find requests between services by pairing spans.
  • ttl: TTL is the time to live for items in the store.
    • Default: 2s
  • max_items: MaxItems is the maximum number of items to keep in the store.
    • Default: 1000
• cache_loop: the interval at which to clean the cache.
  • Default: 1m
• store_expiration_loop: the time to expire old entries from the store periodically.
  • Default: 2s
• virtual_node_peer_attributes: the list of attributes, ordered by priority, whose presence in a client span will result in the creation of a virtual server node. An empty list disables virtual node creation.
  • Default: [peer.service.name, peer.service, db.name, db.namespace, db.system, db.system.name]
• virtual_node_extra_label: adds an extra label virtual_node with an optional value of client or server, indicating which node is the uninstrumented one.
  • Default: false
• metrics_flush_interval: the interval at which metrics are flushed to the exporter.
  • Default: 60s
• metrics_timestamp_offset: the offset to subtract from metric timestamps. If set to a positive duration, metric timestamps will be set to (current time - offset), effectively shifting metrics to appear as if they were generated in the past.
  • Default: 0
• database_name_attributes: the list of attribute names used to identify the database name from span attributes. The attributes are tried in order, selecting the first match.
  • Default: [db.name, db.namespace]

Example configurations Copy Chapter

Sample with custom buckets and dimensions Copy Section

receivers:
  otlp:
    protocols:
      grpc:

connectors:
  service_graph:
    latency_histogram_buckets: [100ms, 250ms, 1s, 5s, 10s]
    dimensions:
      - dimension-1
      - dimension-2
    store:
      ttl: 1s
      max_items: 10

exporters:
  prometheus/servicegraph:
    endpoint: localhost:9090
    namespace: servicegraph

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [service_graph]
    metrics/servicegraph:
      receivers: [service_graph]
      exporters: [prometheus/servicegraph]

Sample with options for uninstrumented services identification Copy Section

receivers:
  otlp:
    protocols:
      grpc:

connectors:
  service_graph:
    dimensions:
      - db.system
      - messaging.system
    virtual_node_peer_attributes:
      - db.name
      - db.system
      - messaging.system
      - peer.service.name
    virtual_node_extra_label: true

exporters:
  prometheus/servicegraph:
    endpoint: localhost:9090
    namespace: servicegraph

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [service_graph]
    metrics/servicegraph:
      receivers: [service_graph]
      exporters: [prometheus/servicegraph]

Warnings Copy Chapter

• Statefulness: The service_graph connector keeps every span that can be paired up to form a request in an in-memory store, periodically cleaned by cache_loop. All spans for a given request must be sent to the same collector instance, or the connector will be unable to pair them and the resulting service graph metrics will be incomplete.

Overview (`connector/signaltometricsconnector/README.md`) Copy Section

Signal to Metrics Connector

Signal to metrics connector produces metrics from all signal types (traces, logs, or metrics).

Supported Pipeline Types Copy Chapter

Configuration Copy Chapter

The component can produce metrics from spans, datapoints (for metrics), and logs.
At least one of the metrics for one signal type MUST be specified correctly for
the component to work.

All signal types can be configured to produce metrics with the same configuration
structure. For example, the below configuration will produce delta temporality counters
for counting number of events for each of the configured signals:

signal_to_metrics:
  spans:
    - name: span.count
      description: Count of spans
      sum:
        value: Int(AdjustedCount()) # Count of total spans represented by each span
        monotonic: true
  datapoints:
    - name: datapoint.count
      description: Count of datapoints
      sum:
        value: "1" # increment by 1 for each datapoint
        monotonic: true
  logs:
    - name: logrecord.count
      description: Count of log records
      sum:
        value: "1" # increment by 1 for each log record
        monotonic: true
  profiles:
    - name: profile.count
      description: Count of profiles
      sum:
        value: "1" # increment by 1 for each profile
        monotonic: true

Path context prefixes in OTTL expressions Copy Section

OTTL strings in conditions, value, count, and keys_expression may use
context path prefixes to disambiguate the target of a path.

Unprefixed paths continue to work. It is recommended to use the new syntax to avoid
future breaking changes.

Example (recommended):

signal_to_metrics:
  spans:
    - name: http.trace.span.duration
      attributes:
        - key: http.response.status_code
      conditions:
        - resource.attributes["service.name"] != nil
      sum:
        value: Int(Seconds(span.end_time - span.start_time))
        monotonic: true

Error Handling Copy Section

The error_mode configuration option determines how the connector handles errors that occur while processing OTTL expressions:

• error_mode (optional): Determines how errors returned from OTTL expressions are handled. Valid values are propagate, ignore, and silent.
  • ignore (default): Errors are logged and the specific record that caused the error is skipped, but processing continues for the rest of the batch.
  • propagate: Errors cause the entire batch to fail and be returned up the pipeline. This will result in the payload being dropped from the collector.
  • silent: Errors are not logged and the specific record that caused the error is skipped, but processing continues for the rest of the batch.

Example with error handling:

signaltometrics:
  error_mode: ignore  # Log errors but continue processing other records
  spans:
    - name: span.count
      description: Count of spans
      sum:
        value: Int(AdjustedCount())

Metrics types Copy Section

signal_to_metrics produces a variety of metric types by utilizing OTTL
to extract the relevant data for a metric type from the incoming data. The
component can produce the following metric types for each signal type:

• Sum
• Gauge
• Histogram
• Exponential Histogram

The component does NOT perform any stateful or time based aggregations. The metric
types are aggregated for the payload sent in each Consume* call. The final metric
is then sent forward in the pipeline.

Sum

Sum metrics have the following configurations:

sum:
  value: <ottl_value_expression>
  monotonic: <bool>

• [Required] value represents an OTTL expression to extract a value from the
  incoming data. Only OTTL expressions that return a value are accepted. The
  returned value determines the value type of the sum metric (int or double).
  OTTL converters
  can be used to transform the data.
• [Optional] monotonic whether the generated metric is monotonic. It defaults to false.

Gauge

Gauge metrics aggregate the last value of a signal and have the following configuration:

gauge:
  value: <ottl_value_expression>

• [Required] valuerepresents an OTTL expression to extract a numeric value from
  the signal. Only OTTL expressions that return a value are accepted. The returned
  value determines the value type of the gauge metric (int or double).
  • For logs: Use e.g. ExtractGrokPatterns with a single key selector (see below).
  • For other signals: Use a field such as value_int, value_double, or a valid OTTL expression.

Examples:

Logs (with Grok pattern):

signal_to_metrics:
  logs:
    - name: logs.memory_mb
      description: Extract memory_mb from log records
      gauge:
        value: ExtractGrokPatterns(body, "Memory usage %{NUMBER:memory_mb:int}MB")["memory_mb"]

Traces:

signal_to_metrics:
  spans:
    - name: span.duration.gauge
      description: Span duration as gauge
      gauge:
        value: Int(Seconds(end_time - start_time))

Histogram

Histogram metrics have the following configurations:

histogram:
  buckets: []float64
  count: <ottl_value_expression>
  value: <ottl_value_expression>

• [Optional] buckets represents the buckets to be used for the histogram.
  If no buckets are configured then it defaults to:

  go Copy

  []float64{2, 4, 6, 8, 10, 50, 100, 200, 400, 800, 1000, 1400, 2000, 5000, 10_000, 15_000}

• [Optional] count represents an OTTL expression to extract the count to be
  recorded in the histogram from the incoming data. If no expression is provided
  then it defaults to the count of the signal. OTTL converters
  can be used to transform the data. For spans, a special converter adjusted count,
  is provided to help calculate the span's adjusted count.

• [Required] value represents an OTTL expression to extract the value to be
  recorded in the histogram from the incoming data. OTTL converters
  can be used to transform the data.

Exponential Histogram

Exponential histogram metrics have the following configurations:

exponential_histogram:
  max_size: <int64>
  count: <ottl_value_expression>
  value: <ottl_value_expression>

• [Optional] max_size represents the maximum number of buckets per positive
  or negative number range. Defaults to 160.
• [Optional] count represents an OTTL expression to extract the count to be
  recorded in the exponential histogram from the incoming data. If no expression
  is provided then it defaults to the count of the signal. OTTL converters
  can be used to transform the data. For spans, a special converter adjusted count,
  is provided to help calculate the span's adjusted count.
• [Required] value represents an OTTL expression to extract the value to be
  recorded in the exponential histogram from the incoming data. OTTL converters
  can be used to transform the data.

Attributes Copy Section

The component can produce metrics categorized by the attributes (span attributes
for traces, datapoint attributes for datapoints, or log record attributes for logs)
from the incoming data by configuring attributes for the configured metrics.

If no attributes are configured then the metrics are produced without any attributes.

attributes:
  - key: datapoint.foo
  - key: datapoint.bar
    default_value: bar
  - key: datapoint.baz
    optional: true
  - keys_expression: otelcol.client.metadata["x-dynamic-attributes"]

Each attribute entry must have exactly one of key or keys_expression set.

If attributes are specified then a separate metric will be generated for each unique
set of attribute values. There are four behaviors that can be configured for an
attribute:

• Without any extra parameters: datapoint.foo in the above yaml is an example
  of such configuration. In this configuration, only the signals which have the said
  attribute are processed with the attribute's value as one of the attributes for
  the output metric. If the attribute is missing then the signal is not processed.
• With default_value: datapoint.bar in the above yaml is an example of such
  configuration. In this configuration all the signals are processed irrespective
  of the attribute being present or not in the input signal. The output metric
  is categorized as per the incoming value of the attribute and an extra bucket
  exists with the attribute set to the configured default value for all the signals
  that were missing the configured attribute.
• With optional set to true: datapoint.baz in the above yaml is an example
  of such configuration. If the attribute is configured with optional and present
  in the incoming signal then it will be added directly to the output metric. If
  it is absent then a new metric with missing attributes will be created. In addition,
  the optional attribute will not impact the decision i.e. even if the optional
  attributes are not present in the incoming signal, the signal will be processed
  and will produce a metric given all other non-optional attributes are present
  or have a default value defined.
• With keys_expression: The OTTL value expression is evaluated at runtime and
  must return a list of attribute keys (pcommon.Slice or []string). Each resolved
  key is looked up in the signal's attributes and included in the output metric.
  If the expression returns nil (e.g. missing client metadata), it is treated as
  an empty list. Expression evaluation errors are governed by the error_mode
  configuration. The optional and default_value options can be combined with
  keys_expression and apply to each resolved key.

Note that resource attributes are handled differently, check the resource attributes
section for more details on this. Think of attributes as conditional filters for
choosing which attributes should be included in the output metric whereas
include_resource_attributes is an include list for customizing resource attributes
of the output metric.

Conditions Copy Section

Conditions are an optional list of OTTL conditions that are evaluated on the incoming
data and are ORed together. For example:

signal_to_metrics:
  datapoints:
    - name: datapoint.bar.sum
      description: Count total number of datapoints as per datapoint.bar attribute
      conditions:
        - resource.attributes["foo"] != nil
        - resource.attributes["bar"] != nil
      sum:
        value: "1"

The above configuration will produce sum metrics from datapoints with either foo
OR bar resource attribute defined.

Conditions can also be ANDed together, for example:

signal_to_metrics:
  datapoints:
    - name: gauge.to.exphistogram
      conditions:
        - metric.type == 1 AND resource.attributes["resource.foo"] != nil
      exponential_histogram:
        count: "1" # 1 count for each datapoint
        value: Double(value_int) + value_double # handle both int and double

The above configuration produces exponential histogram from gauge metrics with resource
attributes resource.foo set.

Customizing resource attributes Copy Section

The component allows customizing the resource attributes for the produced metrics
by specifying a list of attributes that should be included in the final metrics.
If no attributes are specified for include_resource_attributes then no filtering
is performed i.e. all resource attributes of the incoming data is considered.

include_resource_attributes:
  - key: resource.foo # Include resource.foo attribute if present
  - key: resource.bar # Always include resource.bar attribute, default to bar
    default_value: bar
  - key: resource.baz # Optional resource.baz attribute is added if present
    optional: true
  - keys_expression: otelcol.client.metadata["x-dynamic-resource-attributes"]

Each entry must have exactly one of key or keys_expression set.

With the above configuration the produced metrics would have the following
resource attributes:

• resource.foo will be present for the produced metrics if the incoming data also
  has the attribute defined. If the attribute is missing in the incoming data the
  output metric will be produced without the said attribute.
• resource.bar will always be present because of the default_value. If the incoming
  data does not have a resource attribute with name resource.bar then the configured
  default_value of bar will be used.
• resource.baz will behave exactly same as resource.foo. Since resource attributes
  are basically an include list, the optional option is a no-op i.e. the resource
  attributes with optional set to true behaves identical to an attribute configured
  without default_value or optional.
• The keys_expression entry evaluates the OTTL value expression at runtime to
  resolve a list of attribute keys. The expression must return a list of strings
  (pcommon.Slice or []string). Each resolved key is looked up in the resource
  attributes and included in the output metric. If the expression returns nil
  (e.g. missing client metadata), it is treated as an empty list. Expression
  evaluation errors are governed by the error_mode configuration. The optional
  and default_value options can be combined with keys_expression and apply to
  each resolved key. OTTL expressions for include_resource_attributes should
  only reference resource-level paths (e.g. resource.attributes) or context-level
  paths (e.g. otelcol.client.metadata), not signal-specific paths (e.g.
  attributes, span.*, log.*).

Single writer Copy Section

Metrics data streams MUST obey single-writer
principle. However, since signal_to_metrics component produces metrics from all signal
types and also allows customizing the resource attributes, there is a possibility
of violating the single-writer principle. To keep the single-writer principle intact,
the component adds collector instance information as resource attributes. The following
resource attribute is added to each produced metric:

signal_to_metrics.service.instance.id: <service_instance_id_of_the_otel_collector>

Custom OTTL functions Copy Section

The component implements the following custom OTTL functions:

1. AdjustedCount: a converter capable of calculating adjusted count for a span.

Overview (`connector/slowsqlconnector/README.md`) Copy Section

Slow SQL Connector

Supported Pipeline Types Copy Chapter

Deprecation Notice: The component type has been renamed from slowsql to slow_sql
to follow the OpenTelemetry snake_case naming convention.
The old name slowsql still works but is deprecated and will be removed in a future release.
Please update your configuration to use slow_sql.

Overview Copy Chapter

Generate logs from recorded slow database statement associated with spans.

Each log will have at least the following dimensions:

• Service name
• Span kind
• Span name
• Status code
• Trace ID
• Span ID
• Database System
• Database Statement
• Database Statement Duration

Each log will additionally have the following attributes:

• Span attributes. If you want to filter out some attributes (like only copying HTTP attributes starting with http.) use the transform processor.

Configurations Copy Chapter

If you are not already familiar with connectors, you may find it helpful to first visit the Connectors README.

The following settings can be optionally configured:

• dimensions: the list of dimensions to add to logs with the default dimensions defined above.
  Each additional dimension is defined with a name which is looked up in the span's collection of attributes or
  resource attributes (AKA process tags) such as ip, host.name or region.
• db_system: the list value of span attribute db.system, Filter specific db systems, define those database's statements need to be collected. ref: https://opentelemetry.io/docs/specs/semconv/attributes-registry/db/
  • Default: [h2, mongodb, mssql, mysql, oracle, postgresql, mariadb]
• threshold: define a threshold and collect when the db.statement, namely span duration, larger than this value.
  • Default: 500ms

Examples Copy Chapter

The following is a simple example usage of the slow sql connector.

receivers:
  nop:

exporters:
  nop:

connectors:
  slow_sql:
    threshold: 600ms
    dimensions:
      - name: k8s.namespace.name
      - name: k8s.pod.name

service:
  pipelines:
    traces:
      receivers: [nop]
      exporters: [slow_sql]
    logs:
      receivers: [slow_sql]
      exporters: [nop]

The following is a more complex example usage of the slow_sql connector using Elasticsearch as exporters.

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  elasticsearch/slow_sql:
    tls:
      insecure: true
    mapping:
      mode: raw
    endpoints:
      - http://localhost:9200
    user: elastic
    password: elastic

connectors:
  slow_sql:
    threshold: 600ms
    dimensions:
      - name: k8s.namespace.name
      - name: k8s.pod.name

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [slow_sql]
    logs:
      receivers: [slow_sql]
      exporters: [elasticsearch/slow_sql]

The full list of settings exposed for this connector is documented in slowsqlconnector/config.go.

More Examples Copy Section

For more example configuration covering various other use cases, please visit the testdata directory.

Overview (`connector/spanmetricsconnector/README.md`) Copy Section

Span Metrics Connector

Supported Pipeline Types Copy Chapter

Deprecation Notice: The component type has been renamed from spanmetrics to span_metrics
to follow the OpenTelemetry snake_case naming convention.
The old name spanmetrics still works but is deprecated and will be removed in a future release.
Please update your configuration to use span_metrics.

⚠️ Breaking Change Warning:
The default duration metrics unit will change from ms to s to adhere to the OpenTelemetry semantic conventions and a feature gate connector.spanmetrics.useSecondAsDefaultMetricsUnit is also added.

Currently, the feature gate is disabled by default, so the unit will remain ms. After one release cycle, the unit will switch to s and the feature gate will also be enabled by default.

Overview Copy Chapter

Aggregates Request, Error and Duration (R.E.D) OpenTelemetry metrics from span data.

Request counts are computed as the number of spans seen per unique set of
dimensions, including Errors. Multiple metrics can be aggregated if, for instance,
a user wishes to view call counts just on service.name and span.name.

traces.span.metrics.calls{service.name="shipping",span.name="get_shipping/{shippingId}",span.kind="SERVER",status.code="Ok"}

Error counts are computed from the Request counts which have an Error Status Code metric dimension.

traces.span.metrics.calls{service.name="shipping",span.name="get_shipping/{shippingId},span.kind="SERVER",status.code="Error"}

Duration is computed from the difference between the span start and end times and inserted into the
relevant duration histogram time bucket for each unique set dimensions.

traces.span.metrics.duration{service.name="shipping",span.name="get_shipping/{shippingId}",span.kind="SERVER",status.code="Ok"}

Each metric will have at least the following dimensions because they are common
across all spans:

• service.name
• span.name
• span.kind
• status.code (or otel.status_code when the spanmetrics.statusCodeConvention.useOtelPrefix feature gate is enabled)
• collector.instance.id

The collector.instance.id dimension is intended to add a unique UUID to all metrics, ensuring that the spanmetrics connector
does not violate the Single Writer Principle when spanmetrics is used in a multi-deployment model.
To disable, use exclude_dimensions setting:

connectors:
  spanmetrics:
    exclude_dimensions: ['collector.instance.id']

Or, disable via the feature gate: --feature-gates=-connector.spanmetrics.includeCollectorInstanceID.
More detail, please see Known Limitation: the Single Writer Principle

Span to Metrics processor to Span to metrics connector Copy Chapter

The spanmetrics connector replaces spanmetrics processor with multiple improvements
and breaking changes. It was done to bring the spanmetrics connector closer to the OpenTelemetry
specification and make the component agnostic to exporters logic. The spanmetrics processor
essentially was mixing the OTel with Prometheus conventions by using the OTel data model and
the Prometheus metric and attributes naming convention.

The following changes were done to the connector component.

Breaking changes:

• The operation metric attribute was renamed to span.name.
• The latency histogram metric name was changed to duration.
• The _total metric prefix was dropped from generated metrics names.
• The Prometheus-specific metrics labels sanitization was dropped.

Improvements:

• Added support for OTel exponential histograms for recording span duration measurements.
• Added support for the milliseconds and seconds histogram units.
• Added support for generating metrics resource scope attributes. The spanmetrics connector will
  generate the number of metrics resource scopes that corresponds to the number of the spans resource
  scopes meaning that more metrics are generated now. Previously, spanmetrics generated a single
  metrics resource scope.

Configurations Copy Chapter

If you are not already familiar with connectors, you may find it helpful to first
visit the Connectors README.

The following settings can be optionally configured:

• histogram (default: explicit): Use to configure the type of histogram to record
  calculated from spans duration measurements. Must be either explicit or exponential.
  • disable (default: false): Disable all histogram metrics.
  • unit (default: ms): The time unit for recording duration measurements.
    calculated from spans duration measurements. One of either: ms or s.
  • dimensions: additional attributes to add as dimensions to the traces.span.metrics.duration metric,
    which will be included on top of the common and configured dimensions for span attributes and resource attributes.
  • explicit:
    • buckets: the list of durations defining the duration histogram time buckets. Default
      buckets: [2ms, 4ms, 6ms, 8ms, 10ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1s, 1400ms, 2s, 5s, 10s, 15s]
  • exponential:
    • max_size (default: 160) the maximum number of buckets per positive or negative number range.
• dimensions: the list of dimensions to add to traces.span.metrics.calls, traces.span.metrics.duration and traces.span.metrics.event metrics with the default dimensions defined above.
  Each list entry must set exactly one of:

  1. name which is looked up in the span's collection of attributes or resource attributes (AKA process tags) such as ip, host.name or region.

     If the named attribute is missing in the span, the optional provided default is used.

     If no default is provided, this dimension will be omitted from the metric.

  2. glob: a glob pattern with '.' treated as a separator. Every span or resource attribute whose key matches the pattern is emitted as its own dimension.

• calls_dimensions: additional attributes to add as dimensions to the traces.span.metrics.calls metric,
  which will be included on top of the common and configured dimensions for span attributes and resource attributes.
• exclude_dimensions: the list of dimensions to be excluded from the default set of dimensions. Use to exclude unneeded data from metrics.
• dimensions_cache_size: this setting is deprecated, please use aggregation_cardinality_limit instead.
• include_instrumentation_scope: a list of instrumentation scope names to include from the traces.
• resource_metrics_cache_size (default: 1000): the size of the cache holding metrics for a service. This is mostly relevant for
  cumulative temporality to avoid memory leaks and correct metric timestamp resets.
• aggregation_temporality (default: AGGREGATION_TEMPORALITY_CUMULATIVE): Defines the aggregation temporality of the generated metrics.
  One of either AGGREGATION_TEMPORALITY_CUMULATIVE or AGGREGATION_TEMPORALITY_DELTA.
• namespace (default: traces.span.metrics): Defines the namespace of the generated metrics. If namespace provided, generated metric name will be added namespace. prefix.
• metrics_flush_interval (default: 60s): Defines the flush interval of the generated metrics.
• metrics_expiration (default: 0): Defines the expiration time as time.Duration, after which, if no new spans are received, metrics will no longer be exported. Setting to 0 means the metrics will never expire.
• series_expiration (default: 0): Defines the expiration time as time.Duration for individual metric series. When set, stale dimension combinations are removed on a later flush even if the parent metric and resource continue receiving other spans. Setting to 0 disables per-series expiration.
• metric_timestamp_cache_size (default 1000): Only relevant for delta temporality span metrics. Controls the size of the cache used to keep track of a metric's TimestampUnixNano the last time it was flushed. When a metric is evicted from the cache, its next data point will indicate a "reset" in the series. Downstream components converting from delta to cumulative, like prometheusexporter, may handle these resets by setting cumulative counters back to 0.
• exemplars: Use to configure how to attach exemplars to metrics.
  • enabled (default: false): enabling will add spans as Exemplars to all metrics. Exemplars are only kept for one flush interval.rom the cache, its next data point will indicate a "reset" in the series. Downstream components converting from delta to cumulative, like prometheusexporter, may handle these resets by setting cumulative counters back to 0.
  • max_per_data_point (default: 5): The maximum number of exemplars to attach to a single metric data point.
• events: Use to configure the events metric.
  • enabled: (default: false): enabling will add the events metric.
  • dimensions: (mandatory if enabled) the list of the span's event attributes to add as dimensions to the traces.span.metrics.events metric, which will be included on top of the common and configured dimensions for span attributes and resource attributes.
• resource_metrics_key_attributes: Filter the resource attributes used to produce the resource metrics key map hash(It's only used to build the hash key, not copy the attributes to metrics resource attributes).
  Use this in case changing resource attributes (e.g. process id) are breaking counter metrics.
• aggregation_cardinality_limit (default: 0): Defines the maximum number of unique combinations of dimensions that will be tracked for metrics aggregation. When the limit is reached, additional unique combinations will be dropped but registered under a new entry with otel.metric.overflow="true". A value of 0 means no limit is applied.
• add_resource_attributes (default: false): Add the resource attributes to the resulting metrics. This option enables the old behavior before the connector.spanmetrics.excludeResourceMetrics feature gate was introduced. When set to true, resource attributes will be included in the metrics even if the feature gate is enabled. See GitHub issue #42103 for more context.
• enable_metrics_sampling_method (default: false): When enabled, adds the sampling.method attribute to metrics with value "extrapolated" (when the span has a valid tracestate sampling threshold) or "counted" (otherwise).

The feature gate connector.spanmetrics.legacyMetricNames (disabled by default) controls the connector to use legacy metric names.

Examples Copy Chapter

The following is a simple example usage of the spanmetrics connector.

For configuration examples on other use cases, please refer to More Examples.

The full list of settings exposed for this connector are documented in spanmetricsconnector/config.go.

receivers:
  nop:

exporters:
  nop:

connectors:
  span_metrics:
    histogram:
      dimensions:
        - name: url.scheme
          default: https
      explicit:
        buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]  
    dimensions:
      - name: http.method
        default: GET
      - name: http.status_code
      - glob: "k8s.*.name" # match single namespace between 'k8s' and 'name' ('k8s.cluster.name', not 'k8s.cluster.label.name')
      - glob: "db.**.name" # match any number of namespaces between 'db' and 'name'
    calls_dimensions:
      - name: http.url
        default: /ping
    exemplars:
      enabled: true
    exclude_dimensions: ['status.code']
    aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"
    metrics_flush_interval: 15s
    metrics_expiration: 5m
    series_expiration: 5m
    events:
      enabled: true
      dimensions:
        - name: exception.type
        - name: exception.message
    resource_metrics_key_attributes:
      - service.name
      - telemetry.sdk.language
      - telemetry.sdk.name
    include_instrumentation_scope:
      - express

service:
  pipelines:
    traces:
      receivers: [nop]
      exporters: [span_metrics]
    metrics:
      receivers: [span_metrics]
      exporters: [nop]

Using `spanmetrics` with Prometheus components Copy Section

The spanmetrics connector can be used with Prometheus exporter components.

For some functionality of the exporters, e.g. like generation of the target_info metric the
incoming spans resource scope attributes must contain service.name and service.instance.id
attributes.

Let's look at the example of using the spanmetrics connector with the prometheus_remote_write exporter:

receivers:
  otlp:
    protocols:
      http:
      grpc:

exporters:
  prometheus_remote_write:
    endpoint: http://localhost:9090/api/v1/write
    target_info:
      enabled: true

connectors:
  span_metrics:
    namespace: span.metrics

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [span_metrics]
    metrics:
      receivers: [span_metrics]
      exporters: [prometheus_remote_write]

This configures the spanmetrics connector to generate metrics from received spans and export the
metrics to the Prometheus Remote Write exporter. The target_info metric will be generated for each
resource scope, while OpenTelemetry metric names and attributes will be normalized
to be compliant with Prometheus naming rules. For example, the generated calls OTel sum metric can
result in multiple Prometheus calls_total (counter type) time series and the target_info time series.
For example:

target_info{job="shippingservice", instance="...", ...} 1
calls_total{span_name="/Address", service_name="shippingservice", span_kind="SPAN_KIND_SERVER", status_code="STATUS_CODE_UNSET", ...} 142

More Examples Copy Section

For more example configuration covering various other use cases, please visit the testdata directory.

Known Limitation: the Single Writer Principle Copy Chapter

Proper configuration of the spanmetricsconnector ensures compliance with the Single Writer Principle,
which is a core requirement in the OpenTelemetry metrics data model. Misconfiguration, however,
may allow multiple components to write to the same metric stream, resulting in data inconsistency,
metric conflicts, or the dropping of time series by metric backends.

Why this happens Copy Section

This issue typically arises when:

• Multiple pipelines use the same instance of the spanmetricsconnector
• The connector is instantiated more than once without ensuring the resulting metric streams are distinct
• The resource_metrics_key_attributes field is not configured correctly or includes common/shared attributes across all instances

Recommendations Copy Section

To reduce the risk of conflicting writes:

• Add resource_metrics_key_attributes to your configuration.

connectors:
  span_metrics:
    resource_metrics_key_attributes:
      - service.name
      - telemetry.sdk.language
      - telemetry.sdk.name

• The feature gate connector.spanmetrics.includeCollectorInstanceID is enabled by default to produce uniquely identified metrics.
• For exporters like Prometheus, which rely on the single writer assumption, use a dedicated pipeline with a single spanmetricsconnector instance

More context is available in GitHub issue #21101.

About `resource_metrics_key_attributes` Copy Section

The resource_metrics_key_attributes setting are used to build the key map that determines how metrics are grouped.

If this field is left empty, the connector will use all available attributes to compute the resource metric hash.

To avoid problems, be cautious when choosing which attributes to include.

Avoid attributes that:

• Change frequently – such as request_id, timestamp, or trace_id. These increase cardinality and create excessive metric streams.
• Are shared across all sources – values like true, default, or team:backend offer no uniqueness and can lead to multiple writers sharing the same stream.
• Are optional or inconsistently applied – if an attribute is only present in some spans, this can fragment metric streams (e.g., one stream with the attribute and one without).

Instead, use attributes that are stable, present in all spans, and meaningfully distinguish each stream. Good examples include cluster_id, region, or deployment_environment.

Troubleshooting span metrics high cardinality Copy Chapter

High cardinality issues in span metrics commonly manifest in APM dashboards as an excessive number of service operations with non-unique names. Examples include URIs with unique identifiers (e.g., GET /product/1YMWWN1N4O) or HTTP parameters with random values (e.g., GET /?_ga=GA1.2.569539246.1760114706). These patterns render operation lists difficult to interpret and ineffective for monitoring purposes.

This issue stems from violations of OpenTelemetry semantic conventions, which require span names to have low cardinality (e.g. HTTP span name specs).
Beyond degrading APM interfaces with numerous non-meaningful operation names, this problem leads to metric time series explosion, resulting in significant performance degradation and increased costs.

The span metrics connector provides an optional circuit breaker through the aggregation_cardinality_limit attribute (disabled by default) to mitigate cardinality explosion. While this feature addresses performance and cost concerns, it does not resolve the underlying issue of semantically meaningless operation names.

Fixing high cardinality span name issues Copy Section

The ideal long-term solution is to modify the OpenTelemetry instrumentation code to comply with semantic conventions, preventing the generation of non-compliant high cardinality span names.

However, deploying updated instrumentation libraries can be time-consuming, often requiring an immediate interim solution to restore observability backend functionality.

Addressing high cardinality span names in the ingestion pipeline

An effective short-term solution is to implement a span sanitization layer within the observability ingestion pipeline. This can be achieved by using the OpenTelemetry Collector Transform Processor's set_semconv_span_name() function immediately before the Span Metrics Connector to enforce semantic conventions on span names.

Simple example OpenTelemetry Collector configuration to prevent span metrics cardinality explosion:

receivers:
  otlp:
  ...

processors:
  transform/sanitize_spans:
    error_mode: ignore
    trace_statements:
      # Sanitize all span names to prevent span metrics cardinality explosion
      # caused by non-compliant high cardinality span names
      - set_semconv_span_name("1.37.0")
  ...

connectors:
  span_metrics:

exporters:
  otlp_http/observability-backend:
  ...

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [transform/sanitize_spans, ...]
      exporters: [otlp_http/observability-backend, span_metrics]
    metrics:
      receivers: [otlp, span_metrics]
      processors: [...]
      exporters: [otlp_http/observability-backend]
    # ...

Aggressive span name sanitization may be overly restrictive for instrumentations with incomplete resource attributes. For instance, HTTP service operations may be reduced to generic names like GET and POST when HTTP spans lack the http.route attribute. This information loss can impact the monitoring of critical business operations.

To preserve operation granularity, you can manually set the http.route attribute when detailed operation names are required. The missing http.route value can typically be derived through pattern matching on other span attributes such as http.target or url.full.

Example OpenTelemetry Collector configuration that prevents cardinality explosion
while preserving meaningful operation names on a service webshop/frontend:

/* Detailed source-code truncated for AI context efficiency. */

Addressing high cardinality span names in the instrumentation code

The preferred long-term solution is to ensure span names and attributes comply with OpenTelemetry Semantic Conventions directly in the instrumentation code.

Custom web frameworks are a common source of high cardinality span names. While default OpenTelemetry instrumentation (e.g., Java Servlet) may assign generic span names like GET /my-web-fwk/*, your framework has access to more specific routing information. By overwriting span attributes in your framework code, you can create compliant, low-cardinality span names that preserve operational granularity.

Example: Custom Web Framework in Java

Consider a custom web framework that intercepts the generic route /my-web-fwk/* and dispatches requests like /my-web-fwk/product/123456ABCD or /my-web-fwk/user/john.doe.

The default Java Servlet instrumentation produces vague span names (GET /my-web-fwk/*), while directly using request URIs creates high cardinality (GET /my-web-fwk/product/123456ABCD).

The solution is to override span attributes with templated route patterns like /my-web-fwk/product/{productId} or /my-web-fwk/user/{userId}:

@WebServlet(urlPatterns = "/my-web-fwk/*")
public class MyWebFrameworkServlet extends HttpServlet {

    @Override
    protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        // Default Servlet instrumentation sets vague span names: `GET /my-web-fwk/*` (and `http.route=/my-web-fwk/*`)
        // Using the request URI directly would cause high cardinality and violate semantic conventions
        // Instead, use the framework's low-cardinality routing information below

        // Example routing logic
        String uri = request.getRequestURI();
        MyWebOperation myWebOperation = getWebOperation(uri);

        // Fix span details to add details while complying with semantic conventions and maintaining low cardinality
        String httpRoute = "/my-web-fwk/" + myWebOperation.getSubHttpRoute();
        Span.current().setAttribute(HttpAttributes.HTTP_ROUTE, httpRoute);
        Span.current().updateName(request.getMethod() + " " + httpRoute);

        // execute the web operation
        myWebOperation.execute(request, response);
    }
    ...
}

Overview (`connector/sumconnector/README.md`) Copy Section

Sum Connector

Supported Pipeline Types Copy Chapter

The sum connector can be used to sum attribute values from spans, span events, metrics, data points, and log records.

Configuration Copy Chapter

If you are not already familiar with connectors, you may find it helpful to first visit the Connectors README.

Basic configuration Copy Section

This example configuration will sum numerical values found within the attribute attribute.with.numerical.value of any span telemetry routed to the connector. It will then output a metric time series with the name my.example.metric.name with those summed values.

Note: Values found within an attribute will be converted into a float regardless of their original type before being summed and output as a metric value. Non-convertible strings will be dropped and not included.

receivers:
  foo:
connectors:
  sum:
    spans:
      my.example.metric.name:
        source_attribute: attribute.with.numerical.value
exporters:
  bar:

service:
  pipelines:
    metrics/sum:
       receivers: [sum]
       exporters: [bar]
    traces:
       receivers: [foo]
       exporters: [sum]

Required Settings

The sum connector has three required configuration settings and numerous optional settings

• Telemetry type: Nested below the sum: connector declaration. Declared as spans: in the Basic Example.
  • Can be any of spans, spanevents, datapoints, or logs.
  • For metrics use datapoints
  • For traces use spans or spanevents
• Metric name: Nested below the telemetry type; this is the metric name the sum connector will output summed values to. Declared as my.example.metric.name in the Basic Example
• source_attribute: A specific attribute to search for within the source telemetry being fed to the connector. This attribute is where the connector will look for numerical values to sum into the output metric value. Declared as attribute.with.numerical.value in the Basic Example

Optional Settings

• conditions: OTTL syntax can be used to provide conditions for processing incoming telemetry. Conditions are ORed together, so if any condition is met the attribute's value will be included in the resulting sum. Conditions support OTTL path context names (e.g. span.attributes["x"], resource.attributes["x"], log.attributes["x"], metric.name). Paths without an explicit context are interpreted in the context of the enclosing block. It is recommended to switch to the new syntax to avoid breaking changes in the future.
• attributes: Declaration of attributes to include. Any of these attributes found will generate a separate sum for each set of unique combination of attribute values and output as its own datapoint in the metric time series.
  • key: (required for attributes) the attribute name to match against
  • default_value: (optional for attributes) a default value for the attribute when no matches are found. The default_value value can be of type string, integer, or float.

Detailed Example Configuration Copy Section

This example declares that the sum connector is going to be ingesting logs and creating an output metric named checkout.total with numerical values found in the source_attribute total.payment.

It provides a condition to check that the attribute total.payment is not NULL. It also checks any incoming log telemetry for values present in the attribute payment.processor and creates a datapoint within the metric time series for each unique value. Any logs without values in payment.processor will be included in a datapoint with the default_value of unspecified_processor.

receivers:
  foo:
connectors:
  sum:
    logs:
      checkout.total:
        source_attribute: total.payment
        conditions:
          - attributes["total.payment"] != "NULL"
        attributes:
          - key: payment.processor
            default_value: unspecified_processor
exporters:
  bar:

service:
  pipelines:
    metrics/sum:
       receivers: [sum]
       exporters: [bar]
    logs:
       receivers: [foo]
       exporters: [sum]

Note for Log to Metrics: If your logs contain all values in their body rather than in attributes (E.G. JSON payload) use a transform processor in your pipeline to upsert parsed key/value pairs (in this case from JSON) into attributes attached to the log.

processors:
  transform/logs:
    log_statements:
      - context: log
        statements:
          - merge_maps(attributes, ParseJSON(body), "upsert")

Overview (`examples/couchbase/README.md`) Copy Section

Couchbase Example

This example shows how you can configure the collector to scrape a Couchbase server using its built-in Prometheus metrics exporter.

This example config retrieves a subset of metrics important for monitoring a Couchbase server, transforms them to follow OpenTelemetry conventions, then exports again to prometheus so that they may be easily viewed.

Running Copy Chapter

First build the collector docker image by running the following command in the root of this repository:

make docker-otelcontribcol

Then, start Couchbase, the collector, and a Prometheus server by running the following command in this directory:

docker-compose up -d --remove-orphans --build --force-recreate

After Couchbase is running, you'll need to initialize Couchbase and create some buckets:

./scripts/setup.sh

You should now be able to view metrics being scraped from the Couchbase instance every 5 seconds in Prometheus at http://localhost:9090 (all metrics are prefixed with couchbase_)

Once you are done with the example, you may shut it down:

docker-compose down

Overview (`examples/fault-tolerant-logs-collection/README.md`) Copy Section

Fault tolerant log collection example Copy Chapter

File Log receiver's persistence can be covered by the usage of the following extensions:

• filestorage extension,
  to ensure that Collector's restarts do not affect the log collection and offset tracking.
• exporterhelper persistent-queue,
  to ensure that Collector's restarts do not affect the delivery of the already collected logs.

A full configuration example is provided in example config

Overview (`examples/kubernetes/README.md`) Copy Section

OpenTelemetry Collector Demo

This demo is a sample app to build the collector and exercise its kubernetes logs scraping functionality.

Build and Run Copy Chapter

Kubernetes Copy Section

Switch to this directory and run following command: kubectl apply -n <namespace> -f otel-collector.yaml

Include/Exclude Specific Logs

Kubernetes logs are being stored in /var/log/pods.
Path to container logs is constructed using following pattern:

/var/log/pods/<namespace>__/<container>/<restart_count>.log

You can use it to manage from which containers do you want to include and exclude logs.

For example, to include all logs from default namespace, you can use following configuration:

include:
    - /var/log/pods/default_*/*/*.log
exclude: []

Docker Compose Copy Section

Two steps are required to build and run the demo:

1. Build latest docker image in main repository directory make docker-otelcontribcol
2. Switch to this directory and run docker-compose up

Description

varlogpods contains example log files placed in kubernetes-like directory structure.
Each of the directory has different formatted logs in one of three formats (either CRI-O, CRI-Containerd or Docker).
This directory is mounted to standard location (/var/log/pods).

otel-collector-config is a configuration to autodetect and parse logs for all of three mentioned formats

Performance Tests Copy Chapter

There are multiple tests for various configurations in testbed.
Following table shows result of example run:

ToDo Copy Chapter

To cover kubernetes system logs, logs from journald should be supported as well.

Overview (`examples/logline-filtering/README.md`) Copy Section

Filtering log messages based on content Copy Chapter

File Log receiver provides support for filtering logs based on their content. This can be achieved by using
the filter operator,
configured with matching regular expressions.

With this happening at the collection point, a lot of resources at the destination backend
can be saved since no additional processing would need to take place.

A full configuration example on how to filter out logs that start with the INFO: pattern is
provided in the example config.
A full configuration example on how to only collect logs that start with the WARN: pattern is provided in
the example config

Overview (`examples/nomad/README.md`) Copy Section

OpenTelemetry Collector Demo

This demo provides you with all you need to run the OpenTelemetry Collector on a local installation of HashiCorp Nomad using Traefik as your load balancer. It configures both both HTTP and gRPC endpoints on Traefik.

For a detailed, step-by-step tutorial on how to run the files included in this directory, check out Just-in-Time Nomad: Running the OpenTelemetry Collector on HashiCorp Nomad with HashiQube.

The tutorial includes instructions on how to set up a local Nomad environment using HashiQube (a fork of servian/hashiqube). It spins up a Vagrant VM running Nomad, Consul, Vault and Traefik. Detailed instructions can be found in the project readme.

Additional Resources Copy Chapter

Check out the OpenTelemetry Collector Nomad Pack in the Nomad Pack Community Registry.

Overview (`examples/secure-tracing/README.md`) Copy Section

Build A Secure Trace Collection Pipeline

Implementing robust security measures is essential for any tracing ingestion service and infrastructure. This is an illustrative example of a secure setup encompassing the following features for trace ingestion:

• Data Encryption with OTLP receiver supporting TLS. Transport Layer Security (TLS) is employed to encrypt traces in transit between the application and the OpenTelemetry (OTLP) endpoint, fortifying data security.
• Client Authentication via Envoy, a high-performance proxy.
  Even though we can configure OTLP receiver with mTLS for client authentication, authorization is not supported by OpenTelemetry Collector. This is one of the reasons that we use Envoy for client authentication. It allows us to easily add authorization, ensuring that only authorized clients can submit traces to the ingestion service.

In this example, we also include a test via telementrygen: a tool provided from this repository for generating synthetic telemetry data, which helps verify the security features and overall functionality of the set up.

Data Encryption via TLS Copy Chapter

The OpenTelemetry Collector has detailed documentation on how to configure TLS. In this example, we enable TLS for receivers which leverages server configuration.

Example

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: mysite.local:55690
        tls:
          cert_file: server.crt
          key_file: server.key

Client Authentication Copy Chapter

Envoy sandbox for TLS is a good source for reference. A few elements in following configuration are good to call out,

• require_client_certificate is set true
• The matcher under validation_context expects a client certificate with group-id as well as tenat-id which provides more granular control.

typed_config:
    "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
    require_client_certificate: true
    common_tls_context:
    # h2 If the listener is going to support both HTTP/2 and HTTP/1.1.
    alpn_protocols: "h2"
    tls_certificates:
    - certificate_chain: 
        filename: "/etc/envoy.crt"
        private_key: 
        filename: "/etc/envoy.key"
    validation_context:
        match_typed_subject_alt_names:
        - san_type: URI
        matcher:
            Match tenant by two level info: group id and tenant-id.
            exact: "aprn:trace-client:certmgr:::group-x:/ig/5003178/uv/tenant-a"
        trusted_ca:
        filename: "/etc/ca.crt"

Setup Environment Copy Chapter

Generate Certificates Copy Section

To generate various self-signed certificates, including those for Envoy and the OpenTelemetry Collector receiver, as well as tracing client certificate, we utilize the widely renowned open-source tool OpenSSL, OpenSSL 3.1.0 14 was tested.

In the certs folder, you can find a set of .ext files which define the properties for a certificate. A MakeFile is provided to facilate the process.

$ cd certs
$ make clean && make all

Bring up services Copy Section

We use docker compose to bring up Envoy and OpenTelemetry Collection Pipeline. Make sure the current folder is secure-tracing,

$ docker compose up

From the console window, verify that Envoy and Collector are up and running. If you see error similar to following,

secure-tracing-otel-collector-1  | Error: cannot start pipelines: failed to load TLS config: failed to load TLS cert and key: read /etc/otel-collector.crt: is a directory

It's most likely due to missing certificates. Follow the steps from section above to generate certificates.

Run test Copy Chapter

Compile telemetrygen Copy Section

From the root of this repository,

$ cd cmd/telemetrygen
$ go build . 
$ cp telemetrygen ../../examples/secure-tracing

Send a trace Copy Section

From the root of this repository,

$ cd examples/secure-tracing
$ chmod +x telemetrygen
$ ./telemetrygen traces --traces 1 --otlp-endpoint 127.0.0.1:10000 --ca-cert 'certs/ca.crt' --mtls --client-cert 'certs/tracing-client.crt' --client-key 'certs/tracing-client.key'

Verify traces are captured and sent to console running docker compose up, similar to following ...

secure-tracing-otel-collector-1  |      -> service.name: Str(telemetrygen)
secure-tracing-otel-collector-1  | ScopeSpans #0
secure-tracing-otel-collector-1  | ScopeSpans SchemaURL: 
secure-tracing-otel-collector-1  | InstrumentationScope telemetrygen 
secure-tracing-otel-collector-1  | Span #0
secure-tracing-otel-collector-1  |     Trace ID       : 0fe7ca900fda938ce918f8b2d82d6ae9
secure-tracing-otel-collector-1  |     Parent ID      : 1011b3f973049923
secure-tracing-otel-collector-1  |     ID             : 65f3cfe524375dd4
secure-tracing-otel-collector-1  |     Name           : okey-dokey
secure-tracing-otel-collector-1  |     Kind           : Server
...
secure-tracing-otel-collector-1  | Attributes:
secure-tracing-otel-collector-1  |      -> net.sock.peer.addr: Str(1.2.3.4)
secure-tracing-otel-collector-1  |      -> peer.service: Str(telemetrygen-client)
...

Shutdown Services Copy Chapter

$ docker compose down

Overview (`exporter/alertmanagerexporter/README.md`) Copy Section

Alertmanager Exporter

Exports OTEL Events (SpanEvent in Tracing added by AddEvent API and Log Records) as Alerts to Alertmanager back-end to notify Errors or Change events.

Supported pipeline types: traces, logs

Getting Started Copy Chapter

The following settings are required:

• endpoint: Alertmanager endpoint to send events
• severity: Default severity for Alerts

The following settings are optional:

• timeout sending_queue and retry_on_failure settings as provided by Exporter Helper.
• HTTP settings
• TLS and mTLS settings
• generator_url is the source of the alerts to be used in Alertmanager's payload. The default value is "opentelemetry-collector", and can be set to the URL of the opentelemetry collector.
• severity_attribute is the SpanEvent or LogRecord Attribute name which can be used instead of default severity string in Alert payload.
  e.g.: If severity_attribute is set to "foo" and the SpanEvent or LogRecord has an attribute called foo, foo's attribute value will be used as the severity value for that particular Alert generated from the SpanEvent or LogRecord.
• api_version is the API version of Alertmanager. By default the value is set to "v2" and can be overridden to "v1" if using an older version of Alertmanager.
• event_labels is the list of Event or LogRecord Attributes that will be captured as Labels in the Alert payload if value exists.

Example config:

exporters:
  alertmanager:
  alertmanager/2:
    endpoint: "https://a.new.alertmanager.target:9093"
    severity: "debug"
    severity_attribute: "foo"
    api_version: "v2"
    event_labels: ["foo", "bar"]
    tls:
      cert_file: /var/lib/mycert.pem
      key_file: /var/lib/key.pem
    timeout: 10s
    sending_queue:
      enabled: true
      num_consumers: 2
      queue_size: 10
    retry_on_failure:
      enabled: true
      initial_interval: 10s
      max_interval: 60s
      max_elapsed_time: 10m
    generator_url: "opentelemetry-collector"

Overview (`exporter/alibabacloudlogserviceexporter/README.md`) Copy Section

AlibabaCloud LogService Exporter

This exporter supports sending OpenTelemetry data to LogService.

Configuration options:

• endpoint (required): LogService's Endpoint.
• project (required): LogService's Project Name.
• logstore (required): LogService's store Name. For metrics data, you should use metric store.
• access_key_id (optional): AlibabaCloud access key id.
• access_key_secret (optional): AlibabaCloud access key secret.
• security_token (optional): AlibabaCloud security token for STS credentials.
• ecs_ram_role (optional): set AlibabaCLoud ECS ram role if you are using ACK.
• token_file_path (optional): Set token file path if you are using ACK.

Example:

Simple Trace Data Copy Chapter

receivers:
  examplereceiver:

exporters:
  alibabacloud_logservice:
    endpoint: "cn-hangzhou.log.aliyuncs.com"
    project: "demo-project"
    logstore: "traces-store"
    access_key_id: "access-key-id"
    access_key_secret: "access-key-secret"

service:
  pipelines:
    traces:
      receivers: [examplereceiver]
      exporters: [alibabacloud_logservice]

All Telemetry Data Copy Chapter

If you are using OpenTelemetry Collector to collect different types of telemetry data, you should send to different LogService's store.

receivers:
  examplereceiver:

exporters:
  alibabacloud_logservice/logs:
    endpoint: "cn-hangzhou.log.aliyuncs.com"
    project: "demo-project"
    logstore: "logs-store"
    access_key_id: "access-key-id"
    access_key_secret: "access-key-secret"
  alibabacloud_logservice/metrics:
    endpoint: "cn-hangzhou.log.aliyuncs.com"
    project: "demo-project"
    logstore: "metrics-store"
    access_key_id: "access-key-id"
    access_key_secret: "access-key-secret"
  alibabacloud_logservice/traces:
    endpoint: "cn-hangzhou.log.aliyuncs.com"
    project: "demo-project"
    logstore: "traces-store"
    access_key_id: "access-key-id"
    access_key_secret: "access-key-secret"

service:
  pipelines:
    traces:
      receivers: [examplereceiver]
      exporters: [alibabacloud_logservice/traces]
    logs:
      receivers: [examplereceiver]
      exporters: [alibabacloud_logservice/logs]
    metrics:
      receivers: [examplereceiver]
      exporters: [alibabacloud_logservice/metrics]

Overview (`exporter/awscloudwatchlogsexporter/README.md`) Copy Section

AWS CloudWatch Logs Exporter

AWS CloudWatch Logs Exporter sends logs data to AWS CloudWatch Logs.
AWS credentials are retrieved from the default credential chain.

NOTE: OpenTelemetry Logging support is experimental, hence this exporter is subject to change.

Configuration Copy Chapter

The following settings are required:

• log_group_name: The group name of the CloudWatch Logs. If it does not exist it will be created automatically. It supports several placeholder names. One valid example is /aws/metrics/{ClusterName}. It will search for ClusterName (or aws.ecs.cluster.name) resource attribute in the metrics data and replace with the actual cluster name. If none of them are found in the resource attribute map, {ClusterName} will be replaced by undefined. Similar way, for the {TaskId}, it searches for TaskId (or aws.ecs.task.id) key in the resource attribute map. For {NodeName}, it searches for NodeName (or k8s.node.name)
  • List of valid placeholders:
    • {ClusterName}: aws.ecs.cluster.name
    • {TaskId}: aws.ecs.task.id
    • {NodeName}: k8s.node.name
    • {PodName}: pod or k8s.pod.name
    • {ServiceName}: service.name
    • {ContainerInstanceId}: aws.ecs.container.instance.id
    • {TaskDefinitionFamily}: aws.ecs.task.family
    • {InstanceId}: service.instance.id
    • {FaasName}: faas.name
    • {FaasVersion}: faas.version
• log_stream_name: The stream name of the CloudWatch Logs. If it does not exist it will be created automatically. It supports the same placeholders as log_group_name

The following settings can be optionally configured:

• region: The AWS region where the log stream is in. Region must be specified if it is not already set in the default credential chain.
• endpoint: The CloudWatch Logs service endpoint which the requests are forwarded to. See the CloudWatch Logs endpoints for a list.
• log_retention: LogRetention is the option to set the log retention policy for only newly created CloudWatch Log Groups. Defaults to Never Expire if not specified or set to 0. Possible values for retention in days are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, or 3653.
• tags: Tags is the option to set tags for the CloudWatch Log Group. If specified, please add at most 50 tags. Input is a string to string map like so: { 'key': 'value' }. Keys must be between 1-128 characters and follow the regex pattern: ^([\p{L}\p{Z}\p{N}_.:/=+\-@]+)$(alphanumerics, whitespace, and _.:/=+-!). Values must be between 1-256 characters and follow the regex pattern: ^([\p{L}\p{Z}\p{N}_.:/=+\-@]\*)$(alphanumerics, whitespace, and _.:/=+-!). Link to tagging restrictions
• raw_log: Boolean default false. If set to true, only the log message will be exported to CloudWatch Logs. This needs to be set to true for EMF logs.
• max_event_payload_bytes: Per-event payload cap, in bytes, used to truncate oversized log events before they are sent to CloudWatch Logs. As of 2025-04-02 the CloudWatch Logs PutLogEvents API accepts events up to 1 MiB (1048576 bytes); the previous limit was 256 KiB. To preserve backwards compatibility this exporter still defaults to 256 KiB. Set this field to 1048576 (or any value up to that ceiling) to take advantage of the larger service limit. Allowed range: 41 to 1048576. A value of 0 (or omitted) means "use the package default (256 KiB)".
• role_arn: IAM role to upload logs to a different account.
• external_id: Shared identitier used when assuming an IAM role in an external AWS account. See AWS IAM Guide
• sending_queue: Parameters for the sending queue, where you can control parallelism and the size of the sending buffer. Obs.: this component will always have a sending queue enabled.
  • num_consumers: Number of consumers that will consume from the sending queue. This parameter controls how many consumers will consume from the sending queue in parallel.
  • queue_size: Maximum number of batches kept in memory before dropping; ignored if enabled is false

Examples Copy Section

Simplest configuration:

exporters:
  awscloudwatchlogs:
    log_group_name: "testing-logs"
    log_stream_name: "testing-integrations-stream"

Example configuration for EMF logs:

exporters:
  awscloudwatchlogs:
    log_group_name: "testing-logs-emf"
    log_stream_name: "testing-integrations-stream-emf"
    raw_log: true
    region: "us-east-1"
    endpoint: "logs.us-east-1.amazonaws.com"
    log_retention: 365
    tags: { "sampleKey": "sampleValue" }

Additional Notes Copy Chapter

• If the log group and/or log stream are specified in an EMF log, that EMF log will be exported to that log group and/or log stream (i.e. ignores the log group and log stream defined in the configuration)
  • The log group and log stream will also be created automatically if they do not already exist.
  • Example of an EMF log with log group and log stream:

{
  "_aws": {
    "Timestamp": 1574109732004,
    "LogGroupName": "Foo",
    "LogStreamName": "Bar",
    "CloudWatchMetrics": [
      {
        "Namespace": "MyApp",
        "Dimensions": [["Operation"]],
        "Metrics": [
          {
            "Name": "ProcessingLatency",
            "Unit": "Milliseconds",
            "StorageResolution": 60
          }
        ]
      }
    ]
  },
  "Operation": "Aggregator",
  "ProcessingLatency": 100
}

• Resource ARNs (Amazon Resource Name (ARN) of the AWS resource running the collector) are currently not supported with the CloudWatch Logs Exporter.

Overview (`exporter/awsemfexporter/README.md`) Copy Section

AWS CloudWatch EMF Exporter

This exporter converts OpenTelemetry metrics to
AWS CloudWatch Embedded Metric Format(EMF)
and then sends them directly to CloudWatch Logs using the
PutLogEvents API.

Data Conversion Copy Chapter

Convert OpenTelemetry Int64DataPoints, DoubleDataPoints, SummaryDataPoints metrics datapoints into
CloudWatch EMF structured log formats and send it to CloudWatch. Logs and Metrics will be displayed in
CloudWatch console. NaN, Inf values are not supported by CloudWatch EMF and will be dropped by the exporter.

Exporter Configuration Copy Chapter

The following exporter configuration parameters are supported.

metric_declaration Copy Section

A metric_declaration section characterizes a rule to be used to set dimensions for exported metrics, filtered by the incoming metrics' labels and metric names.

label_matcher

A label_matcher section defines a matching rule against the labels of the incoming metric. Only metrics that match the rules will be used by the surrounding metric_declaration.

metric_descriptor Copy Section

A metric descriptor section allows the schema of a metric to be overwritten before sending out to the CloudWatch backend service. Currently, we only support unit override.

AWS Credential Configuration Copy Chapter

This exporter follows default credential resolution for the
aws-sdk-go.

Follow the guidelines for the
credential configuration.

Metric Attributes Copy Chapter

By setting attributes on your metrics you can change how individual metrics are sent to CloudWatch. Attributes can be set in code or using components like the Attribute Processor.

The AWS EMF Exporter will interpret the following metric attributes to change how it publishes metrics to CloudWatch:

Configuration Examples Copy Chapter

Resource Attributes to Metric Labels Copy Section

The resource_to_telemetry_conversion option can be used to convert resource attributes to metric labels. By default, this option is disabled.
You can specify included and excluded wildcard pattern lists to match attribute keys.
Legacy options enabled and exclude_service_attributes are deprecated (and can be disabled via the exporter.awsemf.DisableLegacyResourceToTelemetryConversion feature gate). When migrating from enabled: true, the equivalent configuration is setting included: ["*"]. See the config example below:

exporters:
    awsemf:
        region: 'us-west-2'
        resource_to_telemetry_conversion:
            included:
                - "*"

Metric Declaration Copy Section

The following is an example of how to use metric_declaration to select what metrics should be exported.

exporters:
  awsemf:
    region: 'us-west-2'
    output_destination: stdout
    dimension_rollup_option: "NoDimensionRollup"
    metric_declarations:
      - dimensions: [[]]
        metric_name_selectors:
          # Metric without label
          - "^node_load15$"
      - dimensions: [[device, fstype], []]
        metric_name_selectors:
          - "^node_filesystem_readonly$"

Overview (`exporter/awskinesisexporter/README.md`) Copy Section

AWS Kinesis Exporter

The kinesis exporter currently exports dynamic encodings to the configured kinesis stream.
The exporter relies heavily on the kinesis.PutRecords api to reduce network I/O and reduces records into smallest atomic representation
to avoid hitting the hard limits placed on Records (No greater than 1Mb).
This producer will block until the operation is done to allow for retryable and queued data to help during high loads.

The following settings are required:

• aws
  • stream_name (no default): The name of the Kinesis stream to export to.

The following settings can be optionally configured:

• aws
  • kinesis_endpoint (no default)
  • region (default = us-west-2): the region that the kinesis stream is deployed in
  • role (no default): The role to be used in order to send data to the kinesis stream
• encoding
  • name (default = otlp): defines the export type to be used to send to kinesis (available is otlp_proto, otlp_json, zipkin_proto, zipkin_json, jaeger_proto)
    • Note : otlp_json is considered experimental and should not be used for production environments.
  • compression (default = none): allows to set the compression type (defaults BestSpeed for all) before forwarding to kinesis (available is flate, gzip, zlib or none)
• max_records_per_batch (default = 500, PutRecords limit): The number of records that can be batched together then sent to kinesis.
• max_record_size (default = 1Mb, PutRecord(s) limit on record size): The max allowed size that can be exported to kinesis
• timeout (default = 5s): Is the timeout for every attempt to send data to the backend.
• retry_on_failure
  • enabled (default = true)
  • initial_interval (default = 5s): Time to wait after the first failure before retrying; ignored if enabled is false
  • max_interval (default = 30s): Is the upper bound on backoff; ignored if enabled is false
  • max_elapsed_time (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if enabled is false
• sending_queue
  • enabled (default = true)
  • num_consumers (default = 10): Number of consumers that dequeue batches; ignored if enabled is false
  • queue_size (default = 1000): Maximum number of batches kept in memory before dropping data; ignored if enabled is false;
    User should calculate this as num_seconds * requests_per_second where:
    • num_seconds is the number of seconds to buffer in case of a backend outage
    • requests_per_second is the average number of requests per seconds.

Example Configuration:

exporters:
  awskinesis:
    aws:
      stream_name: raw-trace-stream
      region: us-east-1
      role: arn:test-role

Overview (`exporter/awss3exporter/README.md`) Copy Section

AWS S3 Exporter

Schema supported Copy Chapter

This exporter targets to support proto/json format.

Exporter Configuration Copy Chapter

The following exporter configuration parameters are supported.

Marshaler Copy Section

Marshaler determines the format of data sent to AWS S3. Currently, the following marshalers are implemented:

• otlp_json (default): the OpenTelemetry Protocol format, represented as json.

• otlp_proto: the OpenTelemetry Protocol format, represented as Protocol Buffers. A single protobuf message is written into each object.

• sumo_ic: the Sumo Logic Installed Collector Archive format.

  • _sourceCategory, _sourceHost, and _sourceName is needed

    yaml Copy

    resource/add_source_category:
      attributes:
      - action: insert
        key: _sourceCategory
        value: "value"
      - action: insert
        key: _sourceHost
        value: "value"
      - action: insert
        key: _sourceName
        value: "value"

  This format is supported only for logs.

• body: export the log body as string.
  This format is supported only for logs.

Encoding Copy Section

Encoding overrides marshaler if present and sets to use an encoding extension defined in the collector configuration.

See https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding.

Compression Copy Section

• none (default): No compression will be applied
• gzip: Files will be compressed with gzip.
• zstd: Files will be compressed with zstd.

resource_attrs_to_s3 Copy Section

• s3_bucket: Defines which resource attribute's value should be used as the S3 bucket.
  When this option is set, it dynamically overrides s3uploader/s3_bucket.
  If the specified resource attribute exists in the data,
  its value will be used as the bucket; otherwise, s3uploader/s3_bucket will serve as the fallback.
• s3_prefix: Defines which resource attribute's value should be used as the S3 prefix.
  When this option is set, it dynamically overrides s3uploader/s3_prefix.
  If the specified resource attribute exists in the data,
  its value will be used as the prefix; otherwise, s3uploader/s3_prefix will serve as the fallback.

Example Configurations

Following example configuration defines to store output in 'eu-central' region and bucket named 'databucket'.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'

    # Optional (disabled by default)
    sending_queue:
      enabled: true
      num_consumers: 10
      queue_size: 100

    # Optional (5s by default)
    timeout: 20s

Logs and traces will be stored inside 'databucket' in the following path format.

metric/year=YYYY/month=MM/day=DD/hour=HH/minute=mm

Partition Formatting Copy Chapter

By setting the s3_partition_format option, users can specify the file path for their logs.
See the strftime reference for more formatting options.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'

In this case, logs and traces would be stored in the following path format.

metric/YYYY/MM/DD/HH/mm

Optionally along with s3_partition_format you can provide s3_partition_timezone as name from IANA Time Zone
database to change default local timezone to custom, for example UTC or Europe/London.

Base Path Configuration Copy Chapter

The s3_base_prefix option allows you to specify a root path inside the bucket that is not overridden by resource_attrs_to_s3. If provided, s3_prefix will be appended to this base path.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_base_prefix: 'environment/prod'
      s3_prefix: 'metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'

In this case, logs and traces would be stored in the following path format.

environment/prod/metric/YYYY/MM/DD/HH/mm

Data routing based on resource attributes Copy Chapter

When resource_attrs_to_s3/s3_bucket or resource_attrs_to_s3/s3_prefix is configured, the S3 bucket and/or prefix are dynamically derived from specified resource attributes in your data.
If the attribute values are unavailable, the bucket and prefix will fall back to the values defined in s3uploader/s3_bucket and s3uploader/s3_prefix respectively.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'
    resource_attrs_to_s3:
      s3_bucket: "com.awss3.bucket"
      s3_prefix: "com.awss3.prefix"

In this case, metrics, logs and traces would be stored in the following path format examples:

bucket1/prefix1/YYYY/MM/DD/HH/mm
bucket2/foo-prefix/YYYY/MM/DD/HH/mm
bucket3/prefix-bar/YYYY/MM/DD/HH/mm
databucket/metric/YYYY/MM/DD/HH/mm
...

Base Path with Resource Attributes Copy Chapter

When using both s3_base_prefix and resource_attrs_to_s3/s3_prefix, the s3_base_prefix is always used while s3_prefix can be dynamically overridden by resource attributes.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_base_prefix: 'environment/prod'
      s3_prefix: 'default-metric'
      s3_partition_format: '%Y/%m/%d/%H/%M'
    resource_attrs_to_s3:
      s3_prefix: "com.awss3.prefix"

In this configuration:

• Base Prefix: environment/prod (always included)
• Prefix: Dynamically set from resource attribute com.awss3.prefix if available, otherwise falls back to default-metric

Path format examples:

# When resource attribute com.awss3.prefix = "service-a/metrics"
environment/prod/service-a/metrics/YYYY/MM/DD/HH/mm

# When resource attribute com.awss3.prefix = "service-b/logs"  
environment/prod/service-b/logs/YYYY/MM/DD/HH/mm

# When resource attribute is unavailable (fallback)
environment/prod/default-metric/YYYY/MM/DD/HH/mm

This allows you to maintain consistent organizational structure (via base path) while dynamically routing different data types or services to specific subdirectories.

Retry Copy Chapter

Standard is the default retryer implementation used by service clients. See the retry package documentation for details on what errors are considered as retryable by the standard retryer implementation.

See also the aws-sdk-go reference for more information.

exporters:
  awss3:
    s3uploader:
      region: 'eu-central-1'
      s3_bucket: 'databucket'
      s3_prefix: 'metric'
      retry_mode: "standard"
      retry_max_attempts: 5
      retry_max_backoff: "30s"

AWS Credential Configuration Copy Chapter

This exporter follows default credential resolution for the
aws-sdk-go.

Follow the guidelines for the
credential configuration.

OpenTelemetry Collector Helm Chart for Kubernetes Copy Section

For example, when using OpenTelemetry Collector Helm Chart you could use extraEnvs in the values.yaml.

extraEnvs:
- name: AWS_ACCESS_KEY_ID
  value: "< YOUR AWS ACCESS KEY >"
- name: AWS_SECRET_ACCESS_KEY
  value: "< YOUR AWS SECRET ACCESS KEY >"

Overview (`exporter/awsxrayexporter/README.md`) Copy Section

AWS X-Ray Tracing Exporter

This exporter converts OpenTelemetry spans to
AWS X-Ray Segment Documents
and then sends them directly to X-Ray using the
PutTraceSegments API.

Data Conversion Copy Chapter

Trace IDs and Span IDs are expected to be originally generated by either AWS API Gateway or AWS ALB and
propagated by them using the X-Amzn-Trace-Id HTTP header. However, other generation sources are
supported by replacing fully-random Trace IDs with X-Ray formatted Trace IDs where necessary:

AWS X-Ray IDs in binary are 128 bits, the same size as W3C Trace Context IDs but the string is formatted to
“1-{8 digit hex}-{24 digit hex}“. For example, The W3C format trace ID “4bf92f3577b34da6a3ce929d0e0e4736” is
converted to the X-Ray format trace ID “1-4bf92f35-77b34da6a3ce929d0e0e4736".

The http object is populated when the component attribute value is grpc as well as http. Other
synchronous call types should also result in the http object being populated.

AWS Specific Attributes Copy Chapter

The following AWS-specific Span attributes are supported in addition to the standard names and values
defined in the OpenTelemetry Semantic Conventions.

Any of these values supplied are used to populate the aws object in addition to any relevant data supplied
by the Span Resource object. X-Ray uses this data to generate inferred segments for the remote APIs.

Exporter Configuration Copy Chapter

The following exporter configuration parameters are supported. They mirror and have the same effect as the
comparable AWS X-Ray Daemon configuration values.

Traces and logs correlation Copy Chapter

AWS X-Ray can be integrated with CloudWatch Logs to correlate traces with logs. For this integration to work, the X-Ray
segments must have the AWS Property cloudwatch_logs set. This property is set using the AWS X-Ray exporter with the
following values that are evaluated in this order:

1. aws.log.group.arns resource attribute.
2. aws.log.group.names resource attribute.
3. aws_log_groups configuration property.

In the case of multiple values are defined, the value with higher precedence will be used to set the cloudwatch_logs AWS Property.

aws.log.group.arns and aws.log.group.names are slice resource attributes that can be set programmatically.
Alternatively those resource attributes can be set using the OTEL_RESOURCE_ATTRIBUTES environment variable. To set multiple log group names /log group arns, you can use &
to separate them. For example, 3 log groups log-group1, log-group2, and log-group3 are set in the following command:

export OTEL_RESOURCE_ATTRIBUTES="aws.log.group.names=log-group1&log-group2&log-group3"

AWS Credential Configuration Copy Chapter

This exporter follows default credential resolution for the
aws-sdk-go.

Follow the guidelines for the
credential configuration.

Overview (`exporter/azureblobexporter/README.md`) Copy Section

Azure Blob Storage Exporter

Configuration Copy Chapter

The following settings are required:

• url: Must be specified if auth type is not connection_string. If auth type is connection_string, it's optional or will be override by the auth.connection_string. Azure storage account endpoint. This setting might be replaced with endpoint for future. e.g. https://.blob.core.windows.net/
• auth (no default): Authentication method for exporter to ingest data.
  • type (no default): Authentication type for exporter. supported values are: connection_string, service_principal, system_managed_identity, user_managed_identity and workload_identity.
  • tenand_id: Tenand Id for the client, only needed when type is service_principal and workload_identity.
  • client_id: Client Id for the auth, only needed when type is service_principal, user_managed_identity and workload_identity.
  • client_secret: Secret for the client, only needed when type is service_principal.
  • connection_string: Connection string to the endpoint. Only needed for connection_string auth type. Once provided, it'll override the url parameter to the storage account.
  • federated_token_file: The path of the projected service account token file, only needed when type is workload_identity.

The following settings can be optionally configured and have default values:

• container: container for metrics, logs and traces. A container organizes a set of blobs, similar to a directory in a file system. More details can refer this.
  • metrics (default metrics): container to store metrics. default value is metrics.
  • logs (default logs): container to store logs. default value is logs.
  • traces (default traces): container to store traces. default value is traces.
• blob_name_format: the final blob name will be blob_name
  • template_enabled (default false): enables Go template parsing for blob name formats. If parsing fails, it will not throw an error but will log a warning and continue formatting the blob name using other rules.
  • metrics_format (default 2006/01/02/metrics_15_04_05.json): blob name format. The date format follows constants in Golang, refer here.
  • logs_format (default 2006/01/02/logs_15_04_05.json): blob name format.
  • traces_format (default 2006/01/02/traces_15_04_05.json): blob name format.
  • timezone (default ""): Timezone for blob name formatting. Local time is used if empty. Must be a valid IANA timezone identifier accepted by Golang's time.LoadLocation, such as UTC or America/New_York.
  • serial_num_enabled (default true): toggles whether a random serial number is appended to the blob name.
  • serial_num_range (default 10000): a range of random number to be appended after blob_name. e.g. blob_name_{serial_num}. the number will be in [0, serial_num_range).
  • serial_num_before_extension (default false): places the serial number before the file extension if there is one. e.g blob_name_{serial_num}.json instead of blob_name.json_{serial_num}
  • time_parser_enabled (default true): controls whether the exporter interprets the format string as a Go time layout. When false, values such as 2006 or 15_04_05 are treated as literal text.
  • time_parser_ranges (default nil): limits time formatting to specific parts of the blob name. When not set (nil), the entire blob name is time-formatted if time_parser_enabled is true. Provide a list of character ranges like ["0-10", "15-25"] to only apply time formatting within those positions. For example, if your blob name is prefix/2006/01/02/file.json and you set time_parser_ranges: ["7-17"], only the 2006/01/02 portion will be replaced with actual date values, while prefix/ and /file.json remain unchanged. This is helpful when your blob name contains patterns like 2006 that you want to keep as literal text.
• compression (default ""): sets the algorithm used to process the payload before uploading to Azure Blob Storage. Valid values are gzip, zstd, or no value set (uncompressed). The appropriate extension (.gz or .zst) is automatically appended to the blob name.
• format (default json): json or proto. which present otel json or otel protobuf format, the file extension will be json or pb.
• encodings (default using encoding specified in format, which is json): if specified, uses the encoding extension to encode telemetry data. Overrides format.
  • logs (default nil): encoding component id.
  • metrics (default nil): encoding component id.
  • traces (default nil): encoding component id.
• append_blob: configures append blob behavior. When enabled, telemetry data is appended to a single blob instead of creating new blobs. This can be useful for aggregating data or reducing the number of blobs created.
  • enabled (default false): determines whether to use append blob mode.
  • separator (default \n): string to insert between appended data blocks.
• retry_on_failure
  • enabled (default = true)
  • initial_interval (default = 5s): Time to wait after the first failure before retrying; ignored if enabled is false
  • max_interval (default = 30s): Is the upper bound on backoff; ignored if enabled is false
  • max_elapsed_time (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if enabled is false

Blob Name Templates Copy Section

When template_enabled is true, you can use Go templates in metrics_format, logs_format, and traces_format to create dynamic blob names based on telemetry data. The root object for the template is the telemetry data itself (pmetric.Metrics, plog.Logs, or ptrace.Traces).

The following template functions are available:

An example configuration is provided as follows:

extensions:
  zpages:
    endpoint: localhost:55679
  text_encoding:
    encoding: utf8
    marshaling_separator: "\n"
    unmarshaling_separator: "\r?\n"

exporter:
  azure_blob/1:
    url: "https://<your-account>.blob.core.windows.net/"
    container:
      logs: "logs"
      metrics: "metrics"
      traces: "traces"
    blob_name_format:
      template_enabled: true
      metrics_format: `{{ getResourceMetricAttr . 0 "service.name" }}/2006/01/02/metrics.json`
      logs_format: `{{ getScopeLogAttr . 0 0 "scope.name" }}/2006/01/02/logs.json`
      traces_format: `{{ (getSpan . 0 0 0).Name }}/2006/01/02/traces.json`
      serial_num_enabled: true
      time_parser_enabled: true
    auth:
      type: "connection_string"
      connection_string: "DefaultEndpointsProtocol=https;AccountName=<your-acount>;AccountKey=<account-key>;EndpointSuffix=core.windows.net"
    encodings:
      logs: text_encoding
    append_blob:
      enabled: true
      separator: "\n"
    compression: gzip  # or "zstd" or leave unset for no compression

Append Blob Copy Section

When append_blob is enabled:

• The exporter will create append blobs instead of block blobs
• New data will be appended to existing blobs rather than creating new ones
• The configured separator will be inserted between data blocks
• If the blob doesn't exist, it will be created automatically

Overview (`exporter/azuredataexplorerexporter/README.md`) Copy Section

Azure Data Explorer Exporter

This exporter sends metrics, logs and trace data to
Azure Data Explorer,
Azure Synapse Data Explorer and
Real time analytics in Fabric

Configuration Copy Chapter

The following settings are required:

• cluster_uri (no default): The cluster name of the provisioned ADX cluster to ingest the data.

One authentication method is required:

• Service principal:
  • application_id (no default): The client id to connect to the cluster and ingest data.
  • application_key (no default): The cluster secret corresponding to the client id.
  • tenant_id (no default): The tenant id where the application_id is referenced from.
• Managed identity:
  • managed_identity_id (no default): The managed identity id to authenticate with. Set to "system" for system-assigned managed identity. Set the MI client ID (GUID) for user-assigned managed identity.
• Default authentication:
  • use_azure_auth (default: false): Set to true to use the Azure default authentication.

The following settings can be optionally configured and have default values:

Note that the database tables are expected to be created upfront before the exporter is in operation , the definition of these are in the section Database and Table definition scripts

• db_name (default = "oteldb"): The ADX database where the tables are present to ingest the data.

• metrics_table_name (default = OTELMetrics): The target table in the database db_name that stores exported metric data.

• logs_table_name (default = OTELLogs): The target table in the database db_name that stores exported logs data.

• traces_table_name (default = OTELTraces): The target table in the database db_name that stores exported traces data.

  Optionally the following table mappings can be specified if the data needs to be mapped to a different table on ADX. This uses json table mapping that can be used to map attributes to target tables

• metrics_table_json_mapping (optional, no default): The table mapping name to be used for the table db_name.metrics_table_name

• logs_table_json_mapping (optional, no default): The table mapping name to be used for the table db_name.logs_table_name

• traces_table_json_mapping (optional, no default): The table mapping name to be used for the table db_name.traces_table_name

• ingestion_type (possible values=queued / managed, default = queued): ADX ingest can happen in managed streaming or queued modes.

Note: Streaming ingestion has to be enabled on ADX [configure the ADX cluster] in case of streaming option. Refer the query below to check if streaming is enabled

.show database <DB-Name> policy streamingingestion

An example configuration is provided as follows:

/* Detailed source-code truncated for AI context efficiency. */

Attribute mapping Copy Chapter

This exporter maps OpenTelemetry trace, metric and log attributes to specific structures in the ADX tables. These can then be extended by usage of update policies in ADX if needed

Traces Copy Section

Metrics Copy Section

Logs Copy Section

Database and Table definition scripts Copy Section

The following tables need to be created in the database specified in the configuration.

.create-merge table <Logs-Table-Name> (Timestamp:datetime, ObservedTimestamp:datetime, TraceID:string, SpanID:string, SeverityText:string, SeverityNumber:int, Body:string, ResourceAttributes:dynamic, LogsAttributes:dynamic) 

.create-merge table <Metrics-Table-Name> (Timestamp:datetime, MetricName:string, MetricType:string, MetricUnit:string, MetricDescription:string, MetricValue:real, Host:string, ResourceAttributes:dynamic,MetricAttributes:dynamic) 

.create-merge table <Traces-Table-Name> (TraceID:string, SpanID:string, ParentID:string, SpanName:string, SpanStatus:string, SpanKind:string, StartTime:datetime, EndTime:datetime, ResourceAttributes:dynamic, TraceAttributes:dynamic, Events:dynamic, Links:dynamic) 

// This is an optional column to store the status code and message as a dynamic field. This augments the status field with Status code and status message

.alter-merge table <Traces-Table-Name> (SpanStatusMessage:string) 

//Enable streaming ingestion( for managed streaming) for the created tables using
.alter table <Table-Name> policy streamingingestion enable

Optional configurations/Enhancements suggestions Copy Section

• Using update policies , the collected data can further be processed as per application need. The following is an example where histogram metrics are exported to a histo specific table (buckets and aggregates)

/* Detailed source-code truncated for AI context efficiency. */

OpenTelemetry Exporter Helper Configurations Copy Section

The ADX exporter now includes support for OpenTelemetry exporter helper configurations. This feature allows you to leverage the exporter helper capabilities(retries, timeout etc.) provided natively by Otel. Read more about the exporterhelper.

Please note that this configuration is not enabled by default. To utilize the OpenTelemetry exporter helper, you will need to add it manually to the configuration.

Example Configuration

# Example OpenTelemetry Exporter Configuration
  timeout: 10s
  sending_queue:
      enabled: true
      num_consumers: 2
      queue_size: 10
  retry_on_failure:
      enabled: true
      initial_interval: 10s
      max_interval: 60s
      max_elapsed_time: 10m

---

### Overview (`exporter/azuremonitorexporter/README.md`)

# Azure Monitor Exporter
| Status        |           |
| ------------- |-----------|
| Stability     | [beta]: traces, metrics, logs   |
| Distributions | [contrib] |
| Issues        | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fazuremonitor) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fazuremonitor) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_azuremonitor&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner)    | [@hgaol](https://www.github.com/hgaol) |
| Emeritus      | [@pcwiese](https://www.github.com/pcwiese) |

[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib


This exporter sends logs, traces and metrics to [Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/).

## Configuration

To configure the Azure Monitor Exporter, you must specify one of the following settings:

- `connection_string` (recommended): The Azure Application Insights Connection String is required to send telemetry data to the monitoring service. It is the recommended method for configuring the exporter, aligning with Azure Monitor's best practices. If you need guidance on creating Azure resources, please refer to the step-by-step guides to [Create an Application Insights resource](https://docs.microsoft.com/azure/azure-monitor/app/create-new-resource) and [find your connection string](https://docs.microsoft.com/azure/azure-monitor/app/sdk-connection-string?tabs=net#find-your-connection-string).
- `instrumentation_key`: Application Insights instrumentation key, which can be found in the Application Insights resource in the Azure Portal. While it is currently supported, its use is discouraged and it is slated for deprecation. It is highly encouraged to use the `connection_string` setting for new configurations and migrate existing configurations to use the `connection_string` as soon as possible.

### Environment Variable Support

In addition to the above configuration options, the Azure Monitor Exporter now supports setting the connection string via the `APPLICATIONINSIGHTS_CONNECTION_STRING` environment variable. This method is particularly useful for cloud or containerized environments where managing configuration through environment variables is standard practice.

**Note:** If both the environment variable and the `connection_string` configuration option are provided, the environment variable takes precedence.

### Configuration Options

**Important**: Only one of `connection_string` or `instrumentation_key` should be specified in your configuration. If both are provided, `connection_string` will be used as the priority setting.

The following settings can be optionally configured:

- `endpoint` (default = `https://dc.services.visualstudio.com/v2/track`): The endpoint URL where data will be submitted. While this option remains available, it is important to note that the use of the `connection_string` is recommended, as it encompasses the endpoint information. The direct configuration of the `endpoint` is considered to be on a deprecation path.
- `maxbatchsize` (default = 1024): The maximum number of telemetry items that can be submitted in each request. If this many items are buffered, the buffer will be flushed before `maxbatchinterval` expires.
- `maxbatchinterval` (default = 10s): The maximum time to wait before sending a batch of telemetry.
- `spaneventsenabled` (default = false): Enables export of span events.
- `sending_queue`
  - `enabled` (default = false)
  - `num_consumers` (default = 10): Number of consumers that dequeue batches; ignored if `enabled` is `false`
  - `queue_size` (default = 1000): Maximum number of batches kept in memory before data; ignored if `enabled` is `false`
  - `storage` (default = `none`): When set, enables persistence and uses the component specified as a storage extension for the persistent queue
- `shutdown_timeout` (default = 1s): Timeout to wait for graceful shutdown. Once exceeded, the component will shut down forcibly, dropping any element in queue.
- `custom_events_enabled` (default = `false`): Enables export log record to custom events when there's attribute `microsoft.custom_event.name` or `APPLICATION_INSIGHTS_EVENT_MARKER_ATTRIBUTE`.
- `telemetry_mappings.traces.http.success.server_policy` (default = unset): Set to `otel` to export HTTP server spans with 4xx response codes with Application Insights request `Success=true`, matching OpenTelemetry HTTP semantic conventions. HTTP client dependencies are unaffected.
- `telemetry_mappings.traces.http.success.additional_success_status_codes` (default = `[]`): Additional HTTP status codes to export with Application Insights `Success=true` for both HTTP server requests and HTTP client dependencies.

### Tag mappings (alpha)

The optional `tag_mappings:` block overrides how selected Application Insights envelope tags are populated from OpenTelemetry resource attributes. Each field takes an ordered list of sources; the first one that resolves to a non-empty string wins.

A source entry is interpreted as:
- a **resource attribute key** when it contains a `.` (e.g. `service.instance.id`, `host.name`)
- a **string literal terminal default** when it does not contain a `.` (e.g. `unknown-instance`)

Defaults preserve the historical hardcoded behavior — when a field is unset, zero-config users see no change.

| Mapping key           | Application Insights tag      | Default                  |
| --------------------- | ----------------------------- | ------------------------ |
| `cloud_role_instance` | `ai.cloud.roleInstance`       | `[service.instance.id]`  |
| `application_version` | `ai.application.ver`          | `[service.version]`      |

> The `cloud_role_name` tag (`ai.cloud.role`) is not yet configurable; its hardcoded `service.namespace`/`service.name` concatenation continues to apply.

Example — Azure Container Apps, where `service.instance.id` is set to a random uuid and operators want the revision/replica hostname instead:

```yaml
exporters:
  azure_monitor:
    connection_string: "InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://ingestion.azuremonitor.com/"
    tag_mappings:
      cloud_role_instance: [host.name, service.instance.id, unknown-instance]

Validation rejects an explicitly empty array (cloud_role_instance: []) at startup. The feature is alpha and the schema may evolve.

Example:

# Recommended Configuration:
# It is highly recommended to use the connection string which includes the InstrumentationKey and IngestionEndpoint
# This is the preferred method over using 'instrumentation_key' alone.
exporters:
  azure_monitor:
    connection_string: "InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://ingestion.azuremonitor.com/"

(or)

# Legacy Configuration:
# The use of 'instrumentation_key' alone is not recommended and will be deprecated in the future. It is advised to use the connection_string instead.
# This example is provided primarily for existing configurations that have not yet transitioned to the connection string.
exporters:
  azure_monitor:
    instrumentation_key: b1cd0778-85fc-4677-a3fa-79d3c23e0efd

Example using environment variable:

Ensure APPLICATIONINSIGHTS_CONNECTION_STRING is set in your environment, then configure the exporter without specifying a connection string or instrumentation key:

exporters:
  azure_monitor:

Attribute mapping Copy Chapter

Traces Copy Section

This exporter maps OpenTelemetry trace data to Application Insights data model using the following schema.

The OpenTelemetry SpanKind determines the Application Insights telemetry type.

The exporter follows the semantic conventions to fill the Application Insights specific telemetry properties. The following table shows a basic mapping.

The exact mapping can be found in trace_to_envelope.go.

All attributes are also mapped to custom properties if they are booleans or strings and to custom measurements if they are ints or doubles.

All links are mapped to property _MS.links with JSON array string.

Span Events

Span events are optionally saved to the Application Insights traces table.
Exception events are saved to the Application Insights exception table.

Logs Copy Section

This exporter saves log records to Application Insights traces table.
TraceId is mapped to operation_id column and SpanId is mapped to operation_parentId column.

Custom Events

When custom_events_enabled = true, azure monitor exporter will export log record to custom events when there's attribute microsoft.custom_event.name or APPLICATION_INSIGHTS_EVENT_MARKER_ATTRIBUTE.

Exceptions

This exporter saves exception records to Application Insights exceptions table when log records indicate an exception specification.

When exception_events_enabled = true, azure monitor exporter will export log records to exceptions when either one of exception.message or exception.type attributes are set.

Metrics Copy Section

This exporter saves metrics to Application Insights customMetrics table.

AAD/Entra Authentication Copy Chapter

Details of how to use the Azure Monitor Exporter with AAD/Entra based identities can be found in the Authentication page.

Overview (`exporter/bmchelixexporter/README.md`) Copy Section

BMC Helix Exporter

Note: The exporter type has been renamed from bmchelix to bmc_helix to follow the
snake_case naming convention. The old name bmchelix is preserved as a deprecated alias and
will continue to work, but a deprecation warning will be logged at startup. Please update your
configuration to use bmc_helix:.

This exporter supports sending metrics to BMC Helix Operations Management through its metric ingestion REST API.

Getting Started Copy Chapter

The following settings are required:

• endpoint: is the BMC Helix Portal URL of your environment, at onbmc.com for a BMC Helix SaaS tenant (e.g., https://company.onbmc.com), or your own Helix Portal URL for an on-prem instance.
• api_key: API key to authenticate the exporter. Connect to BMC Helix Operations Management, go to the Administration > Repository page, and click on the Copy API Key button to get your API Key. Alternatively, it is recommended to create and use a dedicated authentication key for external integration.

Example:

exporters:
  bmc_helix/helix1:
    endpoint: https://company.onbmc.com
    api_key: <api-key>
    sending_queue:
      batch:

Optional Settings Copy Section

The following settings can be optionally configured:

• timeout: (default = 10s) Timeout for requests made to the BMC Helix.
• enrich_metric_with_attributes: (default = true) When enabled, creates enriched metrics by appending datapoint attribute values to the metric name. This provides more detailed identification in BMC Helix Operations Management but increases metric cardinality. Set to false to reduce the number of unique metric series.
• retry_on_failure details here
  • enabled (default = true)
  • initial_interval (default = 5s) Time to wait after the first failure before retrying; ignored if enabled is false.
  • max_interval (default = 30s) The upper bound on backoff; ignored if enabled is false.
  • max_elapsed_time (default = 300s) The maximum amount of time spent trying to send a batch; ignored if enabled is false. If set to 0, the retries are never stopped.

Example:

exporters:
  bmc_helix/helix2:
    endpoint: https://company.onbmc.com
    api_key: <api-key>
    timeout: 20s
    enrich_metric_with_attributes: false
    sending_queue:
      batch:
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 1m
      max_elapsed_time: 8m

Setting Required Attributes for Metrics Copy Chapter

To ensure metrics are correctly populated in BMC Helix, the following attributes must be set either at the Resource level, or at the Metric level:

• entityName: Unique identifier for the entity. Used as display name if instanceName is missing.
• entityTypeId: Type identifier for the entity.
• instanceName: Display name of the entity.

Note: If entityName or entityTypeId is missing, the metric will not be exported.

To ensure the necessary attributes are present, it is recommended to leverage the transform processor with OTTL, and include it in the configuration of the telemetry pipeline.

The minimal pipeline most often looks like: OTEL metrics --> (memory limit) --> transform processor --> bmc_helix exporter.

Transformer Example for Hardware Metrics Copy Section

You can use the following OpenTelemetry Transformation Language (OTTL) configuration to map these attributes dynamically:

/* Detailed source-code truncated for AI context efficiency. */

This transformer dynamically sets the attributes required for BMC Helix based on metric names and resource attributes.

Overview (`exporter/cassandraexporter/README.md`) Copy Section

Cassandra Exporter

Configuration options Copy Chapter

The following settings can be optionally configured:

• dsn The Cassandra server DSN (Data Source Name), for example 127.0.0.1.
  reference: github.com/apache/cassandra-gocql-driver/v2
• port (default = 9042): The Cassandra server port
• timeout (default = 10s): The Cassandra server connection timeout
• keyspace (default = otel): The keyspace name.
• trace_table (default = otel_spans): The table name for traces.
• replication (default = class: SimpleStrategy, replication_factor: 1): The strategy of
  replication. https://cassandra.apache.org/doc/4.1/cassandra/architecture/dynamo.html#replication-strategy
• compression (default = LZ4Compressor): https://cassandra.apache.org/doc/4.0/cassandra/operating/compression.html
• auth (default = username: "", password: "") Authorization for the Cassandra.

Example Copy Chapter

exporters:
  cassandra:
    dsn: 127.0.0.1
    port: 9042
    timeout: 10s
    keyspace: "otel"
    trace_table: "otel_spans"
    replication:
      class: "SimpleStrategy"
      replication_factor: 1
    compression:
      algorithm: "ZstdCompressor"
    auth:
      username: "your-username"
      password: "your-password"

Overview (`exporter/clickhouseexporter/README.md`) Copy Section

ClickHouse Exporter

This exporter supports sending OpenTelemetry data to ClickHouse.

ClickHouse is an open-source, high performance columnar OLAP database management system for real-time analytics using
SQL.
Throughput can be measured in rows per second or megabytes per second.
If the data is placed in the page cache, a query that is not too complex is processed on modern hardware at a speed of
approximately 2-10 GB/s of uncompressed data on a single server.
If 10 bytes of columns are extracted, the speed is expected to be around 100-200 million rows per second.

Note:
Batching Recommendation
For optimal performance, ClickHouse recommends inserting data in large batches:

We recommend inserting data in packets of at least 5000 rows, or no more than a single request per second. When inserting to a MergeTree table from a tab-separated dump, the insertion speed can be from 50 to 200 MB/s.

To achieve this natively, enable batching within the exporter's sending_queue configuration. You do not need to add the external batch processor to your collector pipeline. Relying on the exporter's internal batching is the recommended approach to avoid data-loss issues associated with the external processor.

Enable it by adding a batch block inside sending_queue:

exporters:
  clickhouse:
    endpoint: tcp://127.0.0.1:9000
    sending_queue:
      # num_consumers controls how many batches are inserted into ClickHouse
      # concurrently.
      num_consumers: 10
      batch:
        min_size: 5000      # rows per INSERT (items sizer); tune to your workload
        flush_timeout: 5s   # flush a partial batch after this delay

If you are migrating from a pipeline that uses the standalone batch processor, remove batch from the pipeline's processors list and configure sending_queue.batch instead. For durability across restarts, also set sending_queue.storage to a storage extension so queued batches survive a crash (at-least-once delivery).

Visualization Tools Copy Chapter

Official ClickHouse Plugin for Grafana

The official ClickHouse Datasource for Grafana contains features that integrate directly with this exporter.
You can view associated logs and traces, as well as visualize other queries such as tables and time series graphs. Learn how to configure the OpenTelemetry integration.

Altinity's ClickHouse Plugin for Grafana

If the official plugin doesn't meet your needs, you can try the Altinity plugin for ClickHouse, which also supports a wide range of features.

Logs Copy Section

• Get log severity count time series.

SELECT toDateTime(toStartOfInterval(Timestamp, INTERVAL 60 second)) as time, SeverityText, count() as count
FROM otel_logs
WHERE toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
GROUP BY SeverityText, time
ORDER BY time;

The default logs table is ordered by (toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp).
For time range queries, filter on both toStartOfFiveMinutes(Timestamp) and Timestamp, and order by the tuple (toStartOfFiveMinutes(Timestamp), Timestamp) to use the primary key's read-in-order optimization.
Apply toStartOfFiveMinutes to the range bound as well (e.g. toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)) so the time bucket bounds are not truncated off the scan.

• Find any log.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;

• Find log with specific service.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE ServiceName = 'clickhouse-exporter'
  AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;

• Find log with specific attribute.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE LogAttributes['container_name'] = '/example_flog_1'
  AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;

• Find log with body contain string token.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE hasToken(Body, 'http')
  AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;

• Find log with body contain string.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE Body like '%http%'
  AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;

• Find log with body regexp match string.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE match(Body, 'http')
  AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;

• Find log with body json extract.

SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE JSONExtractFloat(Body, 'bytes') > 1000
  AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;

Traces Copy Section

• Find spans with specific attribute.

SELECT Timestamp,
       TraceId,
       SpanId,
       ParentSpanId,
       SpanName,
       SpanKind,
       ServiceName,
       Duration,
       StatusCode,
       StatusMessage,
       toString(SpanAttributes),
       toString(ResourceAttributes),
       toString(Events.Name),
       toString(Links.TraceId)
FROM otel_traces
WHERE ServiceName = 'clickhouse-exporter'
  AND SpanAttributes['peer.service'] = 'telemetrygen-server'
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
Limit 100;

• Find traces with traceID (using time primary index and TraceID skip index).

WITH
    '391dae938234560b16bb63f51501cb6f' as trace_id,
    (SELECT min(Start) FROM otel_traces_trace_id_ts WHERE TraceId = trace_id) as start,
    (SELECT max(End) + 1 FROM otel_traces_trace_id_ts WHERE TraceId = trace_id) as end
SELECT Timestamp,
       TraceId,
       SpanId,
       ParentSpanId,
       SpanName,
       SpanKind,
       ServiceName,
       Duration,
       StatusCode,
       StatusMessage,
       toString(SpanAttributes),
       toString(ResourceAttributes),
       toString(Events.Name),
       toString(Links.TraceId)
FROM otel_traces
WHERE TraceId = trace_id
  AND Timestamp >= start
  AND Timestamp <= end
Limit 100;

• Find spans is error.

SELECT Timestamp,
       TraceId,
       SpanId,
       ParentSpanId,
       SpanName,
       SpanKind,
       ServiceName,
       Duration,
       StatusCode,
       StatusMessage,
       toString(SpanAttributes),
       toString(ResourceAttributes),
       toString(Events.Name),
       toString(Links.TraceId)
FROM otel_traces
WHERE ServiceName = 'clickhouse-exporter'
  AND StatusCode = 'Error'
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
Limit 100;

• Find slow spans.

SELECT Timestamp,
       TraceId,
       SpanId,
       ParentSpanId,
       SpanName,
       SpanKind,
       ServiceName,
       Duration,
       StatusCode,
       StatusMessage,
       toString(SpanAttributes),
       toString(ResourceAttributes),
       toString(Events.Name),
       toString(Links.TraceId)
FROM otel_traces
WHERE ServiceName = 'clickhouse-exporter'
  AND Duration > 1 * 1e9
  AND Timestamp >= NOW() - INTERVAL 1 HOUR
Limit 100;

Metrics Copy Section

Metrics data is stored in different clickhouse tables depending on their types. The tables will have a suffix to
distinguish which type of metrics data is stored.

Before you make a metrics query, you need to know the type of metric you wish to use. If your metrics come from
Prometheus(or someone else uses OpenMetrics protocol), you also need to know the
compatibility
between Prometheus(OpenMetrics) and OTLP Metrics.

• Find a sum metrics with name

select TimeUnix,MetricName,Attributes,Value from otel_metrics_sum
where MetricName='calls' limit 100

• Find a sum metrics with name, attribute.

select TimeUnix,MetricName,Attributes,Value from otel_metrics_sum
where MetricName='calls' and Attributes['service_name']='featureflagservice'
limit 100

The OTLP Metrics define two type value for one datapoint,
clickhouse only use one value of float64 to store them.

Profiles Copy Section

[!IMPORTANT]
Profiles support is at development stability. The OpenTelemetry profiling signal itself is
pre-GA (the OTLP profiles protocol is in v1development), so the schema and behavior may change
in a backwards-incompatible way. To send profiles through a collector pipeline you must enable the
service.profilesSupport feature gate (--feature-gates=+service.profilesSupport).

The profiles table requires ClickHouse 26.2 or newer. It always uses text (full-text-search)
indexes and does not fall back to bloom_filter on older server versions. If you manage the schema
yourself (create_schema: false), you can adapt the DDL for an older version.

Profiles are stored as one denormalized row per OTLP Sample. The interned ProfilesDictionary
(strings, functions, locations, mappings, links, attributes) is resolved at write time so each row
is self-contained and can be queried without joins.

Performance Guide Copy Chapter

A single ClickHouse instance with 32 CPU cores and 128 GB RAM can handle around 20 TB (20 Billion) logs per day,
the data compression ratio is 7 ~ 11, the compressed data store in disk is 1.8 TB ~ 2.85 TB,
add more clickhouse node to cluster can increase linearly.

The otel-collector with otlp receiver/clickhouse tcp exporter (with sending_queue batching enabled) can process
around 40k/s logs entry per CPU cores, add more collector node can increase linearly.

Reading a shared table by a resource attribute Copy Section

The default schemas order primarily by time and ServiceName, and store resource-level
labels (for example a tenant or namespace attribute) in the ResourceAttributes map
rather than in the sort key. A skip index on the map values lets a query that filters on a
single attribute value prune granules, but because every value shares the same
time-partitioned parts, a wide time-range read filtered to one value still scans granules
that also contain other values' rows — the filter is applied after the granule is read.

For a low-cardinality attribute this is negligible. If you query a shared table by a
high-cardinality attribute (for example one tenant out of hundreds or thousands, each
reading concurrently), that shared-parts scan can become the read bottleneck. In that case,
set create_schema: false and manage the DDL yourself so the
attribute participates in the primary key — for example by adding it early in ORDER BY or
via a projection — accepting that this diverges from the default schema.

Configuration options Copy Chapter

The following settings are required:

• endpoint (no default): The ClickHouse server address, support multi host with port, for example:

  • tcp protocol tcp://addr1:port,tcp://addr2:port or TLS tcp://addr1:port,addr2:port?secure=true
  • http protocol http://addr1:port,addr2:port or https https://addr1:port,addr2:port
  • clickhouse protocol clickhouse://addr1:port,addr2:port or TLS clickhouse://addr1:port,addr2:port?secure=true

  When multiple endpoints are provided, the driver handles load balancing and automatic failover. By default, it uses in_order strategy (tries endpoints in the order specified). Alternatively, use connection_open_strategy=round_robin (distributes connections evenly) or connection_open_strategy=random (randomly selects endpoints) in connection_params. See connection_open_strategy documentation.

Many other ClickHouse specific options can be configured through query parameters e.g. addr?dial_timeout=5s&compress=lz4. For a full list of options see the ClickHouse driver documentation

Connection options:

• username (default = ): The authentication username.
• password (default = ): The authentication password.
• ttl (default = 0): The data time-to-live example 30m, 48h. Also, 0 means no ttl.
• database (default = default): The database name. Overrides the database defined in endpoint when this setting is not equal to default.
• connection_params (default = {}). Extra connection parameters with map format. Query parameters provided in endpoint will be individually overwritten if present in this map. Parameters can be either driver parameters (e.g., connection_open_strategy, max_open_conns) that control client-side behavior, or ClickHouse session settings (e.g., max_execution_time) that are passed to the server. See the driver parameters list for recognized driver options; all others are treated as session settings.
• create_schema (default = true): When set to true, will run DDL to create the database and tables. (See schema management)
• compress (default = lz4): Controls the compression algorithm. Valid options: none (disabled), zstd, lz4 (default), gzip, deflate, br, true (lz4). Ignored if compress is set in the endpoint or connection_params.
• async_insert (default = true): Enables async inserts. Ignored if async inserts are configured in the endpoint or connection_params. Async inserts may still be overridden server-side.
• tls Advanced TLS configuration (See TLS).

Additional DSN features:

The underlying clickhouse-go module offers additional configuration. These can be set in the exporter's endpoint or connection_params config values.

• client_info_product Must be in productName/version format with comma separated entries. By default the exporter will append its binary build information. You can use this information to track the origin of INSERT statements in the system.query_log table.

ClickHouse tables:

• logs_table_name (default = otel_logs): The table name for logs.
• traces_table_name (default = otel_traces): The table name for traces.
• profiles_table_name (default = otel_profiles): The table name for profiles.
• metrics_tables
  • gauge
    • name (default = "otel_metrics_gauge")
  • sum
    • name (default = "otel_metrics_sum")
  • summary
    • name (default = "otel_metrics_summary")
  • histogram
    • name (default = "otel_metrics_histogram")
  • exponential_histogram
    • name (default = "otel_metrics_exp_histogram")

Cluster definition:

• cluster_name (default = ): Optional. If present, will include ON CLUSTER cluster_name when creating the database and tables.

Table engine:

• table_engine
  • name (default = MergeTree)
  • params (default = )

Modifies ENGINE definition when table is created. If not set then ENGINE defaults to MergeTree().
Can be combined with cluster_name to enable replication for fault tolerance.

Database engine:

• database_engine
  • name (default = )
  • params (default = )

Modifies ENGINE definition when the database is created. If not set then the server default is used. Not used for the default database.
With Replicated, ON CLUSTER is only used for the database since a Replicated database replicates table DDL itself.

Processing:

• timeout (default = 5s): The timeout for every attempt to send data to the backend.
• sending_queue
  • enabled (default = true)
  • num_consumers (default = 10): Number of concurrent consumers that dequeue and insert data into ClickHouse. Enabling batch does not reduce this parallelism, only the (cheap) queue reader becomes single-threaded, while inserts still run on up to num_consumers workers. Ignored if enabled is false.
  • queue_size (default = 1000): Maximum size of the queue, measured in sizer units. Data is dropped when the queue is full unless block_on_overflow is enabled.
  • sizer (default = requests): How queue_size is measured. One of requests, items, or bytes.
  • block_on_overflow (default = false): If true, waits for space when the queue is full instead of dropping data (applies backpressure to the pipeline).
  • storage (default = none): Name of a storage extension for a persistent, crash-safe queue. When unset, the queue is in-memory and is lost on restart.
  • batch (disabled by default): Batches data inside the sending queue. Add an empty batch: {} to enable it with the defaults below. This replaces the standalone batch processor; see the batching recommendation near the top of this document.
    • flush_timeout (default = 200ms): Time after which a batch is sent regardless of size.
    • min_size (default = 8192): Minimum batch size before it is sent, in batch.sizer units.
    • max_size (default = 0): Maximum batch size; 0 means no limit. When set, larger batches are split, and it must be >= min_size.
    • sizer (default = items): How batch size is measured. One of items or bytes (not requests). If unset, inherits sending_queue.sizer.
• retry_on_failure
  • enabled (default = true)
  • initial_interval (default = 5s): The Time to wait after the first failure before retrying; ignored if enabled
    is false
  • max_interval (default = 30s): The upper bound on backoff; ignored if enabled is false
  • max_elapsed_time (default = 300s): The maximum amount of time spent trying to send a batch; ignored if enabled
    is false