---
# OpenTelemetry Collector Contrib
This is a repository for OpenTelemetry Collector components that are not suitable for the [core repository](https://github.com/open-telemetry/opentelemetry-collector) of the collector.
The official distributions, core and contrib, are available as part of the [opentelemetry-collector-releases](https://github.com/open-telemetry/opentelemetry-collector-releases) repository. Some of the components in this repository are part of the "core" distribution, such as the Jaeger and Prometheus components, but most of the components here are only available as part of the "contrib" distribution. Users of the OpenTelemetry Collector are also encouraged to build their own custom distributions with the [OpenTelemetry Collector Builder](https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder), using the components they need from the core repository, the contrib repository, and possibly third-party or internal repositories.
Each component has its own support levels, as defined in the following sections. For each signal that a component supports, there's a stability level, setting the right expectations. It is possible then that a component will be **Stable** for traces but **Alpha** for metrics and **Development** for logs.
## Stability levels
Stability level for components in this repository follow the [definitions](https://github.com/open-telemetry/opentelemetry-collector#stability-levels) from the OpenTelemetry Collector repository.
## Gated features
Some features are hidden behind feature gates before they are part of the main code path for the component. Note that the feature gates themselves might be at different [lifecycle stages](https://github.com/open-telemetry/opentelemetry-collector/tree/main/featuregate#feature-lifecycle).
## Support
Each component is supported either by the community of OpenTelemetry Collector Contrib maintainers, as defined by the GitHub group [@open-telemetry/collector-contrib-maintainers](https://github.com/orgs/open-telemetry/teams/collector-contrib-maintainers), or by specific vendors. See the individual README files for information about the specific components.
The OpenTelemetry Collector Contrib maintainers may at any time downgrade specific components if they are deemed unmaintained or if they pose a risk to the repository and/or binary distribution.
Even though the OpenTelemetry Collector Contrib maintainers are ultimately responsible for the components hosted here, actual support will likely be provided by individual contributors, typically a code owner for the specific component.
## Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md).
### Maintainers
- [Alex Boten](https://github.com/codeboten), Grafana Labs
- [Andrzej Stencel](https://github.com/andrzej-stencel), Elastic
- [Antoine Toulme](https://github.com/atoulme), Splunk
- [Bogdan Drutu](https://github.com/bogdandrutu), Snowflake
- [Braydon Kains](https://github.com/braydonk), Google
- [Christos Markou](https://github.com/ChrsMark), Elastic
- [Dmitrii Anoshin](https://github.com/dmitryax), Splunk
- [Edmo Vamerlatti Costa](https://github.com/edmocosta), Elastic
- [Evan Bradley](https://github.com/evan-bradley), Dynatrace
- [Pablo Baeyens](https://github.com/mx-psi), DataDog
- [Sean Marciniak](https://github.com/MovieStoreGuy), Splunk
- [Tyler Helmuth](https://github.com/TylerHelmuth), Grafana Labs
- [Yang Song](https://github.com/songy23), DataDog
For more information about the maintainer role, see the [community repository](https://github.com/open-telemetry/community/blob/main/guides/contributor/membership.md#maintainer).
### Approvers
- [Arthur Silva Sens](https://github.com/ArthurSens), Grafana Labs
- [Constança Manteigas](https://github.com/constanca-m), Elastic
- [Curtis Robert](https://github.com/crobert-1), Splunk
- [David Ashpole](https://github.com/dashpole), Google
- [Israel Blancas](https://github.com/iblancasa), Coralogix
- [Joshua MacDonald](https://github.com/jmacd), Microsoft
- [Matt Wear](https://github.com/mwear), Dash0
- [Mike Goldsmith](https://github.com/MikeGoldsmith), Honeycomb
- [Paulo Dias](https://github.com/paulojmdias), Five9
- [Paulo Janotti](https://github.com/pjanotti), Splunk
- [Sam DeHaan](https://github.com/dehaansa), Grafana Labs
- [Vihas Makwana](https://github.com/VihasMakwana), Elastic
For more information about the approver role, see the [community repository](https://github.com/open-telemetry/community/blob/main/guides/contributor/membership.md#approver).
### Triagers
- [Benedikt Bongartz](https://github.com/frzifus), Red Hat
- [Bogdan Stancu](https://github.com/bogdan-st), Adobe
- [Dónal O'Sullivan](https://github.com/osullivandonal), Elastic
- [Douglas Camata](https://github.com/douglascamata), Coralogix
- [Dylan Strohschein](https://github.com/dyl10s), Dynatrace
- [George Krajcsovits](https://github.com/krajorama), Grafana Labs
- [James Moessis](https://github.com/jamesmoessis), Optiver
- [Jared Tan](https://github.com/JaredTan95), DaoCloud
- [Khushi Jain](https://github.com/khushijain21), Elastic
- [Murphy Chen](https://github.com/Frapschen), DaoCloud
- [Ondrej Dubaj](https://github.com/odubajDT), Dynatrace
- [Roger Coll](https://github.com/rogercoll), Elastic
- [Vibhanshu Singh](https://github.com/singhvibhanshu)
- Actively seeking contributors to triage issues
For more information about the triager role, see the [community repository](https://github.com/open-telemetry/community/blob/main/guides/contributor/membership.md#triager).
### Emeritus
- [Alolita Sharma](https://github.com/alolita), Triager
- [Andrew Wilkins](https://github.com/axw), Approver
- [Anthony Mirabella](https://github.com/Aneurysm9), Approver
- [Bryan Aguilar](https://github.com/bryan-aguilar), Approver
- [Daniel Jaglowski](https://github.com/djaglowski), Maintainer
- [Florian Bacher](https://github.com/bacherfl), Triager
- [Gabriel Aszalos](https://github.com/gbbr), Triager
- [Goutham Veeramachaneni](https://github.com/gouthamve), Triager
- [Juraci Paixão Kröhling](https://github.com/jpkrohling), Maintainer
- [Przemek Maciolek](https://github.com/pmm-sumo), Approver
- [Punya Biswal](https://github.com/punya), Triager
- [Ruslan Kovalov](https://github.com/kovrus), Approver
- [Steve Flanders](https://github.com/flands), Triager
- [Tigran Najaryan](https://github.com/tigrannajaryan), Maintainer
- [Ziqi Zhao](https://github.com/fatsheep9146), Approver
For more information about the emeritus role, see the [community repository](https://github.com/open-telemetry/community/blob/main/guides/contributor/membership.md#emeritus-maintainerapprovertriager).
### No Over-Representation
A community member cannot be promoted to be a Collector contrib maintainer if, after their promotion, the resulting maintainers group has more than one-fourth (25%) of the members affiliated with the same employer. Job changes and similar events might result in over-representation, and no new maintainers from the same company can be promoted until representation is balanced again. In the event of confusion or concern, the OpenTelemetry Collector SIG will defer to the CNCF definition of "same employer".
## PRs and Reviews
When creating a PR please follow the process [described
here](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CONTRIBUTING.md#how-to-structure-prs-to-get-expedient-reviews).
New PRs will be automatically associated with the reviewers based on
[CODEOWNERS](.github/CODEOWNERS). PRs will be also automatically assigned to one of the
maintainers or approvers for facilitation.
The facilitator is responsible for helping the PR author and reviewers to make progress
or if progress cannot be made for closing the PR.
If the reviewers do not have approval rights the facilitator is also responsible
for the official approval that is required for the PR to be merged and if the facilitator
is a maintainer they are responsible for merging the PR as well.
The facilitator is not required to perform a thorough review, but they are encouraged to
enforce Collector best practices and consistency across the codebase and component
behavior. The facilitators will typically rely on codeowner's detailed review of the code
when making the final approval decision.
Marking the PR with the `ready to merge` label should only happen (by triagers/approvers/maintainers)
once there is at least one approval from an approver, as per the description above.
Marking the PR with the `waiting-for-approvers` label can be done (by triagers/code owners/approvers/maintainers)
once the PR has been reviewed and approved by code owners, all review comments are addressed, CI is green,
and it is only waiting on a review from an approver or maintainer. Use this label instead of pinging
`@open-telemetry/collector-contrib-approvers`. Once an approver approves the PR, replace it with the `ready to merge` label.
---
### Overview (`.chloggen/README.md`)
### Changelog folder
This repo uses `chloggen` to manage its changelog files. You can find the source code for the tool [here](https://github.com/open-telemetry/opentelemetry-go-build-tools/tree/main/chloggen).
Here is a quick explanation of the `config.yaml` file for chloggen:
```yaml
# The directory that stores individual changelog entries.
# Each entry is stored in a dedicated yaml file.
# - 'chloggen new' will copy the 'template_yaml' to this directory as a new entry file.
# - 'chloggen validate' will validate that all entry files are valid.
# - 'chloggen update' will read and delete all entry files in this directory, and update 'changelog_md'.
# Specify as relative path from root of repo.
# (Optional) Default: .chloggen
entries_dir: .chloggen
# This file is used as the input for individual changelog entries.
# Specify as relative path from root of repo.
# (Optional) Default: .chloggen/TEMPLATE.yaml
template_yaml: .chloggen/TEMPLATE.yaml
summary_template: .chloggen/summary.tmpl
# The CHANGELOG file or files to which 'chloggen update' will write new entries
# (Optional) Default filename: CHANGELOG.md
change_logs:
user: CHANGELOG.md
api: CHANGELOG-API.md
# The default change_log or change_logs to which an entry should be added.
# If 'change_logs' is specified in this file, and no value is specified for 'default_change_logs',
# then 'change_logs' MUST be specified in every entry file.
default_change_logs: [user]
```
---
### Overview (`pkg/README.md`)
# `pkg` folder
The Go modules in this folder can help in building new components (receivers, processors, exporters, extensions...) and are meant for **external** consumption.
Carefully review the public API to ensure that we only expose what is deemed necessary, so that we can easily change the code without affecting external users.
Breaking changes can happen on any module on this folder under the [opentelemetry-collector contribution guidelines](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CONTRIBUTING.md#breaking-changes).
If you are introducing a new module that will only be used by components in the opentelemetry-collector-contrib repository, consider placing it on the `internal` folder instead.
This will reduce our public API footprint and maintenance burden.
---
### Overview (`testbed/README.md`)
# OpenTelemetry Collector Testbed
Testbed is a controlled environment and tools for conducting end-to-end tests for the Otel Collector,
including reproducible short-term benchmarks, correctness tests, long-running stability tests and
maximum load stress tests.
## Usage
For each type of tests that should have a summary report create a new directory and then a test suite function which utilizes `*testing.M`. This function should delegate all functionality to `testbed.DoTestMain` supplying a global instance of `testbed.TestResultsSummary` to it.
Each test case within the suite should create a `testbed.TestCase` and supply implementations of each of the various interfaces the `NewTestCase` function takes as parameters.
## DataFlow
`testbed.TestCase` uses `LoadGenerator` and `MockBackend` to further encapsulate pluggable components. `LoadGenerator`'s reference implementation `ProviderSender`, created via `testbed.NewLoadGenerator()` further encapsulates `DataProvider` and `DataSender` in order to generate and send data. Any type satisfying the `LoadGenerator` interface can be used with `testbed.NewLoadGeneratorTestCase()` to exercise functionality beyond the single component limitation of `ProviderSender`. `MockBackend` further encapsulates `DataReceiver` and provides consume functionality.
For instance, if using the existing end-to-end test, the general dataflow can be (Note that MockBackend does not really have a consumer instance, only to make it intuitive, this diagram draws it a separate module):
## Pluggable Test Components
* `DataProvider` - Generates test data to send to receiver under test.
* `PerfTestDataProvider` - Implementation of the `DataProvider` for use in performance tests. Tracing IDs are based on the incremented batch and data items counters.
* `GoldenDataProvider` - Implementation of `DataProvider` for use in correctness tests. Provides data from the "Golden" dataset generated using pairwise combinatorial testing techniques.
* `DataSender` - Sends data to the collector instance under test.
* `OTLPTraceDataSender`, `OTLPMetricsDataSender`, `OTLPLogsDataSender` - OTLP/gRPC and OTLP/HTTP variants live in the `testbed/testbed` package.
* All other senders live in their own sub-package under `testbed/datasenders/datasender`, e.g. `jaegerdatasender.NewJaegerGRPCDataSender` (`jaeger` receiver) or `zipkindatasender.NewZipkinDataSender` (`zipkin` receiver). See `testbed/datasenders/` for the full set.
* `DataReceiver` - Receives data from the collector instance under test and stores it for use in test assertions.
* `OTLPDataReceiver`, `OTLPHTTPDataReceiver` - live in the `testbed/testbed` package.
* All other receivers live in their own sub-package under `testbed/datareceivers/datareceiver`, e.g. `jaegerdatareceiver.NewJaegerDataReceiver` or `zipkindatareceiver.NewZipkinDataReceiver`. See `testbed/datareceivers/` for the full set.
* `OtelcolRunner` - Configures, starts and stops one or more instances of otelcol which will be the subject of testing being executed.
* `ChildProcess` - Implementation of `OtelcolRunner` runs a single otelcol as a child process on the same machine as the test executor.
* `InProcessCollector` - Implementation of `OtelcolRunner` runs a single otelcol as a go routine within the same process as the test executor.
* `TestCaseValidator` - Validates and reports on test results.
* `PerfTestValidator` - Implementation of `TestCaseValidator` for test suites using `PerformanceResults` for summarizing results.
* `CorrectnessTestValidator` - Implementation of `TestCaseValidator` for test suites using `CorrectnessResults` for summarizing results.
* `TestResultsSummary` - Records itemized test case results plus a summary of one category of testing.
* `PerformanceResults` - Implementation of `TestResultsSummary` with fields suitable for reporting performance test results.
* `CorrectnessResults` - Implementation of `TestResultsSummary` with fields suitable for reporting data translation correctness test results.
## Adding New Receiver and/or Exporters to the testbed
New `DataSender` and `DataReceiver` implementations live in their own sub-package
under `testbed/datasenders/datasender` and
`testbed/datareceivers/datareceiver`. Connector helpers follow the
same pattern under `testbed/dataconnectors/dataconnector`. Keeping
one component per package means each test binary only compiles the senders,
receivers, and connectors it actually exercises.
Generally, when designing a test for new exporter and receiver components, developers should mainly focus on designing and implementing the components with yellow background in the diagram above as the other components are implemented by the testbed framework:
* `DataSender` - This part should provide below interfaces for testing purpose:
* `Start()` - Start sender and connect to the configured endpoint. Must be called before sending data.
* `Flush()` - Send any accumulated data.
* `GetCollectorPort()` - Return the port to which this sender will send data.
* `GenConfigYAMLStr()` - Generate a config string to place in receiver part of collector config so that it can receive data from this sender.
* `ProtocolName()` - Return protocol name to use in collector config pipeline.
* `DataReceiver` - This part should provide below interfaces for testing purpose:
* `Start()` - Start receiver.
* `Stop()` - Stop receiver.
* `GenConfigYAMLStr()` - Generate a config string to place in exporter part of collector config so that it can send data to this receiver.
* `ProtocolName()` - Return protocol name to use in collector config pipeline.
* `Testing` - This part may vary from what kind of testing developers would like to do. In existing implementation, we can refer to [End-to-End testing](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/testbed/tests/e2e_test.go), [Metrics testing](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/testbed/tests/metric_test.go), [Traces testing](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/testbed/tests/trace_test.go), [Correctness Traces testing](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/testbed/correctnesstests/traces/correctness_test.go), and [Correctness Metrics testing](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/testbed/correctnesstests/metrics/metrics_correctness_test.go). For instance, if developers would like to design a trace test for a new exporter and receiver:
* ```go
func TestTrace10kSPS(t *testing.T) {
tests := []struct {
name string
sender testbed.DataSender
receiver testbed.DataReceiver
resourceSpec testbed.ResourceSpec
}{
{
"NewExporterOrReceiver",
xxxdatasender.NewXXXDataSender(testbed.DefaultHost, testutil.GetAvailablePort(t)),
xxxdatareceiver.NewXXXDataReceiver(testutil.GetAvailablePort(t)),
testbed.ResourceSpec{
ExpectedMaxCPU: XX,
ExpectedMaxRAM: XX,
},
},
...
}
processors := []ProcessorNameAndConfigBody{
{
Name: "batch",
Body: `
batch:
`,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
Scenario10kItemsPerSecond(
t,
test.sender,
test.receiver,
test.resourceSpec,
performanceResultsSummary,
processors,
)
})
}
}
```
## Running the tests
To run the tests, use the `e2e-test` Makefile target, which will compile the Collector and run the end-to-end test suites against it.
### Run all tests
Run the following at the root of the repo:
```
make e2e-test
```
### Run a particular test suite
Run the following at the root of the repo:
```
TESTS_DIR=correctnesstests/metrics make e2e-test
```
### Advanced usage
A Makefile is also located at [`testbed/Makefile`](./Makefile) that offers targets to directly run certain test suites. Note that these targets will not compile the Collector before running.
---
### Overview (`cmd/codecovgen/README.md`)
# codecovgen
This tool generates the [component list][1] in the `.codecov.yml` file.
Each Go module gets its own component, which allows to have code coverage results on a per-component basis.
[1]: https://about.codecov.io/product/feature/components/
---
### Overview (`cmd/githubgen/README.md`)
⚠️ This executable has moved to [opentelemetry-go-build-tools](https://github.com/open-telemetry/opentelemetry-go-build-tools/tree/main/githubgen).
See https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/37294.
---
### Overview (`cmd/golden/README.md`)
# Golden tester
| Status | |
| ------------- |-----------|
| Stability | [alpha]: metrics |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Acmd%2Fgolden) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Acmd%2Fgolden) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=cmd_golden&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@atoulme](https://www.github.com/atoulme) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
---
### Overview (`cmd/opampsupervisor/README.md`)
# OpAMP Supervisor for the OpenTelemetry Collector
| Status | |
| ------------- |-----------|
| Stability | [alpha]: metrics |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Acmd%2Fopampsupervisor) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Acmd%2Fopampsupervisor) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@evan-bradley](https://www.github.com/evan-bradley), [@atoulme](https://www.github.com/atoulme), [@tigrannajaryan](https://www.github.com/tigrannajaryan), [@douglascamata](https://www.github.com/douglascamata), [@dpaasman00](https://www.github.com/dpaasman00) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
This is an implementation of an OpAMP Supervisor that runs a Collector instance using configuration provided from an OpAMP server. This implementation
is following a design specified [here](./specification/README.md).
The design is still undergoing changes, and as such this implementation may change as well.
Binary and container images for the Supervisor are available under tags starting with `cmd/opampsupervisor` [here](https://github.com/open-telemetry/opentelemetry-collector-releases/tags).
## More information
If you'd like to learn more about OpAMP, see the
[OpAMP specification](https://github.com/open-telemetry/opamp-spec/blob/main/specification.md#open-agent-management-protocol).
For production-ready client and server implementations in Go, see the
[OpAMP Go repository](https://github.com/open-telemetry/opamp-go).
If you have questions or would like to participate, we'd love to hear from you.
There is a regular SIG meeting, see the
[meeting notes](https://docs.google.com/document/d/19WA5-ex8rNFIBIyVb5VqMXfWNmUQwppGhN8zBeNG0f4)
for more information. You can also get in touch with us in the
[#otel-opamp](https://cloud-native.slack.com/archives/C02J58HR58R) channel on
the CNCF Slack workspace.
## Installing the Supervisor
### Binary
Binaries are available as downloadable assets from [OpenTelemetry Collector releases](https://github.com/open-telemetry/opentelemetry-collector-releases/releases) with tags using the prefix `cmd/opampsupervisor`. You can download the file containing the binary for your OS and architecture, and install it on your machine manually.
#### Linux/macOS
Replace `` with your desired Supervisor version, and `` and `` with your platform-specific information:
```sh
curl --proto '=https' --tlsv1.2 -fL -o opampsupervisor \
"https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fopampsupervisor%2Fv/opampsupervisor___"
chmod +x opampsupervisor
```
#### Windows
Replace `` with your desired Supervisor version:
```sh
Invoke-WebRequest -Uri "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/cmd%2Fopampsupervisor%2Fv/opampsupervisor__windows_amd64.exe" -OutFile "opampsupervisor.exe"
Unblock-File -Path "opampsupervisor.exe"
```
### Container image
Container images are available for download from [GitHub Container registry](https://github.com/open-telemetry/opentelemetry-collector-releases/pkgs/container/opentelemetry-collector-releases%2Fopentelemetry-collector-opampsupervisor):
```sh
docker pull ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-opampsupervisor
```
## Using the Supervisor
To use the Supervisor, you will need four things:
1. A Supervisor binary, which can be obtained through the [installation](#installing-the-supervisor) step above.
2. A Collector binary that you would like to control through the Supervisor.
3. A Supervisor config file. See examples [here](./examples/).
4. A running OpAMP server.
To test the Supervisor with an example server, download the
[opamp-go](https://github.com/open-telemetry/opamp-go) repository, and run the
OpAMP example server in the `internal/examples/server` directory.
```shell
git clone git@github.com:open-telemetry/opamp-go.git
cd opamp-go/internal/examples/server
go run .
```
Visit [localhost:4321](http://localhost:4321) to verify that the server is running.
In a separate directory, create a Supervisor configuration file named `supervisor.yaml`. Copy one of the existing [examples](./examples/) or create a new configuration. The `agent::executable` should be set to the path of your Collector binary:
```yaml
agent:
executable:
```
Start the Supervisor:
```sh
./opampsupervisor --config=supervisor.yaml
```
The supervisor's own log encoding is configured under `telemetry.logs.encoding`.
Use `console` for plain-text logs with ISO8601 timestamps. Leave the field
unset (or set it to `json`) to keep the default JSON output.
```yaml
telemetry:
logs:
encoding: console
```
The supervisor can also configure resource detection for its own telemetry
using the experimental `telemetry.resource.detection/development` field.
```yaml
telemetry:
resource:
detection/development:
detectors:
- host: {}
- process: {}
```
Supported detector entries are `container`, `host`, `process`, and `service`.
Do not set the `OTEL_EXPERIMENTAL_CONFIG_FILE` environment variable for the
Supervisor process. `xotelconf.NewSDK` uses that file to configure the SDK and
it supersedes the configuration passed by the Supervisor. If the variable is
set (for example to control the Collector's resource config), it can also
override the Supervisor's own resource detection and telemetry resource
settings. The Supervisor logs a warning at startup when the variable is set
while `telemetry.resource.detection/development` is configured.
Visit [localhost:4321](http://localhost:4321) again to verify that your Collector appears in the Agents list.
## Collector crash log snippets
The Supervisor can include the tail of the Collector's logs in health and remote config failure messages when the Collector process exits unexpectedly. This is disabled by default because Collector logs may contain sensitive data. To enable it, set `agent::collector_crash_log_snippet_kib` to the maximum number of log KiB to include. A value of `0` disables the feature. `4` is a reasonable starting point. The maximum value is `1024` (1 MiB).
```yaml
agent:
collector_crash_log_snippet_kib: 4
```
## Persistent data storage
The supervisor persists some data to disk in order to mantain state between restarts. The directory where this data is stored may be specified via the supervisor configuration:
```yaml
storage:
directory: "/path/to/storage/dir"
```
By default, the supervisor will use `/var/lib/otelcol/supervisor` on posix systems, and `%ProgramData%/Otelcol/Supervisor` on Windows.
This directory will be created on supervisor startup if it does not exist.
## Healthcheck
The Supervisor can be configured to expose a healthcheck endpoint that can be used to determine whether the Supervisor is running and healthy. This can be configured in the Supervisor configuration file:
```yaml
healthcheck:
endpoint: localhost:13133
```
At the moment this healtcheck checks that the following conditions are met:
- The Supervisor is able to persist state to disk.
- The Supervisor is able to generate the agent's configuration.
For more details on the healthcheck configuration, see the see the [full list of `ServerConfig` options](https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp).
Note that the healthceck endpoint is not enabled by default. To enable it, you must explicitly set at least the `endpoint` field in the configuration.
## Startup Fallback Configuration
The Supervisor supports a startup fallback configuration mechanism that provides resilience when the OpAMP server is unreachable at startup and there's no previous configuration state persisted in disk. This is useful for ensuring the Collector can start with a known-good configuration during network outages or server maintenance. When the Supervisor successfully connects to the OpAMP server, the regular configuration (indicated by `agent::config_files`) is restored and any potential remote configuration received from the OpAMP server is applied.
If, when the Supervisor starts, there's previous configuration state persisted in disk, the Supervisor will use it to start the Collector.
### Configuration Options
The following options can be configured under the `agent` section:
```yaml
agent:
startup_fallback_configs:
- /path/to/fallback_config.yaml
```
| Option | Description |
|--------|-------------|
| `startup_fallback_configs` | List of paths to startup fallback configuration files to use. If more than one path is specified, they are merged in order. Together, these must form a complete, standalone Collector configuration. |
### Important Notes
- The startup fallback configuration is a set of **standalone configuration files**. It is intentionally not merged with the `agent::config_files` setting. This ensures predictable behavior without dependencies on other configuration files.
- The OpAMP extension and "own telemetry" configuration are automatically added to the fallback configuration. This maintains the Collector's own telemetry flowing and the communication between Supervisor and Collector.
- The Supervisor will continue attempting to reconnect to the OpAMP server even while using the fallback configuration.
### Example
See [examples/supervisor_fallback.yaml](./examples/supervisor_fallback.yaml) for a complete example configuration with fallback enabled.
## Automatic Remote Configuration Rollback
The Supervisor supports automatic rollback when a remote configuration received from
the OpAMP backend causes the Collector to fail to start correctly. This feature is
**disabled by default** and can be enabled in the Supervisor configuration file:
```yaml
agent:
automatic_config_rollback: true
```
When enabled, the Supervisor will watch the health of the Collector after applying remote
configurations received from the OpAMP backend and it will cache the last working one in
disk, besides the last received. If a remote configurations causes the Collector
to fail to start correctly, the Supervisor will automatically swap to the last working one
and restart the Collector. During this process, a new "APPLIED" status will be reported for
the last working configuration.
## Status
The OpenTelemetry OpAMP Supervisor is intended to be the reference
implementation of an OpAMP Supervisor, and as such will support all OpAMP
capabilities. Additionally, it follows a design document for the features it
intends to support.
For a list of open issues related to the Supervisor, see [these issues](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Acmd%2Fopampsupervisor).
**Key**:
✅: Fully implemented
⚠️: Implemented with caveats
### OpAMP capabilities
| OpAMP capability | Status |
|--------------------------------|----------------------------------------------------------------------------------|
| AcceptsRemoteConfig | ✅ |
| ReportsEffectiveConfig | ✅ |
| AcceptsPackages | |
| ReportsPackageStatuses | |
| ReportsOwnTraces | ✅ |
| ReportsOwnMetrics | ✅ |
| ReportsOwnLogs | ✅ |
| AcceptsOpAMPConnectionSettings | ✅ |
| AcceptsOtherConnectionSettings | ✅ |
| AcceptsRestartCommand | ✅ |
| ReportsHealth | ⚠️ |
| ReportsStatus | |
| ReportsRemoteConfig | ✅ |
| ReportsAvailableComponents | ✅ |
### Supervisor specification features
| Feature | Status |
|--------------------------------------------------------------------|----------------------------------------------------------------------------------|
| Offers Supervisor configuration including configuring capabilities | ✅ |
| Starts and stops a Collector using remote configuration | ✅ |
| Communicates with OpAMP extension running in the Collector | ✅ |
| Updates the Collector binary | |
| Configures the Collector to report it's own metrics over OTLP | ✅ |
| Configures the Collector to report it's own logs over OTLP | ✅ |
| Sanitization or restriction of Collector config | |
---
### Overview (`cmd/telemetrygen/README.md`)
# Telemetry generator for OpenTelemetry
| Status | |
| ------------- |-----------|
| Stability | [alpha]: metrics, traces, logs |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Acmd%2Ftelemetrygen) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Acmd%2Ftelemetrygen) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@mx-psi](https://www.github.com/mx-psi), [@codeboten](https://www.github.com/codeboten), [@Erog38](https://www.github.com/Erog38), [@bogdan-st](https://www.github.com/bogdan-st) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
This utility simulates a client generating **traces**, **metrics**, and **logs**. It is useful for testing and demonstration purposes.
## Installing
To install the latest version run the following command:
```console
go install github.com/open-telemetry/opentelemetry-collector-contrib/cmd/telemetrygen@latest
```
Check the [`go install` reference](https://go.dev/ref/mod#go-install) to install specific versions.
### Docker build
You can build locally the Docker image with:
```bash
make docker-telemetrygen
```
Using github actions, we also push a docker image on commit to main or on release to this [Github docker registry](https://github.com/orgs/open-telemetry/packages/container/package/opentelemetry-collector-contrib%2Ftelemetrygen).
## Running
First, you'll need an OpenTelemetry Collector to receive the telemetry data. Follow the project's instructions for a detailed setting up guide. The following configuration file should be sufficient:
config.yaml:
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
exporters:
debug:
verbosity: detailed
service:
pipelines:
logs:
receivers: [otlp]
exporters: [debug]
metrics:
receivers: [otlp]
exporters: [debug]
traces:
receivers: [otlp]
exporters: [debug]
```
Starting OpenTelemetry collector via docker:
```
docker run -p 4317:4317 -v $(pwd)/config.yaml:/etc/otelcol-contrib/config.yaml ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.149.0
```
Other options for running the collector are documented here https://opentelemetry.io/docs/collector/getting-started/
## Export timeout
The maximum time to wait for signals to reach the destination can be configured with the following flag:
```console
--timeout=10s #Defaults to 10s
```
## Batching signals
All telemetry signals have batching capability configurable with the following flags:
```console
--batch=false #Defaults to true, no flag needed for batching
```
```console
--batch-size=1000 #Defaults to 100
```
Once the OpenTelemetry Collector instance is up and running, run `telemetrygen` for your desired telemetry:
### Traces
```console
telemetrygen traces --otlp-insecure --duration 5s
```
Or, to generate a specific number of traces:
```console
telemetrygen traces --otlp-insecure --traces 1
```
Or, to generate traces continuously (use `--duration inf` with `--rate` for throttling):
```console
telemetrygen traces --otlp-insecure --duration inf --rate 0.1
```
To send traces in secure connection, see [examples/secure-tracing](../../examples/secure-tracing/)
Check `telemetrygen traces --help` for all the options.
### Logs
```console
telemetrygen logs --otlp-insecure --duration 5s
```
Or, to generate a specific number of logs:
```console
telemetrygen logs --otlp-insecure --logs 1
```
Or, to generate logs continuously:
```console
telemetrygen logs --otlp-insecure --duration inf --rate 0.1
```
To send logs in secure connection, see [examples/secure-tracing](../../examples/secure-tracing/)
Check `telemetrygen logs --help` for all the options.
### Metrics
```console
telemetrygen metrics --otlp-insecure --duration 5s
```
Or, to generate a specific number of metrics:
```console
telemetrygen metrics --otlp-insecure --metrics 1
```
Or, to generate metrics continuously:
```console
telemetrygen metrics --otlp-insecure --duration inf --rate 0.1
```
To send metrics in secure connection, see [examples/secure-tracing](../../examples/secure-tracing/)
Check `telemetrygen metrics --help` for all the options.
## Attributes
Custom attributes can be added at two different levels, using two different flags:
| Flag | Sets | Applied to |
| ---- | ---- | ---------- |
| `--otlp-attributes` | Resource attributes | The OTLP resource, once per run. They describe the entity producing the telemetry and are shared by every span, metric data point and log record. |
| `--telemetry-attributes` | Span, metric data point and log record attributes | Each generated telemetry item individually. |
Because `service.name` is a resource attribute, `--otlp-attributes service.name="my-service"` overrides the `--service` flag.
Both flags accept the same value formats: `key="value"`, `key=true`, `key=false`, `key=`, and `key=["value1", "value2", ...]` for slices. All items of a slice must have the same type. Both flags may be repeated to set multiple attributes.
String values must keep their double quotes when they reach `telemetrygen`, so escape them (or wrap the whole argument in single quotes) to prevent the shell from removing them:
```console
telemetrygen traces --otlp-insecure --traces 1 \
--otlp-attributes service.name=\"my-service\" \
--otlp-attributes host.name=\"host-1\" \
--telemetry-attributes http.request.method=\"GET\" \
--telemetry-attributes http.response.status_code=200
```
This generates a resource carrying `service.name` and `host.name`, and spans that each carry `http.request.method` and `http.response.status_code`.
---
### Overview (`config/configdbauth/README.md`)
# Database Authentication Configuration
This package provides the configuration for wiring a **connection-oriented**
component to a `dbauth` provider that supplies a credential at the moment the
component opens a connection — for example, a database receiver that needs a
username and password (or a short-lived token) to dial its database.
It is the configuration half of the `db_auth` framework. The provider
**interface** it resolves (`Provider`, `Credential`, `Request`) lives in a
separate, dependency-light module,
[`extension/dbauth`](../../extension/dbauth), so a provider extension can
implement the interface without taking on this package's confmap dependency
## How a provider is declared and selected
A credential provider is a Collector **extension** that also implements
`dbauth.Provider`. It is declared once (and listed in `service.extensions`) with
its provider-wide configuration, then each component references it **by component
ID** inside its `db_auth` block — the value of the block is the provider's
component ID:
```yaml
extensions:
aws_iam: # declared once, with provider-wide config
region: us-east-2
aws_iam/west: # a second instance for a different region
region: us-west-2
receivers:
postgresql/this:
endpoint: this-db:5432
username: monitor
db_auth: aws_iam # reference the extension by ID
postgresql/another:
endpoint: another-db:5432
username: reader
db_auth: aws_iam/west # reference a differently-configured instance
service:
extensions: [aws_iam, aws_iam/west]
```
The extension holds all provider-wide configuration (here, the AWS region). To
vary that configuration across components, declare multiple extension instances
(`aws_iam`, `aws_iam/west`) and point each component at the one it needs. There
is no inline override in the `db_auth` block — the value is only a reference.
The **per-connection** inputs a provider needs — the endpoint to connect to, the
database user — are *not* configured on the extension or repeated in the
`db_auth` block. The consuming component already knows them (its own
`endpoint` and `username`) and passes them to the provider with each
`GetCredential` call as a `dbauth.Request`. So one declared extension serves many
components that differ only by endpoint/user, with no per-database extension
instance and nothing repeated.
This means:
- A provider is **declared once per configuration** (one extension instance in
the collector build); any receiver can reference it by ID, with no per-receiver
provider list to maintain, and new providers require no receiver code changes.
- Databases that share provider config reference one instance; a database that
needs a different provider-wide value references a separate declared instance.
- Resolution mirrors `config/configauth`: the component finds the provider in
`host.GetExtensions()` at `Start()`, not via an `init()` registry.
## The credential and the request
The provider interface, `Credential`, and `Request` are defined in
[`extension/dbauth`](../../extension/dbauth):
```go
type Provider interface {
GetCredential(ctx context.Context, req Request) (*Credential, error)
}
type Request struct {
Endpoint string // host:port the credential is for
Username string // the consumer's configured username
}
type Credential struct {
Username *string // nil: use the consumer's configured username
Secret string // opaque: a password or a minted token
NotAfter *time.Time // nil: no expiry applies; advisory only
}
```
`Request` carries the per-connection inputs the consumer supplies on every call;
a provider uses the fields it needs (AWS IAM mints a token scoped to `Endpoint`
for `Username`) and ignores the rest. Provider-wide configuration (such as the
AWS region) lives on the extension's own config, not in the request.
The `Credential` carries credential *material* and nothing about *how* to apply
it — the consuming component decides where the secret goes (for a SQL driver,
usually the password slot of a connection string). The pointer fields are
load-bearing:
- `Username == nil` means the provider supplies no username and the consumer
should fall back to its own configured username. A non-nil pointer (including a
pointer to the empty string) means the provider generated the username and the
consumer must use it. Dynamic providers such as Vault mint both a username and a
secret per lease, which is why this is a pointer rather than a plain string.
- `NotAfter == nil` means no expiry applies (a static password). When set, it is
an advisory hint for observability — the provider owns refresh-before-expiry, so
consumers need not act on it.
## Genericity
The same `Credential` and the same `Provider` interface cover every connection
credential shape without an interface change. How each maps onto the struct:
| Provider | `Username` | `Secret` | `NotAfter` |
| ---------------------- | ----------------- | ---------------------- | ----------------- |
| Static (inline / file) | from config/file | the password | `nil` |
| AWS IAM (RDS/Aurora) | `nil` | minted RDS auth token | now + ~15m |
| Azure Managed Identity | `nil` | minted access token | token expiry |
| GCP Cloud SQL IAM | `nil` | minted access token | token expiry |
| Vault dynamic creds | minted (non-nil) | minted password | lease expiry |
---
### Overview (`connector/countconnector/README.md`)
# Count Connector
| Status | |
| ------------- |-----------|
| Distributions | [contrib], [k8s] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fcount) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fcount) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_count&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@akats7](https://www.github.com/akats7) \| Seeking more code owners! |
| Emeritus | [@djaglowski](https://www.github.com/djaglowski), [@jpkrohling](https://www.github.com/jpkrohling) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | metrics | [alpha] |
| metrics | metrics | [alpha] |
| logs | metrics | [alpha] |
| profiles | metrics | [alpha] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
The `count` connector can be used to count spans, span events, metrics, data points, and log records.
## Configuration
If you are not already familiar with connectors, you may find it helpful to first visit the [Connectors README].
### Default Configuration
The `count` connector may be used without any configuration settings. The following table describes the
default behavior of the connector.
| [Exporter Pipeline Type] | Description | Default Metric Names |
| ------------------------ | ----------------------------------- | -------------------------------------------- |
| traces | Counts all spans and span events. | `trace.span.count`, `trace.span.event.count` |
| metrics | Counts all metrics and data points. | `metric.count`, `metric.datapoint.count` |
| logs | Counts all log records. | `log.record.count` |
For example, in the following configuration the connector will count spans and span events from the `traces/in`
pipeline and emit metrics called `trace.span.count` and `trace.span.event.count` onto the `metrics/out` pipeline.
```yaml
receivers:
foo:
exporters:
bar:
connectors:
count:
service:
pipelines:
traces/in:
receivers: [foo]
exporters: [count]
metrics/out:
receivers: [count]
exporters: [bar]
```
### Custom Counts
Optionally, emit custom counts by defining metrics under one or more of the following sections:
- `spans`
- `spanevents`
- `metrics`
- `datapoints`
- `logs`
Optionally, specify a description for the metric.
Note: If any custom metrics are defined for a data type, the default metric will not be emitted.
#### Conditions
Conditions may be specified for custom metrics. If specified, data that matches any one
of the conditions will be counted. i.e. Conditions are ORed together.
```yaml
receivers:
foo:
exporters:
bar:
connectors:
count:
spanevents:
my.prod.event.count:
description: The number of span events from my prod environment.
conditions:
- 'spanevent.attributes["env"] == "prod"'
- 'spanevent.name == "prodevent"'
```
#### Attributes
`spans`, `spanevents`, `datapoints`, and `logs` may be counted according to attributes. In such cases, attribute precedence follows this order:
span(logRecord, DataPoint, profile) attributes > scope attributes > resource attributes.
If attributes are specified for custom metrics, a separate count will be generated for each unique
set of attribute values. Each count will be emitted as a data point on the same metric.
Optionally, include a `default_value` for an attribute, to count data that does not contain the attribute. The `default_value` value can be of type string, integer, or float.
```yaml
receivers:
foo:
exporters:
bar:
connectors:
count:
logs:
my.log.count:
description: The number of logs from each environment.
attributes:
- key: env
default_value: unspecified_environment
```
### Example Usage
Count spans and span events, only exporting the count metrics.
```yaml
receivers:
foo:
exporters:
bar:
connectors:
count:
service:
pipelines:
traces:
receivers: [foo]
exporters: [count]
metrics:
receivers: [count]
exporters: [bar]
```
Count spans and span events, exporting both the original traces and the count metrics.
```yaml
receivers:
foo:
exporters:
bar/traces_backend:
bar/metrics_backend:
connectors:
count:
service:
pipelines:
traces:
receivers: [foo]
exporters: [bar/traces_backend, count]
metrics:
receivers: [count]
exporters: [bar/metrics_backend]
```
Count spans, span events, metrics, data points, and log records, exporting count metrics to a separate backend.
```yaml
receivers:
foo/traces:
foo/metrics:
foo/logs:
exporters:
bar/all_types:
bar/counts_only:
connectors:
count:
service:
pipelines:
traces:
receivers: [foo/traces]
exporters: [bar/all_types, count]
metrics:
receivers: [foo/metrics]
exporters: [bar/all_types, count]
logs:
receivers: [foo/logs]
exporters: [bar/all_types, count]
metrics/counts:
receivers: [count]
exporters: [bar/counts_only]
```
Count logs with a severity of ERROR or higher.
```yaml
receivers:
foo:
exporters:
bar:
connectors:
count:
logs:
my.error.log.count:
description: Error+ logs.
conditions:
- `severity_number >= SEVERITY_NUMBER_ERROR`
service:
pipelines:
logs:
receivers: [foo]
exporters: [count]
metrics:
receivers: [count]
exporters: [bar]
```
Count logs with a severity of ERROR or higher. Maintain a separate count for each environment.
```yaml
receivers:
foo:
exporters:
bar:
connectors:
count:
logs:
my.error.log.count:
description: Error+ logs.
conditions:
- `severity_number >= SEVERITY_NUMBER_ERROR`
attributes:
- key: env
service:
pipelines:
logs:
receivers: [foo]
exporters: [count]
metrics:
receivers: [count]
exporters: [bar]
```
Count all spans and span events (default behavior). Count metrics and data points based on the `env` attribute.
```yaml
receivers:
foo/traces:
foo/metrics:
foo/logs:
exporters:
bar/all_types:
bar/counts_only:
connectors:
count:
metrics:
my.prod.metric.count:
conditions:
- `attributes["env"] == "prod"
my.test.metric.count:
conditions:
- `attributes["env"] == "test"
datapoints:
my.prod.datapoint.count:
conditions:
- `attributes["env"] == "prod"
my.test.datapoint.count:
conditions:
- `attributes["env"] == "test"
service:
pipelines:
traces:
receivers: [foo/traces]
exporters: [bar/all_types, count]
metrics:
receivers: [foo/metrics]
exporters: [bar/all_types, count]
metrics/counts:
receivers: [count]
exporters: [bar/counts_only]
```
[Connectors README]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md
---
### Overview (`connector/datadogconnector/README.md`)
# Datadog Connector
| Status | |
| ------------- |-----------|
| Unsupported Platforms | aix, solaris |
| Distributions | [contrib] |
| Warnings | [Statefulness](#warnings) |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fdatadog) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fdatadog) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_datadog&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@mx-psi](https://www.github.com/mx-psi), [@dineshg13](https://www.github.com/dineshg13), [@jade-guiton-dd](https://www.github.com/jade-guiton-dd), [@IbraheemA](https://www.github.com/IbraheemA) |
| Emeritus | [@gbbr](https://www.github.com/gbbr), [@ankitpatel96](https://www.github.com/ankitpatel96) |
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | metrics | [beta] |
| traces | traces | [beta] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
## Description
The Datadog Connector is a connector component that derives APM statistics, in the form of metrics, from service traces, for display in the Datadog APM product. This component is *required* for trace-emitting services and their statistics to appear in Datadog APM.
The Datadog connector can also forward the traces passed into it into another trace pipeline. Notably, if you plan to sample your traces with the [tailsamplingprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor#tail-sampling-processor) or the [probabilisticsamplerprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/probabilisticsamplerprocessor), you should place the Datadog connector upstream to ensure that the metrics are computed before sampling, ensuring their accuracy. An example is given below.
## Usage
```yaml
# ...
processors:
# ...
probabilistic_sampler:
sampling_percentage: 20
connectors:
# add the "datadog" connector definition and further configurations
datadog/connector:
exporters:
datadog:
api:
key: ${env:DD_API_KEY}
sending_queue:
batch:
service:
pipelines:
traces:
receivers: [otlp]
exporters: [datadog/connector]
traces/2: # this pipeline uses sampling
receivers: [datadog/connector]
processors: [probabilistic_sampler]
exporters: [datadog]
metrics:
receivers: [datadog/connector]
exporters: [datadog]
```
In this example configuration, incoming traces are received through OTLP, and processed by the Datadog connector in the `traces` pipeline. The traces are then forwarded to the `traces/2` pipeline, where a sample of them is exported to Datadog. In parallel, the APM stats computed from the full stream of traces are sent to the `metrics` pipeline, where they are exported to Datadog as well.
## Configurations
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Warnings
- [Statefulness](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/standard-warnings.md#statefulness): The `datadog` connector aggregates trace metrics into buckets defined by `bucket_interval` before flushing them, keeping this aggregation state in memory on a single collector instance. All spans that should be aggregated together must be routed to the same collector instance, or the resulting trace metrics will be incomplete or inconsistent.
---
### Overview (`connector/exceptionsconnector/README.md`)
# Exceptions Connector
| Status | |
| ------------- |-----------|
| Distributions | [contrib], [k8s] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fexceptions) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fexceptions) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_exceptions&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@marctc](https://www.github.com/marctc) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | metrics | [alpha] |
| traces | logs | [alpha] |
| logs | logs | [alpha] |
| logs | metrics | [alpha] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
## Overview
Generate metrics and logs from recorded [application exceptions](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/exceptions/exceptions-spans.md/) associated with spans.
Each **metric** and **log** will have _at least_ the following dimensions:
- Service name
- Span name
- Span kind
- Status code
With the provided default config, each **metric** and **log** will also have the following dimensions:
- Exception message
- Exception type
Each log will additionally have the following attributes:
- Exception stacktrace
- Span attributes. If you want to filter out some attributes (like only copying HTTP attributes starting with `http.`) use the [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor/).
## Configurations
If you are not already familiar with connectors, you may find it helpful to first visit the [Connectors README].
The following settings can be optionally configured:
- `dimensions`: the list of dimensions to add together with the default dimensions defined above.
Each additional dimension is defined with a `name` which is looked up in the span's collection of attributes or resource attributes.
The provided default config includes `exception.type` and `exception.message` as additional dimensions.
- `exemplars`: Use to configure how to attach exemplars to metrics.
- `enabled` (default: `false`): enabling will add spans as Exemplars.
## Examples
The following is a simple example usage of the `exceptions` connector.
```yaml
receivers:
nop:
exporters:
nop:
connectors:
exceptions:
service:
pipelines:
traces:
receivers: [nop]
exporters: [exceptions]
metrics:
receivers: [exceptions]
exporters: [nop]
logs:
receivers: [exceptions]
exporters: [nop]
```
The following is a more complex example usage of the `exceptions` connector using Prometheus and Loki as exporters.
```yaml
receivers:
otlp:
protocols:
grpc:
http:
exporters:
prometheus_remote_write:
endpoint: http://prometheus:9090/api/v1/write
loki:
endpoint: http://loki:3100/loki/api/v1/push
connectors:
exceptions:
service:
pipelines:
traces:
receivers: [otlp]
exporters: [exceptions]
metrics:
receivers: [exceptions]
exporters: [prometheus_remote_write]
logs:
receivers: [exceptions]
exporters: [loki]
```
The full list of settings exposed for this connector are documented in [exceptionsconnector/config.go](../../connector/exceptionsconnector/config.go).
### More Examples
For more example configuration covering various other use cases, please visit the [testdata directory](../../connector/exceptionsconnector/testdata).
[Connectors README]:https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md
---
### Overview (`connector/failoverconnector/README.md`)
# Failover Connector
| Status | |
| ------------- |-----------|
| Distributions | [contrib], [k8s] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Ffailover) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Ffailover) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_failover&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@akats7](https://www.github.com/akats7), [@khushijain21](https://www.github.com/khushijain21) |
| Emeritus | [@fatsheep9146](https://www.github.com/fatsheep9146) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | traces | [alpha] |
| metrics | metrics | [alpha] |
| logs | logs | [alpha] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
Allows for health based routing between trace, metric, and log pipelines depending on the health of target downstream exporters.
## Configuration
If you are not already familiar with connectors, you may find it helpful to first visit the [Connectors README].
The following settings are available:
- `priority_levels (required)`: list of pipeline level priorities in a 1 - n configuration, multiple pipelines can sit at a single priority level.
- `retry_interval (optional)`: the frequency at which the pipeline levels will attempt to reestablish connection with all higher priority levels. Default value is 10 minutes. (See Example below for further explanation)
- `sending_queue (optional)`: queues and batches data in the connector before it is routed to a priority level. Supports the common [`sending_queue` settings][exporterhelper] and is enabled by default.
We recommend configuring the queue here instead of on the exporters in the priority pipelines, and disabling `sending_queue` on those exporters.
Moving the queue to the connector avoids data being stuck in the queue of an unhealthy exporter, where it cannot be failed over to another priority level.
- `condition`: Conditional statements that decide when the connector should failover. Currently supports below types
- `error`:
- `contains`: ["network failure", "connection refused"] (required list of non-empty, case-insensitive substrings)
Only errors whose message contains any configured string trigger failover. Non-matching errors are returned
to the upstream caller without failing over, and they do not mark the pipeline level unhealthy during
retry sampling either. When `condition` is omitted, all errors trigger failover.
The connector intakes a list of `priority_levels` each of which can contain multiple pipelines.
If any pipeline at a stable level fails, the level is considered unhealthy and the connector will move down one priority level and route all data to the new level (assuming it is stable).
The connector will periodically try to reestablish a stable connection with the higher priority levels. `retry_interval` will be the frequency at which the connector will try to iterate through all unhealthy higher priority levels.
#### Configuration Example:
```yaml
connectors:
failover:
priority_levels:
- [traces/first, traces/also_first]
- [traces/second]
- [traces/third]
retry_interval: 10s
service:
pipelines:
traces:
receivers: [otlp]
exporters: [failover]
traces/first:
receivers: [failover]
exporters: [otlp_grpc/first]
traces/second:
receivers: [failover]
exporters: [otlp_grpc/second]
traces/third:
receivers: [failover]
exporters: [otlp_grpc/third]
traces/also_first:
receivers: [failover]
exporters: [otlp_grpc/fourth]
```
[Connectors README]:https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md
[Exporter Pipeline Type]:https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]:https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[exporterhelper]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md
---
### Overview (`connector/grafanacloudconnector/README.md`)
# Grafana Cloud Connector
| Status | |
| ------------- |-----------|
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fgrafanacloud) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fgrafanacloud) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_grafanacloud&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@rlankfo](https://www.github.com/rlankfo), [@jcreixell](https://www.github.com/jcreixell) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | metrics | [alpha] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
> **Deprecation Notice:** The component type has been renamed from `grafanacloud` to `grafana_cloud`
> to follow the OpenTelemetry snake_case naming convention.
> The old name `grafanacloud` still works but is deprecated and will be removed in a future release.
> Please update your configuration to use `grafana_cloud`.
## Overview
The Grafana Cloud Connector (grafanacloudconnector) is a connector component that can analyze telemetry in pipelines to generate usage metrics for the following Grafana Cloud products:
* Application Observability
## Usage
```yaml
connectors:
grafana_cloud/connector:
extensions:
basicauth/grafana_cloud:
client_auth:
username: "${env:GRAFANA_CLOUD_INSTANCE_ID}"
password: "${env:GRAFANA_CLOUD_API_KEY}"
exporters:
otlphttp/grafana_cloud:
endpoint: "${env:GRAFANA_CLOUD_OTLP_ENDPOINT}"
auth:
authenticator: basicauth/grafana_cloud
service:
pipelines:
traces:
receivers: [otlp]
exporters: [otlphttp/grafana_cloud, grafana_cloud/connector]
metrics:
receivers: [otlp, grafana_cloud/connector]
exporters: [otlphttp/grafana_cloud]
```
In this example configuration, usage metrics for Grafana Cloud Application Observability are derived from incoming spans.
## Configuration
```yaml
connectors:
grafanacloud/connector:
## @param host_identifiers List of span attributes used to identify the host running the workloads.
## The first matching attribute is used.
# host_identifiers: ["k8s.node.name", "host.id", "k8s.node.uid"]
## @param metrics_flush_interval Interval at which host metrics are flushed.
## Valid values are between 15s and 5m.
# metrics_flush_interval: 60s
```
### Example configuration for the component
```yaml
connectors:
grafana_cloud:
host_identifiers: ["mycompany.myHostAttribute", "k8s.node.name", "host.id", "k8s.node.uid"]
metrics_flush_interval: 60s
```
This connector will generate a host info metric based on the first `host_identifiers` resource attribute found on spans. The rest are skipped. Valid flush intervals are between 15s and 5m.
---
### Overview (`connector/metricsaslogsconnector/README.md`)
# Metrics as Logs Connector
| Status | |
| ------------- |-----------|
| Distributions | [] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fmetricsaslogs) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fmetricsaslogs) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_metricsaslogs&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@atoulme](https://www.github.com/atoulme) |
[development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| metrics | logs | [development] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
> **Deprecation Notice:** The component type has been renamed from `metricsaslogs` to `metrics_as_logs`
> to follow the OpenTelemetry snake_case naming convention.
> The old name `metricsaslogs` still works but is deprecated and will be removed in a future release.
> Please update your configuration to use `metrics_as_logs`.
This connector converts OpenTelemetry metrics into logs, creating one log entry per metric data point. Each metric data point is transformed into a structured log record with configurable JSON body format.
## Current Limitations
⚠️ **Current implementation discards the following metric features:**
- Metric exemplars
- Advanced metadata
These features may be added in future iterations.
## Configuration
The following settings can be optionally configured:
- `include_resource_attributes` (default = `true`): Whether to include resource attributes in the generated logs
- `include_scope_info` (default = `true`): Whether to include instrumentation scope information in the generated logs
## Log Body Format
The connector always generates log bodies in the following JSON format:
```json
{"metric_name": "$NAME", "value": "$VALUE"}
```
Where:
- `$NAME` is the actual metric name
- `$VALUE` is the metric value (simple values for gauge/sum, complex JSON for histogram/summary)
## Example Usage
### Basic Configuration
```yaml
connectors:
metrics_as_logs:
service:
pipelines:
logs:
receivers: [metrics_as_logs]
processors: []
exporters: [debug]
metrics:
receivers: [otlp]
processors: []
exporters: [metrics_as_logs]
```
### Advanced Configuration
```yaml
connectors:
metrics_as_logs:
include_resource_attributes: false
include_scope_info: false
```
### Example Metric Conversions
For a gauge metric `cpu_usage` with value `85.2`:
```json
{
"body": {"metric_name": "cpu_usage", "value": "85.2"},
"attributes": {
"metric.name": "cpu_usage",
"metric.type": "Gauge",
"metric.description": "CPU usage percentage",
"metric.unit": "%"
}
}
```
For a histogram metric `request_duration`:
```json
{
"body": {
"metric_name": "request_duration",
"value": "{\"count\":100,\"sum\":1.5,\"bucket_counts\":[10,50,40],\"explicit_bounds\":[0.1,0.5,1.0]}"
},
"attributes": {
"metric.name": "request_duration",
"metric.type": "Histogram",
"metric.description": "Request duration in seconds",
"metric.unit": "s",
"metric.aggregation_temporality": "Delta"
}
}
```
## Output Structure
Each metric data point is converted to a log record with:
- **Body**: Fixed JSON format: `{"metric_name": "$NAME", "value": "$VALUE"}`
- **Timestamp**: Metric data point timestamp
- **Observed Timestamp**: Metric data point start timestamp (if available)
- **Attributes**:
- Original metric data point attributes (labels)
- `metric.name`: The metric name
- `metric.type`: The metric type (Gauge, Sum, Histogram, etc.)
- `metric.description`: Metric description (if available)
- `metric.unit`: Metric unit (if available)
- Additional type-specific attributes:
- For Sum metrics: `metric.is_monotonic`, `metric.aggregation_temporality`
- For Histogram/ExponentialHistogram: `metric.aggregation_temporality`
- Resource attributes (if `include_resource_attributes` is true)
- Instrumentation scope information (if `include_scope_info` is true)
## Supported Metric Types
All OpenTelemetry metric types are supported:
- **Gauge**: Point-in-time measurements
- **Sum**: Cumulative or delta measurements
- **Histogram**: Distribution of measurements with buckets
- **Exponential Histogram**: Distribution with exponentially sized buckets
- **Summary**: Distribution with quantile values
## Value Encoding
The `value` field in the JSON body contains:
- For simple metrics (Gauge, Sum): numeric value as string
- For complex metrics (Histogram, etc.): JSON-encoded object as string
All JSON special characters in metric names and values are properly escaped.
---
### Overview (`connector/otlpjsonconnector/README.md`)
# OTLP JSON Connector
| Status | |
| ------------- |-----------|
| Distributions | [contrib], [k8s] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fotlpjson) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fotlpjson) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_otlpjson&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@ChrsMark](https://www.github.com/ChrsMark) |
| Emeritus | [@djaglowski](https://www.github.com/djaglowski) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| logs | metrics | [alpha] |
| logs | traces | [alpha] |
| logs | logs | [alpha] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
> **Deprecation Notice:** The component type has been renamed from `otlpjson` to `otlp_json`
> to follow the OpenTelemetry snake_case naming convention.
> The old name `otlpjson` still works but is deprecated and will be removed in a future release.
> Please update your configuration to use `otlp_json`.
Allows to extract otlpjson data from incoming Logs and specifically the `Body` field.
The data is written in
[Protobuf JSON
encoding](https://developers.google.com/protocol-buffers/docs/proto3#json)
using [OpenTelemetry
protocol](https://github.com/open-telemetry/opentelemetry-proto).
## Configuration
#### Configuration Example:
```yaml
receivers:
file_log:
include:
- /var/log/foo.log
exporters:
debug:
connectors:
# Deprecated (still works):
# otlpjson:
# New name:
otlp_json:
service:
pipelines:
logs/raw:
receivers: [file_log]
exporters: [otlpjson]
metrics/otlp:
receivers: [otlpjson]
exporters: [debug]
logs/otlp:
receivers: [otlpjson]
exporters: [debug]
traces/otlp:
receivers: [otlpjson]
exporters: [debug]
```
[Connectors README]:https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md
[Exporter Pipeline Type]:https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]:https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
---
### Overview (`connector/roundrobinconnector/README.md`)
# Round-Robin Connector
| Status | |
| ------------- |-----------|
| Distributions | [contrib], [k8s] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Froundrobin) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Froundrobin) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_roundrobin&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@bogdandrutu](https://www.github.com/bogdandrutu) |
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | traces | [beta] |
| metrics | metrics | [beta] |
| logs | logs | [beta] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
> **Deprecation Notice:** The component type has been renamed from `roundrobin` to `round_robin`
> to follow the OpenTelemetry snake_case naming convention.
> The old name `roundrobin` still works but is deprecated and will be removed in a future release.
> Please update your configuration to use `round_robin`.
The `round_robin` connector can fork pipelines of the same type and equally split the load between them.
## Configuration
If you are not already familiar with connectors, you may find it helpful to first visit the [Connectors README].
The `roundrobin` connector does not have any configuration settings.
```yaml
receivers:
otlp:
exporters:
prometheus_remote_write/1:
prometheus_remote_write/2:
connectors:
round_robin:
```
Preprocess data, then export using multiple exporter instances to scale the throughput if the exporter
does not support scale well (e.g. prometheus_remote_write).
```yaml
receivers:
otlp:
processors:
resource_detection:
exporters:
prometheus_remote_write/1:
prometheus_remote_write/2:
connectors:
round_robin:
service:
pipelines:
metrics:
receivers: [otlp]
processors: [resource_detection]
exporters: [round_robin]
metrics/1:
receivers: [round_robin]
exporters: [prometheus_remote_write/1]
metrics/2:
receivers: [round_robin]
exporters: [prometheus_remote_write/2]
```
[Connectors README]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md
---
### Overview (`connector/routingconnector/README.md`)
# Routing Connector
| Status | |
| ------------- |-----------|
| Distributions | [contrib], [k8s] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Frouting) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Frouting) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_routing&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@TylerHelmuth](https://www.github.com/TylerHelmuth), [@evan-bradley](https://www.github.com/evan-bradley), [@edmocosta](https://www.github.com/edmocosta), [@bogdandrutu](https://www.github.com/bogdandrutu), [@mwear](https://www.github.com/mwear) \| Seeking more code owners! |
| Emeritus | [@jpkrohling](https://www.github.com/jpkrohling) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | traces | [alpha] |
| metrics | metrics | [alpha] |
| logs | logs | [alpha] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
Routes logs, metrics or traces based on resource attributes to specific pipelines using [OpenTelemetry Transformation Language (OTTL)](../../pkg/ottl/README.md) statements as routing conditions.
## Configuration
If you are not already familiar with connectors, you may find it helpful to first visit the [Connectors README].
The following settings are available:
- `table (required)`: the routing table for this connector.
- `table.condition`: the routing condition provided as the [OTTL] condition. Required if `table.statement` is not provided. Use context-qualified paths (e.g., `resource.attributes["key"]`, `span.attributes["key"]`) to automatically infer the context (see [Context Inference](#context-inference)).
- `table.statement`: the routing condition provided as the [OTTL] statement. Required if `table.condition` is not provided. Generally `condition` is preferred since it is more terse. May not be used with the deprecated `request` context.
- `table.context (optional)`: the [OTTL Context](#supported-contexts) in which the condition/statement will be evaluated. **Deprecated:** `request` — use `otelcol.client.metadata` or `otelcol.grpc.metadata` paths instead (see [Limitations](#limitations)). In most cases this field should be omitted; the context is inferred automatically from context-qualified paths. If specified, it takes precedence over inference.
- `table.action (optional, default: move)`: determines what happens to the data when the routing condition is met. Valid values are `move` and `copy`.
- `move`: Matched data is moved to the target pipeline(s) and removed from subsequent route evaluation. This is the default behavior.
- `copy`: Matched data is copied to the target pipeline(s) but remains available for evaluation by subsequent routes. This allows the same data to be routed to multiple pipelines.
- `table.pipelines (required)`: the list of pipelines to use when the routing condition is met.
- `default_pipelines (optional)`: contains the list of pipelines to use when a record does not meet any of specified conditions.
- `error_mode (optional)`: determines how errors returned from OTTL statements are handled. Valid values are `propagate`, `ignore` and `silent`. If `ignore` or `silent` is used and a statement's condition has an error then the payload will be routed to the default pipelines. When `silent` is used the error is not logged. If not supplied, `ignore` is used.
### Context Inference
The routing connector supports OTTL context inference, allowing you to write clearer and more maintainable routing conditions using context-qualified paths. This is the recommended approach for specifying routing conditions.
```yaml
- condition: resource.attributes["env"] == "prod"
pipelines: [logs/prod]
- condition: span.attributes["http.method"] == "GET"
pipelines: [traces/http]
- condition: log.severity_text == "ERROR"
pipelines: [logs/errors]
```
This approach makes it immediately clear which attributes you're accessing without needing a separate `context` field.
### Supported contexts
| Context | Path prefix | Example |
|-------------|--------------|----------------------------------------------------------------------------------|
| [Resource] | `resource.` | `resource.attributes["service.name"]` |
| [Span] | `span.` | `span.attributes["http.method"]` |
| [Log] | `log.` | `log.body`, `log.attributes["level"]` |
| [Metric] | `metric.` | `metric.name` |
| [Datapoint] | `datapoint.` | `datapoint.attributes["host"]` |
| [OtelCol] | `otelcol.` | `otelcol.client.metadata["X-Tenant"][0]`, `otelcol.grpc.metadata["x-tenant"][0]` |
[resource]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlresource/README.md
[span]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlspan/README.md
[metric]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlmetric/README.md
[datapoint]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottldatapoint/README.md
[log]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottllog/README.md
[otelcol]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/contexts/ottlotelcol/README.md
The `otelcol.client.metadata` and `otelcol.grpc.metadata` paths provide access to incoming HTTP and gRPC request metadata respectively, and are valid in all signal contexts.
### Limitations
- **Deprecated:** The `request` context is deprecated. Use `otelcol.client.metadata["key"]` (HTTP/client metadata) or `otelcol.grpc.metadata["key"]` (gRPC metadata) paths instead. These are supported in all signal contexts. A warning is logged when the `request` context is used. The `request` context only supports the `condition` field with a very limited grammar: `request["key"] == "value"` or `request["key"] != "value"`.
- When using context inference without an explicit `context` field, the inferred context must be compatible with the pipeline signal type (e.g., `span` context can only be used in traces pipelines).
### Supported [OTTL] functions
- [Standard OTTL Converter Functions](../../pkg/ottl/ottlfuncs/README.md#converters)
- [delete_key](../../pkg/ottl/ottlfuncs/README.md#delete_key)
- [delete_matching_keys](../../pkg/ottl/ottlfuncs/README.md#delete_matching_keys)
## Additional Settings
The full list of settings exposed for this connector are documented in [config.go](./config.go) with detailed sample configuration files:
- [logs](./testdata/config/logs.yaml)
- [metrics](./testdata/config/metrics.yaml)
- [traces](./testdata/config/traces.yaml)
## Examples
> [!NOTE]
> The examples below use context-qualified paths, which is the recommended configuration style. The explicit `context` field is still supported for backward compatibility but is no longer the primary documentation style. See [Context Inference](#context-inference) for details.
### Route logs based on tenant
```yaml
receivers:
otlp:
exporters:
file/other:
path: ./other.log
file/acme:
path: ./acme.log
file/ecorp:
path: ./ecorp.log
connectors:
routing:
default_pipelines: [logs/other]
table:
- context: resource
condition: otelcol.client.metadata["X-Tenant"][0] == "acme"
pipelines: [logs/acme]
- context: resource
condition: otelcol.client.metadata["X-Tenant"][0] == "ecorp"
pipelines: [logs/ecorp]
service:
pipelines:
logs/in:
receivers: [otlp]
exporters: [routing]
logs/acme:
receivers: [routing]
exporters: [file/acme]
logs/ecorp:
receivers: [routing]
exporters: [file/ecorp]
logs/other:
receivers: [routing]
exporters: [file/other]
```
### Route logs based on region
```yaml
receivers:
otlp:
exporters:
file/other:
path: ./other.log
file/east:
path: ./east.log
file/west:
path: ./west.log
connectors:
routing:
default_pipelines: [logs/other]
table:
- condition: log.attributes["region"] == "east"
pipelines: [logs/east]
- condition: log.attributes["region"] == "west"
pipelines: [logs/west]
service:
pipelines:
logs/in:
receivers: [otlp]
exporters: [routing]
logs/east:
receivers: [routing]
exporters: [file/east]
logs/west:
receivers: [routing]
exporters: [file/west]
logs/other:
receivers: [routing]
exporters: [file/other]
```
### Route low-severity logs to cheap storage, remainder by service name
```yaml
receivers:
otlp:
exporters:
file/cheap:
path: ./cheap.log
file/service1:
path: ./service1-important.log
file/service2:
path: ./service2-important.log
connectors:
routing:
table:
- condition: log.severity_number < SEVERITY_NUMBER_ERROR
pipelines: [logs/cheap]
- condition: resource.attributes["service.name"] == "service1"
pipelines: [logs/service1]
- condition: resource.attributes["service.name"] == "service2"
pipelines: [logs/service2]
service:
pipelines:
logs/in:
receivers: [otlp]
exporters: [routing]
logs/cheap:
receivers: [routing]
exporters: [file/cheap]
logs/service1:
receivers: [routing]
exporters: [file/service1]
logs/service2:
receivers: [routing]
exporters: [file/service2]
```
### Route low-severity logs to cheap storage, remainder by tenant
```yaml
receivers:
otlp:
exporters:
file/cheap:
path: ./cheap.log
file/acme:
path: ./acme.log
file/ecorp:
path: ./ecorp.log
connectors:
routing:
table:
- condition: log.severity_number < SEVERITY_NUMBER_ERROR
pipelines: [logs/cheap]
- context: resource
condition: otelcol.client.metadata["X-Tenant"][0] == "acme"
pipelines: [logs/acme]
- context: resource
condition: otelcol.client.metadata["X-Tenant"][0] == "ecorp"
pipelines: [logs/ecorp]
service:
pipelines:
logs/in:
receivers: [otlp]
exporters: [routing]
logs/cheap:
receivers: [routing]
exporters: [file/cheap]
logs/acme:
receivers: [routing]
exporters: [file/acme]
logs/ecorp:
receivers: [routing]
exporters: [file/ecorp]
```
### Route all logs to an archive, while also routing errors using `action: copy`
Conditions with no OTTL paths (such as the literal `"true"`) cannot be inferred, so an explicit `context` field is required.
```yaml
receivers:
otlp:
exporters:
file/archive:
path: ./archive.log
file/errors:
path: ./errors.log
file/other:
path: ./other.log
connectors:
routing:
default_pipelines: [logs/other]
table:
- context: resource
condition: "true"
action: copy
pipelines: [logs/archive]
- condition: log.severity_number >= SEVERITY_NUMBER_ERROR
pipelines: [logs/errors]
service:
pipelines:
logs/in:
receivers: [otlp]
exporters: [routing]
logs/archive:
receivers: [routing]
exporters: [file/archive]
logs/errors:
receivers: [routing]
exporters: [file/errors]
logs/other:
receivers: [routing]
exporters: [file/other]
```
In this example:
- All logs are first copied to the archive pipeline (using `action: copy`), which means the original data remains available for subsequent route evaluation.
- Error logs are then moved to the errors pipeline (using the default `action: move`).
- Any remaining logs (non-errors) go to the default pipeline.
### Route traces to multiple pipelines using `action: copy`
```yaml
receivers:
otlp:
exporters:
file/prod:
path: ./prod.json
file/high-latency:
path: ./high-latency.json
file/other:
path: ./other.json
connectors:
routing:
default_pipelines: [traces/other]
table:
- condition: resource.attributes["env"] == "prod"
action: copy
pipelines: [traces/prod]
- condition: span.attributes["http.duration_ms"] > 1000
pipelines: [traces/high-latency]
service:
pipelines:
traces/in:
receivers: [otlp]
exporters: [routing]
traces/prod:
receivers: [routing]
exporters: [file/prod]
traces/high-latency:
receivers: [routing]
exporters: [file/high-latency]
traces/other:
receivers: [routing]
exporters: [file/other]
```
In this example:
- Production traces are copied to the prod pipeline. Since `action: copy` is used, the traces remain available for subsequent evaluation.
- High-latency spans (>1000ms) are then moved to the high-latency pipeline. A production trace with high latency will appear in both the prod and high-latency pipelines.
- Remaining traces go to the default pipeline.
## `match_once`
The `match_once` field was deprecated as of `v0.116.0` and removed in `v0.120.0`.
The following examples demonstrate some strategies for migrating a configuration from `match_once`.
### Example without `default_pipelines`
If not using `default_pipelines`, you may be able to split the router into multiple parallel routers.
In the following example, the `"env"` and `"region"` are not directly related.
```yaml
routing:
match_once: false
table:
- condition: attributes["env"] == "prod"
pipelines: [ logs/prod ]
- condition: attributes["env"] == "dev"
pipelines: [ logs/dev ]
- condition: attributes["region"] == "east"
pipelines: [ logs/east ]
- condition: attributes["region"] == "west"
pipelines: [ logs/west ]
service:
pipelines:
logs/in::exporters: [routing]
logs/prod::receivers: [routing]
logs/dev::receivers: [routing]
logs/east::receivers: [routing]
logs/west::receivers: [routing]
```
Therefore, the same behavior can be achieved using separate routers. Listing both routers in the pipeline configuration will
result in each receiving an independent handle to the data. The same data can then match routes in both routers.
```yaml
routing/env:
table:
- condition: resource.attributes["env"] == "prod"
pipelines: [ logs/prod ]
- condition: resource.attributes["env"] == "dev"
pipelines: [ logs/dev ]
routing/region:
table:
- condition: resource.attributes["region"] == "east"
pipelines: [ logs/east ]
- condition: resource.attributes["region"] == "west"
pipelines: [ logs/west ]
service:
pipelines:
logs/in::exporters: [routing/env, routing/region]
logs/prod::receivers: [routing/env]
logs/dev::receivers: [routing/env]
logs/east::receivers: [routing/region]
logs/west::receivers: [routing/region]
```
### Example with `default_pipelines`
The following example demonstrates strategies for migrating from `match_once: true` while using `default_pipelines`.
```yaml
routing:
match_once: true
default_pipelines: [ logs/default ]
table:
- condition: resource.attributes["env"] == "prod"
pipelines: [ logs/prod ]
- condition: resource.attributes["env"] == "dev"
pipelines: [ logs/dev ]
- condition: resource.attributes["region"] == "east"
pipelines: [ logs/east ]
- condition: resource.attributes["region"] == "west"
pipelines: [ logs/west ]
service:
pipelines:
logs/in::exporters: [routing]
logs/default::receivers: [routing]
logs/prod::receivers: [routing]
logs/dev::receivers: [routing]
logs/east::receivers: [routing]
logs/west::receivers: [routing]
```
If the number of routes are limited, you may be able to articulate a route for each combination of conditions. This avoids the need to change any pipelines.
```yaml
routing:
default_pipelines: [ logs/default ]
table:
- condition: resource.attributes["env"] == "prod" and resource.attributes["region"] == "east"
pipelines: [ logs/prod, logs/east ]
- condition: resource.attributes["env"] == "prod" and resource.attributes["region"] == "west"
pipelines: [ logs/prod, logs/west ]
- condition: resource.attributes["env"] == "dev" and resource.attributes["region"] == "east"
pipelines: [ logs/dev, logs/east ]
- condition: resource.attributes["env"] == "dev" and resource.attributes["region"] == "west"
pipelines: [ logs/dev, logs/west ]
service:
pipelines:
logs/in::exporters: [routing]
logs/default::receivers: [routing]
logs/prod::receivers: [routing]
logs/dev::receivers: [routing]
logs/east::receivers: [routing]
logs/west::receivers: [routing]
```
A more general solution is to use a layered approach. In this design, the first layer is a single router that sorts data according to whether it matches
_any route_ or _no route_. This allows the second layer to work without `default_pipelines`. The downside to this approach is that the set of conditions
in the first and second layers must be kept in sync.
```yaml
# First layer separates logs that match no routes
routing:
default_pipelines: [ logs/default ]
table: # all routes forward to second layer
- condition: resource.attributes["env"] == "prod"
pipelines: [ logs/env, logs/region ]
- condition: resource.attributes["env"] == "dev"
pipelines: [ logs/env, logs/region ]
- condition: resource.attributes["region"] == "east"
pipelines: [ logs/env, logs/region ]
- condition: resource.attributes["region"] == "west"
pipelines: [ logs/env, logs/region ]
# Second layer routes logs based on environment and region
routing/env:
table:
- condition: resource.attributes["env"] == "prod"
pipelines: [ logs/prod ]
- condition: resource.attributes["env"] == "dev"
pipelines: [ logs/dev ]
routing/region:
table:
- condition: resource.attributes["region"] == "east"
pipelines: [ logs/east ]
- condition: resource.attributes["region"] == "west"
pipelines: [ logs/west ]
service:
pipelines:
logs/in::exporters: [routing]
logs/prod::receivers: [routing/env]
logs/dev::receivers: [routing/env]
logs/east::receivers: [routing/region]
logs/west::receivers: [routing/region]
```
[Connectors README]:https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md
[OTTL]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md
[OTTL Context]: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/LANGUAGE.md#contexts
---
### Overview (`connector/servicegraphconnector/README.md`)
# Service Graph Connector
| Status | |
| ------------- |-----------|
| Distributions | [contrib], [k8s] |
| Warnings | [Statefulness](#warnings) |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fservicegraph) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fservicegraph) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_servicegraph&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@mapno](https://www.github.com/mapno), [@JaredTan95](https://www.github.com/JaredTan95) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | metrics | [alpha] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
> **Note:** The component type has been renamed from `servicegraph` to `service_graph`.
> The old name still works but is deprecated and will be removed in a future release.
> Please update your configuration to use `service_graph`.
## Overview
The service graphs connector builds a map representing the interrelationships between various services in a system.
The connector will analyse trace data and generate metrics describing the relationship between the services.
These metrics can be used by data visualization apps (e.g. Grafana) to draw a service graph.
Service graphs are useful for a number of use-cases:
* Infer the topology of a distributed system. As distributed systems grow, they become more complex. Service graphs can help you understand the structure of the system.
* Provide a high level overview of the health of your system.
Service graphs show error rates, latencies, among other relevant data.
* Provide an historic view of a system’s topology.
Distributed systems change very frequently,
and service graphs offer a way of seeing how these systems have evolved over time.
This component is based on [Grafana Tempo's service graph processor](https://github.com/grafana/tempo/tree/main/modules/generator/processor/servicegraphs).
## How it works
Service graphs work by inspecting traces and looking for spans with parent-children relationship that represent a request.
The connector uses the OpenTelemetry semantic conventions to detect a myriad of requests.
It currently supports the following requests:
* A direct request between two services where the outgoing and the incoming span must have `span.kind` client and server respectively.
* A request across a messaging system where the outgoing and the incoming span must have `span.kind` producer and consumer respectively.
* A database request; in this case the connector looks for spans containing attributes `span.kind`=client as well as db.name.
Every span that can be paired up to form a request is kept in an in-memory store,
until its corresponding pair span is received or the maximum waiting time has passed.
When either of these conditions are reached, the request is recorded and removed from the local store.
Each emitted metrics series have the client and server label corresponding with the service doing the request and the service receiving the request.
```
traces_service_graph_request_total{client="app", server="db", connection_type="database"} 20
```
TLDR: The connector will try to find spans belonging to requests as seen from the client and the server and will create a metric representing an edge in the graph.
## Metrics
The following metrics are emitted by the connector:
| Metric | Type | Labels | Description |
|---------------------------------------------|-----------|---------------------------------|---------------------------------------------------------------------------|
| traces_service_graph_request_total | Counter | client, server, connection_type | Total count of requests between two nodes |
| traces_service_graph_request_failed_total | Counter | client, server, connection_type | Total count of failed requests between two nodes |
| traces_service_graph_request_server | Histogram | client, server, connection_type | Number of seconds for a request between two nodes as seen from the server |
| traces_service_graph_request_client | Histogram | client, server, connection_type | Number of seconds for a request between two nodes as seen from the client |
| traces_service_graph_unpaired_spans_total | Counter | client, server, connection_type | Total count of unpaired spans |
| traces_service_graph_dropped_spans_total | Counter | client, server, connection_type | Total count of dropped spans |
Duration is measured both from the client and the server sides.
Possible values for `connection_type`: unset, `messaging_system`, or `database`.
Additional labels can be included using the `dimensions` configuration option. Those labels will have a prefix to mark where they originate (client or server span kinds).
The `client_` prefix relates to the dimensions coming from spans with `SPAN_KIND_CLIENT`, and the `server_` prefix relates to the
dimensions coming from spans with `SPAN_KIND_SERVER`.
Since the service graph connector has to process both sides of an edge,
it needs to process all spans of a trace to function properly.
If spans of a trace are spread out over multiple instances, spans are not paired up reliably.
A possible solution to this problem is using the [load balancing exporter](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/loadbalancingexporter)
in a layer on front of collector instances running this connector.
## Visualization
Service graph metrics are natively supported by Grafana since v9.0.4.
To run it, configure a Tempo data source's 'Service Graphs' by linking to the Prometheus backend where metrics are being sent:
```yaml
apiVersion: 1
datasources:
# Prometheus backend where metrics are sent
- name: Prometheus
type: prometheus
uid: prometheus
url:
jsonData:
httpMethod: GET
version: 1
- name: Tempo
type: tempo
uid: tempo
url:
jsonData:
httpMethod: GET
serviceMap:
datasourceUid: 'prometheus'
version: 1
```
## Configuration
The following settings are required:
- `latency_histogram_buckets`: the list of durations defining the latency histogram buckets. Make sure use either `latency_histogram_buckets` or `exponential_histogram_max_size`.
- Default: `[2ms, 4ms, 6ms, 8ms, 10ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1s, 1400ms, 2s, 5s, 10s, 15s]`
- `exponential_histogram_max_size`: (no default) the maximum number of buckets per positive or negative number range.
- `dimensions`: the list of dimensions to add together with the default dimensions defined above.
The following settings can be optionally configured:
- `store`: defines the config for the in-memory store used to find requests between services by pairing spans.
- `ttl`: TTL is the time to live for items in the store.
- Default: `2s`
- `max_items`: MaxItems is the maximum number of items to keep in the store.
- Default: `1000`
- `cache_loop`: the interval at which to clean the cache.
- Default: `1m`
- `store_expiration_loop`: the time to expire old entries from the store periodically.
- Default: `2s`
- `virtual_node_peer_attributes`: the list of attributes, ordered by priority, whose presence in a client span will result in the creation of a virtual server node. An empty list disables virtual node creation.
- Default: `[peer.service.name, peer.service, db.name, db.namespace, db.system, db.system.name]`
- `virtual_node_extra_label`: adds an extra label `virtual_node` with an optional value of `client` or `server`, indicating which node is the uninstrumented one.
- Default: `false`
- `metrics_flush_interval`: the interval at which metrics are flushed to the exporter.
- Default: `60s`
- `metrics_timestamp_offset`: the offset to subtract from metric timestamps. If set to a positive duration, metric timestamps will be set to (current time - offset), effectively shifting metrics to appear as if they were generated in the past.
- Default: `0`
- `database_name_attributes`: the list of attribute names used to identify the database name from span attributes. The attributes are tried in order, selecting the first match.
- Default: `[db.name, db.namespace]`
## Example configurations
### Sample with custom buckets and dimensions
```yaml
receivers:
otlp:
protocols:
grpc:
connectors:
service_graph:
latency_histogram_buckets: [100ms, 250ms, 1s, 5s, 10s]
dimensions:
- dimension-1
- dimension-2
store:
ttl: 1s
max_items: 10
exporters:
prometheus/servicegraph:
endpoint: localhost:9090
namespace: servicegraph
service:
pipelines:
traces:
receivers: [otlp]
exporters: [service_graph]
metrics/servicegraph:
receivers: [service_graph]
exporters: [prometheus/servicegraph]
```
### Sample with options for uninstrumented services identification
```yaml
receivers:
otlp:
protocols:
grpc:
connectors:
service_graph:
dimensions:
- db.system
- messaging.system
virtual_node_peer_attributes:
- db.name
- db.system
- messaging.system
- peer.service.name
virtual_node_extra_label: true
exporters:
prometheus/servicegraph:
endpoint: localhost:9090
namespace: servicegraph
service:
pipelines:
traces:
receivers: [otlp]
exporters: [service_graph]
metrics/servicegraph:
receivers: [service_graph]
exporters: [prometheus/servicegraph]
```
## Warnings
- [Statefulness](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/standard-warnings.md#statefulness): The `service_graph` connector keeps every span that can be paired up to form a request in an in-memory store, periodically cleaned by `cache_loop`. All spans for a given request must be sent to the same collector instance, or the connector will be unable to pair them and the resulting service graph metrics will be incomplete.
---
### Overview (`connector/signaltometricsconnector/README.md`)
# Signal to Metrics Connector
Signal to metrics connector produces metrics from all signal types (traces, logs, or metrics).
| Status | |
| ------------- |-----------|
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fsignaltometrics) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fsignaltometrics) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_signaltometrics&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@ChrsMark](https://www.github.com/ChrsMark), [@lahsivjar](https://www.github.com/lahsivjar) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | metrics | [alpha] |
| logs | metrics | [alpha] |
| metrics | metrics | [alpha] |
| profiles | metrics | [alpha] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
## Configuration
The component can produce metrics from spans, datapoints (for metrics), and logs.
At least one of the metrics for one signal type MUST be specified correctly for
the component to work.
All signal types can be configured to produce metrics with the same configuration
structure. For example, the below configuration will produce delta temporality counters
for counting number of events for each of the configured signals:
```yaml
signal_to_metrics:
spans:
- name: span.count
description: Count of spans
sum:
value: Int(AdjustedCount()) # Count of total spans represented by each span
monotonic: true
datapoints:
- name: datapoint.count
description: Count of datapoints
sum:
value: "1" # increment by 1 for each datapoint
monotonic: true
logs:
- name: logrecord.count
description: Count of log records
sum:
value: "1" # increment by 1 for each log record
monotonic: true
profiles:
- name: profile.count
description: Count of profiles
sum:
value: "1" # increment by 1 for each profile
monotonic: true
```
### Path context prefixes in OTTL expressions
OTTL strings in `conditions`, `value`, `count`, and `keys_expression` may use
context path prefixes to disambiguate the target of a path.
Unprefixed paths continue to work. It is recommended to use the new syntax to avoid
future breaking changes.
**Example (recommended):**
```yaml
signal_to_metrics:
spans:
- name: http.trace.span.duration
attributes:
- key: http.response.status_code
conditions:
- resource.attributes["service.name"] != nil
sum:
value: Int(Seconds(span.end_time - span.start_time))
monotonic: true
```
### Error Handling
The `error_mode` configuration option determines how the connector handles errors that occur while processing OTTL expressions:
- `error_mode` (optional): Determines how errors returned from OTTL expressions are handled. Valid values are `propagate`, `ignore`, and `silent`.
- `ignore` (default): Errors are logged and the specific record that caused the error is skipped, but processing continues for the rest of the batch.
- `propagate`: Errors cause the entire batch to fail and be returned up the pipeline. This will result in the payload being dropped from the collector.
- `silent`: Errors are not logged and the specific record that caused the error is skipped, but processing continues for the rest of the batch.
**Example with error handling:**
```yaml
signaltometrics:
error_mode: ignore # Log errors but continue processing other records
spans:
- name: span.count
description: Count of spans
sum:
value: Int(AdjustedCount())
```
### Metrics types
`signal_to_metrics` produces a variety of metric types by utilizing [OTTL](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/README.md)
to extract the relevant data for a metric type from the incoming data. The
component can produce the following metric types for each signal type:
- [Sum](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#sums)
- [Gauge](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#gauge)
- [Histogram](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#histogram)
- [Exponential Histogram](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram)
The component does NOT perform any stateful or time based aggregations. The metric
types are aggregated for the payload sent in each `Consume*` call. The final metric
is then sent forward in the pipeline.
#### Sum
Sum metrics have the following configurations:
```yaml
sum:
value:
monotonic:
```
- [**Required**] `value` represents an OTTL expression to extract a value from the
incoming data. Only OTTL expressions that return a value are accepted. The
returned value determines the value type of the `sum` metric (`int` or `double`).
[OTTL converters](https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/ottlfuncs#readme-converters)
can be used to transform the data.
- [**Optional**] `monotonic` whether the generated metric is monotonic. It defaults to `false`.
#### Gauge
Gauge metrics aggregate the last value of a signal and have the following configuration:
```yaml
gauge:
value:
```
- [**Required**] `value`represents an OTTL expression to extract a numeric value from
the signal. Only OTTL expressions that return a value are accepted. The returned
value determines the value type of the `gauge` metric (`int` or `double`).
- For logs: Use e.g. `ExtractGrokPatterns` with a single key selector (see below).
- For other signals: Use a field such as `value_int`, `value_double`, or a valid OTTL expression.
**Examples:**
_Logs (with Grok pattern):_
```yaml
signal_to_metrics:
logs:
- name: logs.memory_mb
description: Extract memory_mb from log records
gauge:
value: ExtractGrokPatterns(body, "Memory usage %{NUMBER:memory_mb:int}MB")["memory_mb"]
```
_Traces:_
```yaml
signal_to_metrics:
spans:
- name: span.duration.gauge
description: Span duration as gauge
gauge:
value: Int(Seconds(end_time - start_time))
```
#### Histogram
Histogram metrics have the following configurations:
```yaml
histogram:
buckets: []float64
count:
value:
```
- [**Optional**] `buckets` represents the buckets to be used for the histogram.
If no buckets are configured then it defaults to:
```go
[]float64{2, 4, 6, 8, 10, 50, 100, 200, 400, 800, 1000, 1400, 2000, 5000, 10_000, 15_000}
```
- [**Optional**] `count` represents an OTTL expression to extract the count to be
recorded in the histogram from the incoming data. If no expression is provided
then it defaults to the count of the signal. [OTTL converters](https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/ottlfuncs#readme-converters)
can be used to transform the data. For spans, a special converter [adjusted count](#custom-ottl-functions),
is provided to help calculate the span's [adjusted count](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling-experimental/#adjusted-count).
- [**Required**] `value` represents an OTTL expression to extract the value to be
recorded in the histogram from the incoming data. [OTTL converters](https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/ottlfuncs#readme-converters)
can be used to transform the data.
#### Exponential Histogram
Exponential histogram metrics have the following configurations:
```yaml
exponential_histogram:
max_size:
count:
value:
```
- [**Optional**] `max_size` represents the maximum number of buckets per positive
or negative number range. Defaults to `160`.
- [**Optional**] `count` represents an OTTL expression to extract the count to be
recorded in the exponential histogram from the incoming data. If no expression
is provided then it defaults to the count of the signal. [OTTL converters](https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/ottlfuncs#readme-converters)
can be used to transform the data. For spans, a special converter [adjusted count](#custom-ottl-functions),
is provided to help calculate the span's [adjusted count](https://opentelemetry.io/docs/specs/otel/trace/tracestate-probability-sampling-experimental/#adjusted-count).
- [**Required**] `value` represents an OTTL expression to extract the value to be
recorded in the exponential histogram from the incoming data. [OTTL converters](https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/ottlfuncs#readme-converters)
can be used to transform the data.
### Attributes
The component can produce metrics categorized by the attributes (span attributes
for traces, datapoint attributes for datapoints, or log record attributes for logs)
from the incoming data by configuring `attributes` for the configured metrics.
If no `attributes` are configured then the metrics are produced without any attributes.
```yaml
attributes:
- key: datapoint.foo
- key: datapoint.bar
default_value: bar
- key: datapoint.baz
optional: true
- keys_expression: otelcol.client.metadata["x-dynamic-attributes"]
```
Each attribute entry must have exactly one of `key` or `keys_expression` set.
If attributes are specified then a separate metric will be generated for each unique
set of attribute values. There are four behaviors that can be configured for an
attribute:
- Without any extra parameters: `datapoint.foo` in the above yaml is an example
of such configuration. In this configuration, only the signals which have the said
attribute are processed with the attribute's value as one of the attributes for
the output metric. If the attribute is missing then the signal is not processed.
- With `default_value`: `datapoint.bar` in the above yaml is an example of such
configuration. In this configuration all the signals are processed irrespective
of the attribute being present or not in the input signal. The output metric
is categorized as per the incoming value of the attribute and an extra bucket
exists with the attribute set to the configured default value for all the signals
that were missing the configured attribute.
- With `optional` set to `true`: `datapoint.baz` in the above yaml is an example
of such configuration. If the attribute is configured with `optional` and present
in the incoming signal then it will be added directly to the output metric. If
it is absent then a new metric with missing attributes will be created. In addition,
the `optional` attribute will not impact the decision i.e. even if the `optional`
attributes are not present in the incoming signal, the signal will be processed
and will produce a metric given all other non-optional attributes are present
or have a default value defined.
- With `keys_expression`: The OTTL value expression is evaluated at runtime and
must return a list of attribute keys (`pcommon.Slice` or `[]string`). Each resolved
key is looked up in the signal's attributes and included in the output metric.
If the expression returns `nil` (e.g. missing client metadata), it is treated as
an empty list. Expression evaluation errors are governed by the `error_mode`
configuration. The `optional` and `default_value` options can be combined with
`keys_expression` and apply to each resolved key.
Note that resource attributes are handled differently, check the resource attributes
section for more details on this. Think of `attributes` as conditional filters for
choosing which attributes should be included in the output metric whereas
`include_resource_attributes` is an include list for customizing resource attributes
of the output metric.
### Conditions
Conditions are an optional list of OTTL conditions that are evaluated on the incoming
data and are ORed together. For example:
```yaml
signal_to_metrics:
datapoints:
- name: datapoint.bar.sum
description: Count total number of datapoints as per datapoint.bar attribute
conditions:
- resource.attributes["foo"] != nil
- resource.attributes["bar"] != nil
sum:
value: "1"
```
The above configuration will produce sum metrics from datapoints with either `foo`
OR `bar` resource attribute defined.
Conditions can also be ANDed together, for example:
```yaml
signal_to_metrics:
datapoints:
- name: gauge.to.exphistogram
conditions:
- metric.type == 1 AND resource.attributes["resource.foo"] != nil
exponential_histogram:
count: "1" # 1 count for each datapoint
value: Double(value_int) + value_double # handle both int and double
```
The above configuration produces exponential histogram from gauge metrics with resource
attributes `resource.foo` set.
### Customizing resource attributes
The component allows customizing the resource attributes for the produced metrics
by specifying a list of attributes that should be included in the final metrics.
If no attributes are specified for `include_resource_attributes` then no filtering
is performed i.e. all resource attributes of the incoming data is considered.
```yaml
include_resource_attributes:
- key: resource.foo # Include resource.foo attribute if present
- key: resource.bar # Always include resource.bar attribute, default to bar
default_value: bar
- key: resource.baz # Optional resource.baz attribute is added if present
optional: true
- keys_expression: otelcol.client.metadata["x-dynamic-resource-attributes"]
```
Each entry must have exactly one of `key` or `keys_expression` set.
With the above configuration the produced metrics would have the following
resource attributes:
- `resource.foo` will be present for the produced metrics if the incoming data also
has the attribute defined. If the attribute is missing in the incoming data the
output metric will be produced without the said attribute.
- `resource.bar` will always be present because of the `default_value`. If the incoming
data does not have a resource attribute with name `resource.bar` then the configured
`default_value` of `bar` will be used.
- `resource.baz` will behave exactly same as `resource.foo`. Since resource attributes
are basically an include list, the `optional` option is a no-op i.e. the resource
attributes with `optional` set to `true` behaves identical to an attribute configured
without `default_value` or `optional`.
- The `keys_expression` entry evaluates the OTTL value expression at runtime to
resolve a list of attribute keys. The expression must return a list of strings
(`pcommon.Slice` or `[]string`). Each resolved key is looked up in the resource
attributes and included in the output metric. If the expression returns `nil`
(e.g. missing client metadata), it is treated as an empty list. Expression
evaluation errors are governed by the `error_mode` configuration. The `optional`
and `default_value` options can be combined with `keys_expression` and apply to
each resolved key. OTTL expressions for `include_resource_attributes` should
only reference resource-level paths (e.g. `resource.attributes`) or context-level
paths (e.g. `otelcol.client.metadata`), not signal-specific paths (e.g.
`attributes`, `span.*`, `log.*`).
### Single writer
Metrics data streams MUST obey [single-writer](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#single-writer)
principle. However, since `signal_to_metrics` component produces metrics from all signal
types and also allows customizing the resource attributes, there is a possibility
of violating the single-writer principle. To keep the single-writer principle intact,
the component adds collector instance information as resource attributes. The following
resource attribute is added to each produced metric:
```yaml
signal_to_metrics.service.instance.id:
```
### Custom OTTL functions
The component implements the following custom OTTL functions:
1. `AdjustedCount`: a converter capable of calculating [adjusted count for a span](https://github.com/open-telemetry/oteps/blob/main/text/trace/0235-sampling-threshold-in-trace-state.md).
---
### Overview (`connector/slowsqlconnector/README.md`)
# Slow SQL Connector
| Status | |
| ------------- |-----------|
| Distributions | [] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fslowsql) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fslowsql) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_slowsql&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@JaredTan95](https://www.github.com/JaredTan95), [@Frapschen](https://www.github.com/Frapschen) |
| Emeritus | [@atoulme](https://www.github.com/atoulme) |
[development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | logs | [development] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
> **Deprecation Notice:** The component type has been renamed from `slowsql` to `slow_sql`
> to follow the OpenTelemetry snake_case naming convention.
> The old name `slowsql` still works but is deprecated and will be removed in a future release.
> Please update your configuration to use `slow_sql`.
## Overview
Generate logs from recorded [slow database statement](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/exceptions/exceptions-spans.md/) associated with spans.
Each **log** will have _at least_ the following dimensions:
- Service name
- Span kind
- Span name
- Status code
- Trace ID
- Span ID
- Database System
- Database Statement
- Database Statement Duration
Each log will additionally have the following attributes:
- Span attributes. If you want to filter out some attributes (like only copying HTTP attributes starting with `http.`) use the [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor/).
## Configurations
If you are not already familiar with connectors, you may find it helpful to first visit the [Connectors README].
The following settings can be optionally configured:
- `dimensions`: the list of dimensions to add to *logs* with the default dimensions defined above.
Each additional dimension is defined with a `name` which is looked up in the span's collection of attributes or
resource attributes (AKA process tags) such as `ip`, `host.name` or `region`.
- `db_system:` the list value of span attribute `db.system`, Filter specific db systems, define those database's statements need to be collected. ref: https://opentelemetry.io/docs/specs/semconv/attributes-registry/db/
- Default: `[h2, mongodb, mssql, mysql, oracle, postgresql, mariadb]`
- `threshold`: define a threshold and collect when the `db.statement`, namely span duration, larger than this value.
- Default: `500ms`
## Examples
The following is a simple example usage of the `slow sql` connector.
```yaml
receivers:
nop:
exporters:
nop:
connectors:
slow_sql:
threshold: 600ms
dimensions:
- name: k8s.namespace.name
- name: k8s.pod.name
service:
pipelines:
traces:
receivers: [nop]
exporters: [slow_sql]
logs:
receivers: [slow_sql]
exporters: [nop]
```
The following is a more complex example usage of the `slow_sql` connector using Elasticsearch as exporters.
```yaml
receivers:
otlp:
protocols:
grpc:
http:
exporters:
elasticsearch/slow_sql:
tls:
insecure: true
mapping:
mode: raw
endpoints:
- http://localhost:9200
user: elastic
password: elastic
connectors:
slow_sql:
threshold: 600ms
dimensions:
- name: k8s.namespace.name
- name: k8s.pod.name
service:
pipelines:
traces:
receivers: [otlp]
exporters: [slow_sql]
logs:
receivers: [slow_sql]
exporters: [elasticsearch/slow_sql]
```
The full list of settings exposed for this connector is documented in [slowsqlconnector/config.go](../../connector/slowsqlconnector/config.go).
### More Examples
For more example configuration covering various other use cases, please visit the [testdata directory](../../connector/slowsqlconnector/testdata).
[Connectors README]:https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md
---
### Overview (`connector/spanmetricsconnector/README.md`)
# Span Metrics Connector
| Status | |
| ------------- |-----------|
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fspanmetrics) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fspanmetrics) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_spanmetrics&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@portertech](https://www.github.com/portertech), [@Frapschen](https://www.github.com/Frapschen), [@iblancasa](https://www.github.com/iblancasa) \| Seeking more code owners! |
| Emeritus | [@albertteoh](https://www.github.com/albertteoh) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | metrics | [alpha] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
> **Deprecation Notice:** The component type has been renamed from `spanmetrics` to `span_metrics`
> to follow the OpenTelemetry snake_case naming convention.
> The old name `spanmetrics` still works but is deprecated and will be removed in a future release.
> Please update your configuration to use `span_metrics`.
⚠️ Breaking Change Warning:
The default duration metrics unit will change from `ms` to `s` to adhere to the OpenTelemetry semantic conventions and a feature gate `connector.spanmetrics.useSecondAsDefaultMetricsUnit` is also added.
Currently, the feature gate is disabled by default, so the unit will remain `ms`. After one release cycle, the unit will switch to `s` and the feature gate will also be enabled by default.
## Overview
Aggregates Request, Error and Duration (R.E.D) OpenTelemetry metrics from span data.
**Request** counts are computed as the number of spans seen per unique set of
dimensions, including Errors. Multiple metrics can be aggregated if, for instance,
a user wishes to view call counts just on `service.name` and `span.name`.
```
traces.span.metrics.calls{service.name="shipping",span.name="get_shipping/{shippingId}",span.kind="SERVER",status.code="Ok"}
```
**Error** counts are computed from the Request counts which have an `Error` Status Code metric dimension.
```
traces.span.metrics.calls{service.name="shipping",span.name="get_shipping/{shippingId},span.kind="SERVER",status.code="Error"}
```
**Duration** is computed from the difference between the span start and end times and inserted into the
relevant duration histogram time bucket for each unique set dimensions.
```
traces.span.metrics.duration{service.name="shipping",span.name="get_shipping/{shippingId}",span.kind="SERVER",status.code="Ok"}
```
Each metric will have _at least_ the following dimensions because they are common
across all spans:
- `service.name`
- `span.name`
- `span.kind`
- `status.code` (or `otel.status_code` when the `spanmetrics.statusCodeConvention.useOtelPrefix` feature gate is enabled)
- `collector.instance.id`
The `collector.instance.id` dimension is intended to add a unique UUID to all metrics, ensuring that the spanmetrics connector
does not violate the **Single Writer Principle** when spanmetrics is used in a multi-deployment model.
To disable, use `exclude_dimensions` setting:
```yaml
connectors:
spanmetrics:
exclude_dimensions: ['collector.instance.id']
```
Or, disable via the feature gate: `--feature-gates=-connector.spanmetrics.includeCollectorInstanceID`.
More detail, please see [Known Limitation: the Single Writer Principle](#known-limitation-the-single-writer-principle)
## Span to Metrics processor to Span to metrics connector
The spanmetrics connector replaces [spanmetrics](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/processor/spanmetricsprocessor/v0.95.0/processor/spanmetricsprocessor/README.md) processor with multiple improvements
and breaking changes. It was done to bring the `spanmetrics` connector closer to the OpenTelemetry
specification and make the component agnostic to exporters logic. The `spanmetrics` processor
essentially was mixing the OTel with Prometheus conventions by using the OTel data model and
the Prometheus metric and attributes naming convention.
The following changes were done to the connector component.
Breaking changes:
- The `operation` metric attribute was renamed to `span.name`.
- The `latency` histogram metric name was changed to `duration`.
- The `_total` metric prefix was dropped from generated metrics names.
- The Prometheus-specific metrics labels sanitization was dropped.
Improvements:
- Added support for OTel exponential histograms for recording span duration measurements.
- Added support for the milliseconds and seconds histogram units.
- Added support for generating metrics resource scope attributes. The `spanmetrics` connector will
generate the number of metrics resource scopes that corresponds to the number of the spans resource
scopes meaning that more metrics are generated now. Previously, `spanmetrics` generated a single
metrics resource scope.
## Configurations
If you are not already familiar with connectors, you may find it helpful to first
visit the [Connectors README].
The following settings can be optionally configured:
- `histogram` (default: `explicit`): Use to configure the type of histogram to record
calculated from spans duration measurements. Must be either `explicit` or `exponential`.
- `disable` (default: `false`): Disable all histogram metrics.
- `unit` (default: `ms`): The time unit for recording duration measurements.
calculated from spans duration measurements. One of either: `ms` or `s`.
- `dimensions`: additional attributes to add as dimensions to the `traces.span.metrics.duration` metric,
which will be included _on top of_ the common and configured `dimensions` for span attributes and resource attributes.
- `explicit`:
- `buckets`: the list of durations defining the duration histogram time buckets. Default
buckets: `[2ms, 4ms, 6ms, 8ms, 10ms, 50ms, 100ms, 200ms, 400ms, 800ms, 1s, 1400ms, 2s, 5s, 10s, 15s]`
- `exponential`:
- `max_size` (default: `160`) the maximum number of buckets per positive or negative number range.
- `dimensions`: the list of dimensions to add to `traces.span.metrics.calls`, `traces.span.metrics.duration` and `traces.span.metrics.event` metrics with the default dimensions defined above.
Each list entry must set **exactly one** of:
1. `name` which is looked up in the span's collection of attributes or resource attributes (AKA process tags) such as `ip`, `host.name` or `region`.
If the `name`d attribute is missing in the span, the optional provided `default` is used.
If no `default` is provided, this dimension will be **omitted** from the metric.
2. `glob`: a glob pattern with '.' treated as a separator. Every span or resource attribute whose key matches the pattern is emitted as its own dimension.
- `calls_dimensions`: additional attributes to add as dimensions to the `traces.span.metrics.calls` metric,
which will be included _on top of_ the common and configured `dimensions` for span attributes and resource attributes.
- `exclude_dimensions`: the list of dimensions to be excluded from the default set of dimensions. Use to exclude unneeded data from metrics.
- `dimensions_cache_size`: this setting is deprecated, please use aggregation_cardinality_limit instead.
- `include_instrumentation_scope`: a list of instrumentation scope names to include from the traces.
- `resource_metrics_cache_size` (default: `1000`): the size of the cache holding metrics for a service. This is mostly relevant for
cumulative temporality to avoid memory leaks and correct metric timestamp resets.
- `aggregation_temporality` (default: `AGGREGATION_TEMPORALITY_CUMULATIVE`): Defines the aggregation temporality of the generated metrics.
One of either `AGGREGATION_TEMPORALITY_CUMULATIVE` or `AGGREGATION_TEMPORALITY_DELTA`.
- `namespace` (default: `traces.span.metrics`): Defines the namespace of the generated metrics. If `namespace` provided, generated metric name will be added `namespace.` prefix.
- `metrics_flush_interval` (default: `60s`): Defines the flush interval of the generated metrics.
- `metrics_expiration` (default: `0`): Defines the expiration time as `time.Duration`, after which, if no new spans are received, metrics will no longer be exported. Setting to `0` means the metrics will never expire.
- `series_expiration` (default: `0`): Defines the expiration time as `time.Duration` for individual metric series. When set, stale dimension combinations are removed on a later flush even if the parent metric and resource continue receiving other spans. Setting to `0` disables per-series expiration.
- `metric_timestamp_cache_size` (default `1000`): Only relevant for delta temporality span metrics. Controls the size of the cache used to keep track of a metric's TimestampUnixNano the last time it was flushed. When a metric is evicted from the cache, its next data point will indicate a "reset" in the series. Downstream components converting from delta to cumulative, like `prometheusexporter`, may handle these resets by setting cumulative counters back to 0.
- `exemplars`: Use to configure how to attach exemplars to metrics.
- `enabled` (default: `false`): enabling will add spans as Exemplars to all metrics. Exemplars are only kept for one flush interval.rom the cache, its next data point will indicate a "reset" in the series. Downstream components converting from delta to cumulative, like `prometheusexporter`, may handle these resets by setting cumulative counters back to 0.
- `max_per_data_point` (default: `5`): The maximum number of exemplars to attach to a single metric data point.
- `events`: Use to configure the events metric.
- `enabled`: (default: `false`): enabling will add the events metric.
- `dimensions`: (mandatory if `enabled`) the list of the span's event attributes to add as dimensions to the `traces.span.metrics.events` metric, which will be included _on top of_ the common and configured `dimensions` for span attributes and resource attributes.
- `resource_metrics_key_attributes`: Filter the resource attributes used to produce the resource metrics key map hash(It's only used to build the hash key, not copy the attributes to metrics resource attributes).
Use this in case changing resource attributes (e.g. process id) are breaking counter metrics.
- `aggregation_cardinality_limit` (default: `0`): Defines the maximum number of unique combinations of dimensions that will be tracked for metrics aggregation. When the limit is reached, additional unique combinations will be dropped but registered under a new entry with `otel.metric.overflow="true"`. A value of `0` means no limit is applied.
- `add_resource_attributes` (default: `false`): Add the resource attributes to the resulting metrics. This option enables the old behavior before the `connector.spanmetrics.excludeResourceMetrics` feature gate was introduced. When set to `true`, resource attributes will be included in the metrics even if the feature gate is enabled. See [GitHub issue #42103](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/42103) for more context.
- `enable_metrics_sampling_method` (default: `false`): When enabled, adds the `sampling.method` attribute to metrics with value `"extrapolated"` (when the span has a valid tracestate sampling threshold) or `"counted"` (otherwise).
The feature gate `connector.spanmetrics.legacyMetricNames` (disabled by default) controls the connector to use legacy metric names.
## Examples
The following is a simple example usage of the `spanmetrics` connector.
For configuration examples on other use cases, please refer to [More Examples](#more-examples).
The full list of settings exposed for this connector are documented in [spanmetricsconnector/config.go](../../connector/spanmetricsconnector/config.go).
```yaml
receivers:
nop:
exporters:
nop:
connectors:
span_metrics:
histogram:
dimensions:
- name: url.scheme
default: https
explicit:
buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]
dimensions:
- name: http.method
default: GET
- name: http.status_code
- glob: "k8s.*.name" # match single namespace between 'k8s' and 'name' ('k8s.cluster.name', not 'k8s.cluster.label.name')
- glob: "db.**.name" # match any number of namespaces between 'db' and 'name'
calls_dimensions:
- name: http.url
default: /ping
exemplars:
enabled: true
exclude_dimensions: ['status.code']
aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"
metrics_flush_interval: 15s
metrics_expiration: 5m
series_expiration: 5m
events:
enabled: true
dimensions:
- name: exception.type
- name: exception.message
resource_metrics_key_attributes:
- service.name
- telemetry.sdk.language
- telemetry.sdk.name
include_instrumentation_scope:
- express
service:
pipelines:
traces:
receivers: [nop]
exporters: [span_metrics]
metrics:
receivers: [span_metrics]
exporters: [nop]
```
### Using `spanmetrics` with Prometheus components
The `spanmetrics` connector can be used with Prometheus exporter components.
For some functionality of the exporters, e.g. like generation of the `target_info` metric the
incoming spans resource scope attributes must contain `service.name` and `service.instance.id`
attributes.
Let's look at the example of using the `spanmetrics` connector with the `prometheus_remote_write` exporter:
```yaml
receivers:
otlp:
protocols:
http:
grpc:
exporters:
prometheus_remote_write:
endpoint: http://localhost:9090/api/v1/write
target_info:
enabled: true
connectors:
span_metrics:
namespace: span.metrics
service:
pipelines:
traces:
receivers: [otlp]
exporters: [span_metrics]
metrics:
receivers: [span_metrics]
exporters: [prometheus_remote_write]
```
This configures the `spanmetrics` connector to generate metrics from received spans and export the
metrics to the Prometheus Remote Write exporter. The `target_info` metric will be generated for each
resource scope, while OpenTelemetry metric names and attributes will be [normalized](../../exporter/prometheusremotewriteexporter/README.md)
to be compliant with Prometheus naming rules. For example, the generated `calls` OTel sum metric can
result in multiple Prometheus `calls_total` (counter type) time series and the `target_info` time series.
For example:
```
target_info{job="shippingservice", instance="...", ...} 1
calls_total{span_name="/Address", service_name="shippingservice", span_kind="SPAN_KIND_SERVER", status_code="STATUS_CODE_UNSET", ...} 142
```
### More Examples
For more example configuration covering various other use cases, please visit the [testdata directory](../../connector/spanmetricsconnector/testdata).
[Connectors README]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md
## Known Limitation: the Single Writer Principle
Proper configuration of the `spanmetricsconnector` ensures compliance with the [Single Writer Principle](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#single-writer),
which is a core requirement in the OpenTelemetry metrics data model. Misconfiguration, however,
may allow multiple components to write to the same metric stream, resulting in data inconsistency,
metric conflicts, or the dropping of time series by metric backends.
### Why this happens
This issue typically arises when:
* Multiple pipelines use the same instance of the `spanmetricsconnector`
* The connector is instantiated more than once without ensuring the resulting metric streams are distinct
* The `resource_metrics_key_attributes` field is not configured correctly or includes common/shared attributes across all instances
### Recommendations
To reduce the risk of conflicting writes:
* Add `resource_metrics_key_attributes` to your configuration.
```
connectors:
span_metrics:
resource_metrics_key_attributes:
- service.name
- telemetry.sdk.language
- telemetry.sdk.name
```
* The feature gate `connector.spanmetrics.includeCollectorInstanceID` is enabled by default to produce uniquely identified metrics.
* For exporters like Prometheus, which rely on the single writer assumption, use a dedicated pipeline with a single `spanmetricsconnector` instance
More context is available in [GitHub issue #21101](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/21101).
### About `resource_metrics_key_attributes`
The `resource_metrics_key_attributes` setting are used to build the key map that determines how metrics are grouped.
If this field is left empty, the connector will use **all** available attributes to compute the resource metric hash.
To avoid problems, be cautious when choosing which attributes to include.
Avoid attributes that:
* **Change frequently** – such as `request_id`, `timestamp`, or `trace_id`. These increase cardinality and create excessive metric streams.
* **Are shared across all sources** – values like `true`, `default`, or `team:backend` offer no uniqueness and can lead to multiple writers sharing the same stream.
* **Are optional or inconsistently applied** – if an attribute is only present in some spans, this can fragment metric streams (e.g., one stream with the attribute and one without).
Instead, use attributes that are stable, present in all spans, and meaningfully distinguish each stream. Good examples include `cluster_id`, `region`, or `deployment_environment`.
## Troubleshooting span metrics high cardinality
High cardinality issues in span metrics commonly manifest in APM dashboards as an excessive number of service operations with non-unique names. Examples include URIs with unique identifiers (e.g., `GET /product/1YMWWN1N4O`) or HTTP parameters with random values (e.g., `GET /?_ga=GA1.2.569539246.1760114706`). These patterns render operation lists difficult to interpret and ineffective for monitoring purposes.
This issue stems from violations of [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/), which require span names to have low cardinality (e.g. [HTTP span name specs](https://opentelemetry.io/docs/specs/semconv/http/http-spans/#name)).
Beyond degrading APM interfaces with numerous non-meaningful operation names, this problem leads to metric time series explosion, resulting in significant performance degradation and increased costs.
The span metrics connector provides an optional circuit breaker through the `aggregation_cardinality_limit` attribute (disabled by default) to mitigate cardinality explosion. While this feature addresses performance and cost concerns, it does not resolve the underlying issue of semantically meaningless operation names.
### Fixing high cardinality span name issues
The ideal long-term solution is to modify the OpenTelemetry instrumentation code to comply with semantic conventions, preventing the generation of non-compliant high cardinality span names.
However, deploying updated instrumentation libraries can be time-consuming, often requiring an immediate interim solution to restore observability backend functionality.
#### Addressing high cardinality span names in the ingestion pipeline
An effective short-term solution is to implement a span sanitization layer within the observability ingestion pipeline. This can be achieved by using the OpenTelemetry Collector Transform Processor's [`set_semconv_span_name()` function](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor#set_semconv_span_name) immediately before the Span Metrics Connector to enforce semantic conventions on span names.
Simple example OpenTelemetry Collector configuration to prevent span metrics cardinality explosion:
```yaml
receivers:
otlp:
...
processors:
transform/sanitize_spans:
error_mode: ignore
trace_statements:
# Sanitize all span names to prevent span metrics cardinality explosion
# caused by non-compliant high cardinality span names
- set_semconv_span_name("1.37.0")
...
connectors:
span_metrics:
exporters:
otlp_http/observability-backend:
...
service:
pipelines:
traces:
receivers: [otlp]
processors: [transform/sanitize_spans, ...]
exporters: [otlp_http/observability-backend, span_metrics]
metrics:
receivers: [otlp, span_metrics]
processors: [...]
exporters: [otlp_http/observability-backend]
# ...
```
Aggressive span name sanitization may be overly restrictive for instrumentations with incomplete resource attributes. For instance, HTTP service operations may be reduced to generic names like `GET` and `POST` when HTTP spans lack the `http.route` attribute. This information loss can impact the monitoring of critical business operations.
To preserve operation granularity, you can manually set the `http.route` attribute when detailed operation names are required. The missing `http.route` value can typically be derived through pattern matching on other span attributes such as `http.target` or `url.full`.
Example OpenTelemetry Collector configuration that prevents cardinality explosion
while preserving meaningful operation names on a service `webshop/frontend`:
```
/* Detailed source-code truncated for AI context efficiency. */
```
#### Addressing high cardinality span names in the instrumentation code
The preferred long-term solution is to ensure span names and attributes comply with [OpenTelemetry Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/) directly in the instrumentation code.
Custom web frameworks are a common source of high cardinality span names. While default OpenTelemetry instrumentation (e.g., Java Servlet) may assign generic span names like `GET /my-web-fwk/*`, your framework has access to more specific routing information. By overwriting span attributes in your framework code, you can create compliant, low-cardinality span names that preserve operational granularity.
**Example: Custom Web Framework in Java**
Consider a custom web framework that intercepts the generic route `/my-web-fwk/*` and dispatches requests like `/my-web-fwk/product/123456ABCD` or `/my-web-fwk/user/john.doe`.
The default Java Servlet instrumentation produces vague span names (`GET /my-web-fwk/*`), while directly using request URIs creates high cardinality (`GET /my-web-fwk/product/123456ABCD`).
The solution is to override span attributes with templated route patterns like `/my-web-fwk/product/{productId}` or `/my-web-fwk/user/{userId}`:
```java
@WebServlet(urlPatterns = "/my-web-fwk/*")
public class MyWebFrameworkServlet extends HttpServlet {
@Override
protected void service(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
// Default Servlet instrumentation sets vague span names: `GET /my-web-fwk/*` (and `http.route=/my-web-fwk/*`)
// Using the request URI directly would cause high cardinality and violate semantic conventions
// Instead, use the framework's low-cardinality routing information below
// Example routing logic
String uri = request.getRequestURI();
MyWebOperation myWebOperation = getWebOperation(uri);
// Fix span details to add details while complying with semantic conventions and maintaining low cardinality
String httpRoute = "/my-web-fwk/" + myWebOperation.getSubHttpRoute();
Span.current().setAttribute(HttpAttributes.HTTP_ROUTE, httpRoute);
Span.current().updateName(request.getMethod() + " " + httpRoute);
// execute the web operation
myWebOperation.execute(request, response);
}
...
}
```
---
### Overview (`connector/sumconnector/README.md`)
# Sum Connector
| Status | |
| ------------- |-----------|
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fsum) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fsum) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=connector_sum&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@greatestusername](https://www.github.com/greatestusername), [@shalper2](https://www.github.com/shalper2), [@crobert-1](https://www.github.com/crobert-1) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
## Supported Pipeline Types
| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | metrics | [alpha] |
| metrics | metrics | [alpha] |
| logs | metrics | [alpha] |
[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#stability-levels
The `sum` connector can be used to sum attribute values from spans, span events, metrics, data points, and log records.
## Configuration
If you are not already familiar with connectors, you may find it helpful to first visit the [Connectors README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md).
### Basic configuration
This example configuration will sum numerical values found within the attribute `attribute.with.numerical.value` of any span telemetry routed to the connector. It will then output a metric time series with the name `my.example.metric.name` with those summed values.
Note: Values found within an attribute will be converted into a float regardless of their original type before being summed and output as a metric value. Non-convertible strings will be dropped and not included.
```yaml
receivers:
foo:
connectors:
sum:
spans:
my.example.metric.name:
source_attribute: attribute.with.numerical.value
exporters:
bar:
service:
pipelines:
metrics/sum:
receivers: [sum]
exporters: [bar]
traces:
receivers: [foo]
exporters: [sum]
```
#### Required Settings
The sum connector has three required configuration settings and numerous optional settings
- Telemetry type: Nested below the `sum:` connector declaration. Declared as `spans:` in the [Basic Example](#basic-configuration).
- Can be any of `spans`, `spanevents`, `datapoints`, or `logs`.
- For metrics use `datapoints`
- For traces use `spans` or `spanevents`
- Metric name: Nested below the telemetry type; this is the metric name the sum connector will output summed values to. Declared as `my.example.metric.name` in the [Basic Example](#basic-configuration)
- `source_attribute`: A specific attribute to search for within the source telemetry being fed to the connector. This attribute is where the connector will look for numerical values to sum into the output metric value. Declared as `attribute.with.numerical.value` in the [Basic Example](#basic-configuration)
#### Optional Settings
- `conditions`: [OTTL syntax](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/LANGUAGE.md) can be used to provide conditions for processing incoming telemetry. Conditions are ORed together, so if any condition is met the attribute's value will be included in the resulting sum. Conditions support OTTL path context names (e.g. `span.attributes["x"]`, `resource.attributes["x"]`, `log.attributes["x"]`, `metric.name`). Paths without an explicit context are interpreted in the context of the enclosing block. It is recommended to switch to the new syntax to avoid breaking changes in the future.
- `attributes`: Declaration of attributes to include. Any of these attributes found will generate a separate sum for each set of unique combination of attribute values and output as its own datapoint in the metric time series.
- `key`: (required for `attributes`) the attribute name to match against
- `default_value`: (optional for `attributes`) a default value for the attribute when no matches are found. The `default_value` value can be of type string, integer, or float.
### Detailed Example Configuration
This example declares that the `sum` connector is going to be ingesting `logs` and creating an output metric named `checkout.total` with numerical values found in the `source_attribute` `total.payment`.
It provides a condition to check that the attribute `total.payment` is not `NULL`. It also checks any incoming log telemetry for values present in the attribute `payment.processor` and creates a datapoint within the metric time series for each unique value. Any logs without values in `payment.processor` will be included in a datapoint with the `default_value` of `unspecified_processor`.
```yaml
receivers:
foo:
connectors:
sum:
logs:
checkout.total:
source_attribute: total.payment
conditions:
- attributes["total.payment"] != "NULL"
attributes:
- key: payment.processor
default_value: unspecified_processor
exporters:
bar:
service:
pipelines:
metrics/sum:
receivers: [sum]
exporters: [bar]
logs:
receivers: [foo]
exporters: [sum]
```
**Note for Log to Metrics:** If your logs contain all values in their `body` rather than in attributes (E.G. JSON payload) use a transform processor in your pipeline to upsert [parsed key/value pairs](https://github.com/open-telemetry/opentelemetry-log-collection/tree/main/docs/operators) (in this case from JSON) into attributes attached to the log.
```yaml
processors:
transform/logs:
log_statements:
- context: log
statements:
- merge_maps(attributes, ParseJSON(body), "upsert")
```
[Connectors README]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md
---
### Overview (`examples/couchbase/README.md`)
# Couchbase Example
This example shows how you can configure the collector to scrape a [Couchbase server](https://www.couchbase.com/products/server) using its built-in Prometheus metrics exporter.
This [example config](./otel-collector-config.yaml) retrieves a subset of metrics important for monitoring a Couchbase server, transforms them to follow OpenTelemetry conventions, then exports again to prometheus so that they may be easily viewed.
## Running
First build the collector docker image by running the following command in the root of this repository:
```sh
make docker-otelcontribcol
```
Then, start Couchbase, the collector, and a Prometheus server by running the following command in this directory:
```sh
docker-compose up -d --remove-orphans --build --force-recreate
```
After Couchbase is running, you'll need to initialize Couchbase and create some buckets:
```sh
./scripts/setup.sh
```
You should now be able to view metrics being scraped from the Couchbase instance every 5 seconds in Prometheus at http://localhost:9090 (all metrics are prefixed with `couchbase_`)
Once you are done with the example, you may shut it down:
```sh
docker-compose down
```
---
### Overview (`examples/fault-tolerant-logs-collection/README.md`)
## Fault tolerant log collection example
File Log receiver's persistence can be covered by the usage of the following extensions:
- [filestorage](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage) extension,
to ensure that Collector's restarts do not affect the log collection and offset tracking.
- [exporterhelper persistent-queue](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md#persistent-queue),
to ensure that Collector's restarts do not affect the delivery of the already collected logs.
A full configuration example is provided in [example config](./otel-col-config.yaml)
---
### Overview (`examples/kubernetes/README.md`)
# OpenTelemetry Collector Demo
This demo is a sample app to build the collector and exercise its kubernetes logs scraping functionality.
## Build and Run
### Kubernetes
Switch to this directory and run following command: `kubectl apply -n -f otel-collector.yaml`
#### Include/Exclude Specific Logs
Kubernetes logs are being stored in `/var/log/pods`.
Path to container logs is constructed using following pattern:
`/var/log/pods/__//.log`
You can use it to manage from which containers do you want to include and exclude logs.
For example, to include all logs from `default` namespace, you can use following configuration:
```yaml
include:
- /var/log/pods/default_*/*/*.log
exclude: []
```
### Docker Compose
Two steps are required to build and run the demo:
1. Build latest docker image in main repository directory `make docker-otelcontribcol`
1. Switch to this directory and run `docker-compose up`
#### Description
`varlogpods` contains example log files placed in kubernetes-like directory structure.
Each of the directory has different formatted logs in one of three formats (either `CRI-O`, `CRI-Containerd` or `Docker`).
This directory is mounted to standard location (`/var/log/pods`).
`otel-collector-config` is a configuration to autodetect and parse logs for all of three mentioned formats
## Performance Tests
There are multiple tests for various configurations in [`testbed`](../../testbed/tests/log_test.go).
Following table shows result of example run:
Test |Result|Duration|CPU Avg%|CPU Max%|RAM Avg MiB|RAM Max MiB|Sent Items|Received Items|
----------------------------------------|------|-------:|-------:|-------:|----------:|----------:|---------:|-------------:|
Log10kDPS/OTLP |PASS | 15s| 10.4| 11.0| 68| 94| 150100| 150100|
Log10kDPS/filelog |PASS | 15s| 7.8| 8.7| 65| 93| 150100| 150100|
Log10kDPS/kubernetes_containers |PASS | 15s| 18.3| 19.7| 67| 96| 150100| 150100|
Log10kDPS/kubernetes_containers_parser |PASS | 15s| 18.2| 19.0| 66| 95| 150100| 150100|
Log10kDPS/k8s_CRI-Containerd |PASS | 15s| 15.4| 16.3| 66| 95| 150100| 150100|
Log10kDPS/k8s_CRI-Containerd_no_attr_ops|PASS | 15s| 15.1| 16.3| 66| 95| 150100| 150100|
Log10kDPS/CRI-Containerd |PASS | 15s| 11.1| 13.0| 65| 93| 150100| 150100|
## ToDo
To cover kubernetes system logs, logs from journald should be supported as well.
---
### Overview (`examples/logline-filtering/README.md`)
## Filtering log messages based on content
File Log receiver provides support for filtering logs based on their content. This can be achieved by using
the [filter operator](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/stanza/docs/operators/filter.md),
configured with matching regular expressions.
With this happening at the collection point, a lot of resources at the destination backend
can be saved since no additional processing would need to take place.
A full configuration example on how to filter out logs that start with the `INFO:` pattern is
provided in the [example config](./otel-col-config-filter-out-logs.yaml).
A full configuration example on how to only collect logs that start with the `WARN:` pattern is provided in
the [example config](./otel-col-config-filter-in-logs.yaml)
---
### Overview (`examples/nomad/README.md`)
# OpenTelemetry Collector Demo
This demo provides you with all you need to run the OpenTelemetry Collector on a local installation of HashiCorp [Nomad](https://nomadproject.io) using [Traefik](https://traefik.io) as your load balancer. It configures both both HTTP and gRPC endpoints on Traefik.
For a detailed, step-by-step tutorial on how to run the files included in this directory, check out [Just-in-Time Nomad: Running the OpenTelemetry Collector on HashiCorp Nomad with HashiQube](https://storiesfromtheherd.com/just-in-time-nomad-running-the-opentelemetry-collector-on-hashicorp-nomad-with-hashiqube-4eaf009b8382).
The tutorial includes instructions on how to set up a local Nomad environment using [HashiQube](https://github.com/avillela/hashiqube) (a fork of [servian/hashiqube](https://github.com/servian/hashiqube)). It spins up a Vagrant VM running [Nomad](https://nomadproject.io), [Consul](https://www.consul.io), [Vault](https://www.vaultproject.io) and [Traefik](https://traefik.io). Detailed instructions can be found in the project [readme](https://github.com/avillela/hashiqube#readme).
## Additional Resources
Check out the [OpenTelemetry Collector Nomad Pack](https://github.com/hashicorp/nomad-pack-community-registry/tree/main/packs/opentelemetry_collector) in the [Nomad Pack Community Registry](https://github.com/hashicorp/nomad-pack-community-registry).
---
### Overview (`examples/secure-tracing/README.md`)
# Build A Secure Trace Collection Pipeline
Implementing robust security measures is essential for any tracing ingestion service and infrastructure. This is an illustrative example of a secure setup encompassing the following features for trace ingestion:
- Data Encryption with OTLP receiver supporting TLS. Transport Layer Security (TLS) is employed to encrypt traces in transit between the application and the OpenTelemetry (OTLP) endpoint, fortifying data security.
- Client Authentication via [Envoy](https://www.envoyproxy.io/docs/envoy/latest/start/start), a high-performance proxy.
Even though we can configure OTLP receiver with mTLS for client authentication, authorization is not supported by OpenTelemetry Collector. This is one of the reasons that we use Envoy for client authentication. It allows us to easily add authorization, ensuring that only authorized clients can submit traces to the ingestion service.
In this example, we also include a test via telementrygen: a tool provided from this repository for generating synthetic telemetry data, which helps verify the security features and overall functionality of the set up.
## Data Encryption via TLS
The OpenTelemetry Collector has detailed [documentation](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md) on how to configure TLS. In this example, we enable TLS for receivers which leverages server configuration.
Example
```
receivers:
otlp:
protocols:
grpc:
endpoint: mysite.local:55690
tls:
cert_file: server.crt
key_file: server.key
```
## Client Authentication
Envoy [sandbox for TLS](https://www.envoyproxy.io/docs/envoy/latest/start/sandboxes/tls) is a good source for reference. A few elements in following configuration are good to call out,
- `require_client_certificate` is set true
- The `matcher` under `validation_context` expects a client certificate with `group-id` as well as `tenat-id` which provides more granular control.
```
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
require_client_certificate: true
common_tls_context:
# h2 If the listener is going to support both HTTP/2 and HTTP/1.1.
alpn_protocols: "h2"
tls_certificates:
- certificate_chain:
filename: "/etc/envoy.crt"
private_key:
filename: "/etc/envoy.key"
validation_context:
match_typed_subject_alt_names:
- san_type: URI
matcher:
Match tenant by two level info: group id and tenant-id.
exact: "aprn:trace-client:certmgr:::group-x:/ig/5003178/uv/tenant-a"
trusted_ca:
filename: "/etc/ca.crt"
```
## Setup Environment
### Generate Certificates
To generate various self-signed certificates, including those for Envoy and the OpenTelemetry Collector receiver, as well as tracing client certificate, we utilize the widely renowned open-source tool [OpenSSL](https://www.openssl.org/source/), OpenSSL 3.1.0 14 was tested.
In the `certs` folder, you can find a set of `.ext` files which define the properties for a certificate. A `MakeFile` is provided to facilate the process.
```
$ cd certs
$ make clean && make all
```
### Bring up services
We use docker compose to bring up Envoy and OpenTelemetry Collection Pipeline. Make sure the current folder is `secure-tracing`,
```
$ docker compose up
```
From the console window, verify that `Envoy` and `Collector` are up and running. If you see error similar to following,
```
secure-tracing-otel-collector-1 | Error: cannot start pipelines: failed to load TLS config: failed to load TLS cert and key: read /etc/otel-collector.crt: is a directory
```
It's most likely due to missing certificates. Follow the steps from section above to generate certificates.
## Run test
### Compile telemetrygen
From the root of this repository,
```
$ cd cmd/telemetrygen
$ go build .
$ cp telemetrygen ../../examples/secure-tracing
```
### Send a trace
From the root of this repository,
```
$ cd examples/secure-tracing
$ chmod +x telemetrygen
$ ./telemetrygen traces --traces 1 --otlp-endpoint 127.0.0.1:10000 --ca-cert 'certs/ca.crt' --mtls --client-cert 'certs/tracing-client.crt' --client-key 'certs/tracing-client.key'
```
Verify traces are captured and sent to console running `docker compose up`, similar to following ...
```
secure-tracing-otel-collector-1 | -> service.name: Str(telemetrygen)
secure-tracing-otel-collector-1 | ScopeSpans #0
secure-tracing-otel-collector-1 | ScopeSpans SchemaURL:
secure-tracing-otel-collector-1 | InstrumentationScope telemetrygen
secure-tracing-otel-collector-1 | Span #0
secure-tracing-otel-collector-1 | Trace ID : 0fe7ca900fda938ce918f8b2d82d6ae9
secure-tracing-otel-collector-1 | Parent ID : 1011b3f973049923
secure-tracing-otel-collector-1 | ID : 65f3cfe524375dd4
secure-tracing-otel-collector-1 | Name : okey-dokey
secure-tracing-otel-collector-1 | Kind : Server
...
secure-tracing-otel-collector-1 | Attributes:
secure-tracing-otel-collector-1 | -> net.sock.peer.addr: Str(1.2.3.4)
secure-tracing-otel-collector-1 | -> peer.service: Str(telemetrygen-client)
...
```
## Shutdown Services
```
$ docker compose down
```
---
### Overview (`exporter/alertmanagerexporter/README.md`)
# Alertmanager Exporter
| Status | |
| ------------- |-----------|
| Stability | [development]: traces, logs |
| Distributions | [] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Falertmanager) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Falertmanager) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_alertmanager&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@sokoide](https://www.github.com/sokoide), [@mcube8](https://www.github.com/mcube8) |
[development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development
Exports OTEL Events (SpanEvent in Tracing added by AddEvent API and Log Records) as Alerts to [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) back-end to notify Errors or Change events.
Supported pipeline types: traces, logs
## Getting Started
The following settings are required:
- `endpoint`: Alertmanager endpoint to send events
- `severity`: Default severity for Alerts
The following settings are optional:
- `timeout` `sending_queue` and `retry_on_failure` settings as provided by [Exporter Helper](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper#configuration).
- [HTTP settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md)
- [TLS and mTLS settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md)
- `generator_url` is the source of the alerts to be used in Alertmanager's payload. The default value is "opentelemetry-collector", and can be set to the URL of the opentelemetry collector.
- `severity_attribute` is the SpanEvent or LogRecord Attribute name which can be used instead of default severity string in Alert payload.
e.g.: If `severity_attribute` is set to "foo" and the SpanEvent or LogRecord has an attribute called foo, foo's attribute value will be used as the severity value for that particular Alert generated from the SpanEvent or LogRecord.
- `api_version` is the API version of [Alertmanager](https://prometheus.io/docs/alerting/latest/clients/). By default the value is set to "v2" and can be overridden to "v1" if using an older version of Alertmanager.
- `event_labels` is the list of Event or LogRecord Attributes that will be captured as Labels in the Alert payload if value exists.
Example config:
```yaml
exporters:
alertmanager:
alertmanager/2:
endpoint: "https://a.new.alertmanager.target:9093"
severity: "debug"
severity_attribute: "foo"
api_version: "v2"
event_labels: ["foo", "bar"]
tls:
cert_file: /var/lib/mycert.pem
key_file: /var/lib/key.pem
timeout: 10s
sending_queue:
enabled: true
num_consumers: 2
queue_size: 10
retry_on_failure:
enabled: true
initial_interval: 10s
max_interval: 60s
max_elapsed_time: 10m
generator_url: "opentelemetry-collector"
```
---
### Overview (`exporter/alibabacloudlogserviceexporter/README.md`)
# AlibabaCloud LogService Exporter
| Status | |
| ------------- |-----------|
| Stability | [beta]: traces, metrics, logs |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Falibabacloudlogservice) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Falibabacloudlogservice) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_alibabacloudlogservice&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@vyagh](https://www.github.com/vyagh) |
| Emeritus | [@shabicheng](https://www.github.com/shabicheng), [@kongluoxing](https://www.github.com/kongluoxing), [@qiansheng91](https://www.github.com/qiansheng91) |
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
This exporter supports sending OpenTelemetry data to [LogService](https://www.alibabacloud.com/product/log-service).
# Configuration options:
- `endpoint` (required): LogService's [Endpoint](https://www.alibabacloud.com/help/doc-detail/29008.htm).
- `project` (required): LogService's Project Name.
- `logstore` (required): LogService's store Name. For metrics data, you should use metric store.
- `access_key_id` (optional): AlibabaCloud access key id.
- `access_key_secret` (optional): AlibabaCloud access key secret.
- `security_token` (optional): AlibabaCloud security token for STS credentials.
- `ecs_ram_role` (optional): set AlibabaCLoud ECS ram role if you are using ACK.
- `token_file_path` (optional): Set token file path if you are using ACK.
# Example:
## Simple Trace Data
```yaml
receivers:
examplereceiver:
exporters:
alibabacloud_logservice:
endpoint: "cn-hangzhou.log.aliyuncs.com"
project: "demo-project"
logstore: "traces-store"
access_key_id: "access-key-id"
access_key_secret: "access-key-secret"
service:
pipelines:
traces:
receivers: [examplereceiver]
exporters: [alibabacloud_logservice]
```
## All Telemetry Data
If you are using OpenTelemetry Collector to collect different types of telemetry data, you should send to different LogService's store.
```yaml
receivers:
examplereceiver:
exporters:
alibabacloud_logservice/logs:
endpoint: "cn-hangzhou.log.aliyuncs.com"
project: "demo-project"
logstore: "logs-store"
access_key_id: "access-key-id"
access_key_secret: "access-key-secret"
alibabacloud_logservice/metrics:
endpoint: "cn-hangzhou.log.aliyuncs.com"
project: "demo-project"
logstore: "metrics-store"
access_key_id: "access-key-id"
access_key_secret: "access-key-secret"
alibabacloud_logservice/traces:
endpoint: "cn-hangzhou.log.aliyuncs.com"
project: "demo-project"
logstore: "traces-store"
access_key_id: "access-key-id"
access_key_secret: "access-key-secret"
service:
pipelines:
traces:
receivers: [examplereceiver]
exporters: [alibabacloud_logservice/traces]
logs:
receivers: [examplereceiver]
exporters: [alibabacloud_logservice/logs]
metrics:
receivers: [examplereceiver]
exporters: [alibabacloud_logservice/metrics]
```
---
### Overview (`exporter/awscloudwatchlogsexporter/README.md`)
# AWS CloudWatch Logs Exporter
| Status | |
| ------------- |-----------|
| Stability | [alpha]: logs |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fawscloudwatchlogs) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fawscloudwatchlogs) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_awscloudwatchlogs&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@yaten2302](https://www.github.com/yaten2302) \| Seeking more code owners! |
| Emeritus | [@bryan-aguilar](https://www.github.com/bryan-aguilar), [@boostchicken](https://www.github.com/boostchicken), [@rapphil](https://www.github.com/rapphil) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
AWS CloudWatch Logs Exporter sends logs data to AWS [CloudWatch Logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html).
AWS credentials are retrieved from the [default credential chain](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html#specifying-credentials).
NOTE: OpenTelemetry Logging support is experimental, hence this exporter is subject to change.
## Configuration
The following settings are required:
- `log_group_name`: The group name of the CloudWatch Logs. If it does not exist it will be created automatically. It supports several placeholder names. One valid example is `/aws/metrics/{ClusterName}`. It will search for ClusterName (or aws.ecs.cluster.name) resource attribute in the metrics data and replace with the actual cluster name. If none of them are found in the resource attribute map, {ClusterName} will be replaced by undefined. Similar way, for the {TaskId}, it searches for TaskId (or aws.ecs.task.id) key in the resource attribute map. For {NodeName}, it searches for NodeName (or k8s.node.name)
- List of valid placeholders:
- `{ClusterName}`: `aws.ecs.cluster.name` or `k8s.cluster.name`
- `{TaskId}`: `aws.ecs.task.id`
- `{NodeName}`: `k8s.node.name`
- `{PodName}`: `pod` or `k8s.pod.name`
- `{ServiceName}`: `service.name`
- `{ContainerInstanceId}`: `aws.ecs.container.instance.id`
- `{TaskDefinitionFamily}`: `aws.ecs.task.family`
- `{InstanceId}`: `service.instance.id`
- `{FaasName}`: `faas.name`
- `{FaasVersion}`: `faas.version`
- `log_stream_name`: The stream name of the CloudWatch Logs. If it does not exist it will be created automatically. It supports the same placeholders as `log_group_name`
The following settings can be optionally configured:
- `region`: The AWS region where the log stream is in. Region must be specified if it is not already set in the default credential chain.
- `endpoint`: The CloudWatch Logs service endpoint which the requests are forwarded to. [See the CloudWatch Logs endpoints](https://docs.aws.amazon.com/general/latest/gr/cwl_region.html) for a list.
- `log_retention`: LogRetention is the option to set the log retention policy for only newly created CloudWatch Log Groups. Defaults to Never Expire if not specified or set to 0. Possible values for retention in days are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, or 3653.
- `tags`: Tags is the option to set tags for the CloudWatch Log Group. If specified, please add at most 50 tags. Input is a string to string map like so: { 'key': 'value' }. Keys must be between 1-128 characters and follow the regex pattern: `^([\p{L}\p{Z}\p{N}_.:/=+\-@]+)$`(alphanumerics, whitespace, and _.:/=+-!). Values must be between 1-256 characters and follow the regex pattern: `^([\p{L}\p{Z}\p{N}_.:/=+\-@]\*)$`(alphanumerics, whitespace, and \_.:/=+-!). [Link to tagging restrictions](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateLogGroup.html#:~:text=Required%3A%20Yes-,tags,-The%20key%2Dvalue)
- `raw_log`: Boolean default false. If set to true, only the log message will be exported to CloudWatch Logs. This needs to be set to true for [EMF logs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html).
- `max_event_payload_bytes`: Per-event payload cap, in bytes, used to truncate oversized log events before they are sent to CloudWatch Logs. As of [2025-04-02](https://aws.amazon.com/about-aws/whats-new/2025/04/amazon-cloudwatch-logs-increases-log-event-size-1-mb/) the CloudWatch Logs PutLogEvents API accepts events up to 1 MiB (1048576 bytes); the previous limit was 256 KiB. To preserve backwards compatibility this exporter still defaults to 256 KiB. Set this field to `1048576` (or any value up to that ceiling) to take advantage of the larger service limit. Allowed range: 41 to 1048576. A value of 0 (or omitted) means "use the package default (256 KiB)".
- `role_arn`: IAM role to upload logs to a different account.
- `external_id`: Shared identitier used when assuming an IAM role in an external AWS account. [See AWS IAM Guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_common-scenarios_third-party.html#id_roles_third-party_external-id)
- `sending_queue`: [Parameters for the sending queue](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md), where you can control parallelism and the size of the sending buffer. Obs.: this component will always have a sending queue enabled.
- `num_consumers`: Number of consumers that will consume from the sending queue. This parameter controls how many consumers will consume from the sending queue in parallel.
- `queue_size`: Maximum number of batches kept in memory before dropping; ignored if enabled is false
### Examples
Simplest configuration:
```yaml
exporters:
awscloudwatchlogs:
log_group_name: "testing-logs"
log_stream_name: "testing-integrations-stream"
```
Example configuration for EMF logs:
```yaml
exporters:
awscloudwatchlogs:
log_group_name: "testing-logs-emf"
log_stream_name: "testing-integrations-stream-emf"
raw_log: true
region: "us-east-1"
endpoint: "logs.us-east-1.amazonaws.com"
log_retention: 365
tags: { "sampleKey": "sampleValue" }
```
## Additional Notes
- If the log group and/or log stream are specified in an EMF log, that EMF log will be exported to that log group and/or log stream (i.e. ignores the log group and log stream defined in the configuration)
- The log group and log stream will also be created automatically if they do not already exist.
- Example of an EMF log with log group and log stream:
```json
{
"_aws": {
"Timestamp": 1574109732004,
"LogGroupName": "Foo",
"LogStreamName": "Bar",
"CloudWatchMetrics": [
{
"Namespace": "MyApp",
"Dimensions": [["Operation"]],
"Metrics": [
{
"Name": "ProcessingLatency",
"Unit": "Milliseconds",
"StorageResolution": 60
}
]
}
]
},
"Operation": "Aggregator",
"ProcessingLatency": 100
}
```
- Resource ARNs (Amazon Resource Name (ARN) of the AWS resource running the collector) are currently not supported with the CloudWatch Logs Exporter.
---
### Overview (`exporter/awsemfexporter/README.md`)
# AWS CloudWatch EMF Exporter
| Status | |
| ------------- |-----------|
| Stability | [beta]: metrics |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fawsemf) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fawsemf) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_awsemf&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@Aneurysm9](https://www.github.com/Aneurysm9), [@mxiamxia](https://www.github.com/mxiamxia) |
| Emeritus | [@shaochengwang](https://www.github.com/shaochengwang), [@bryan-aguilar](https://www.github.com/bryan-aguilar) |
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
This exporter converts OpenTelemetry metrics to
[AWS CloudWatch Embedded Metric Format(EMF)](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Embedded_Metric_Format_Specification.html)
and then sends them directly to CloudWatch Logs using the
[PutLogEvents](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html) API.
## Data Conversion
Convert OpenTelemetry ```Int64DataPoints```, ```DoubleDataPoints```, ```SummaryDataPoints``` metrics datapoints into
CloudWatch ```EMF``` structured log formats and send it to CloudWatch. Logs and Metrics will be displayed in
CloudWatch console. NaN, Inf values are not supported by CloudWatch EMF and will be dropped by the exporter.
## Exporter Configuration
The following exporter configuration parameters are supported.
| Name | Description | Default |
|:---------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ------- |
| `log_group_name` | Customized log group name which supports `{ClusterName}` and `{TaskId}` placeholders. One valid example is `/aws/metrics/{ClusterName}`. It will search for `ClusterName` (or `aws.ecs.cluster.name`) resource attribute in the metrics data and replace with the actual cluster name. If none of them are found in the resource attribute map, `{ClusterName}` will be replaced by `undefined`. Similar way, for the `{TaskId}`, it searches for `TaskId` (or `aws.ecs.task.id`) key in the resource attribute map. For `{NodeName}`, it searches for `NodeName` (or `k8s.node.name`) |"/metrics/default"|
| `log_stream_name` | Customized log stream name which supports `{TaskId}`, `{ClusterName}`, `{NodeName}`, `{ContainerInstanceId}`, and `{TaskDefinitionFamily}` placeholders. One valid example is `{TaskId}`. It will search for `TaskId` (or `aws.ecs.task.id`) resource attribute in the metrics data and replace with the actual task id. If none of them are found in the resource attribute map, `{TaskId}` will be replaced by `undefined`. Similarly, for the `{TaskDefinitionFamily}`, it searches for `TaskDefinitionFamily` (or `aws.ecs.task.family`). For the `{ClusterName}`, it searches for `ClusterName` (or `aws.ecs.cluster.name`). For `{NodeName}`, it searches for `NodeName` (or `k8s.node.name`). For `{ContainerInstanceId}`, it searches for `ContainerInstanceId` (or `aws.ecs.container.instance.id`). (Note: ContainerInstanceId (or `aws.ecs.container.instance.id`) only works for AWS ECS EC2 launch type. |"otel-stream"|
| `log_retention` | LogRetention is the option to set the log retention policy for only newly created CloudWatch Log Groups. Defaults to Never Expire if not specified or set to 0. Possible values for retention in days are 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, 2192, 2557, 2922, 3288, or 3653. |"Never Expire"|
| `tags` | Tags is the option to set tags for the CloudWatch Log Group. If specified, please add at most 50 tags. Input is a string to string map like so: { 'key': 'value' }. Keys must be between 1-128 characters and follow the regex pattern: `^([\p{L}\p{Z}\p{N}_.:/=+\-@]+)$`(alphanumerics, whitespace, and _.:/=+-!). Values must be between 1-256 characters and follow the regex pattern: `^([\p{L}\p{Z}\p{N}_.:/=+\-@]*)$`(alphanumerics, whitespace, and _.:/=+-!). [Link to tagging restrictions](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_CreateLogGroup.html#:~:text=Required%3A%20Yes-,tags,-The%20key%2Dvalue) | No tags set |
| `namespace` | Customized CloudWatch metrics namespace | "default" |
| `endpoint` | Optionally override the default CloudWatch service endpoint. | |
| `no_verify_ssl` | Enable or disable TLS certificate verification. | false |
| `proxy_address` | Upload Structured Logs to AWS CloudWatch through a proxy. | |
| `region` | Send Structured Logs to AWS CloudWatch in a specific region. If this field is not present in config, environment variable "AWS_REGION" can then be used to set region. | determined by metadata |
| `role_arn` | IAM role to upload segments to a different account. | |
| `external_id` | Shared identitier used when assuming an IAM role in an external AWS account. | |
| `max_retries` | Maximum number of retries before abandoning an attempt to post data. | 1 |
| `dimension_rollup_option` | DimensionRollupOption is the option for metrics dimension rollup. Three options are available: `NoDimensionRollup`, `SingleDimensionRollupOnly` and `ZeroAndSingleDimensionRollup`. The default value is `ZeroAndSingleDimensionRollup`. Enabling feature gate `awsemf.nodimrollupdefault` will set default to `NoDimensionRollup`. |"ZeroAndSingleDimensionRollup" (Enable both zero dimension rollup and single dimension rollup)|
| `resource_to_telemetry_conversion` | "resource_to_telemetry_conversion" is the option for converting resource attributes to telemetry attributes. It has only one config option- `enabled`. For metrics, if `enabled=true`, all the resource attributes will be converted to metric labels by default. See `Resource Attributes to Metric Labels` section below for examples. | `enabled=false` |
| `output_destination` | "output_destination" is an option to specify the EMFExporter output. Currently, two options are available. "cloudwatch" or "stdout" | `cloudwatch` |
| `detailed_metrics` | Retain detailed datapoint values in exported metrics (e.g instead of exporting a quantile as a statistical value, preserve the quantile's population) | `false` |
| `parse_json_encoded_attr_values` | List of attribute keys whose corresponding values are JSON-encoded strings and will be converted to JSON structures in emf logs. For example, the attribute string value "{\\"x\\":5,\\"y\\":6}" will be converted to a json object: ```{"x": 5, "y": 6}``` | [ ] |
| [`metric_declarations`](#metric_declaration) | List of rules for filtering exported metrics and their dimensions. | [ ] |
| [`metric_descriptors`](#metric_descriptor) | List of rules for inserting or updating metric descriptors. | [ ] |
| `retain_initial_value_of_delta_metric` | This option specifies how the first value of a metric is handled. AWS EMF expects metric values to only contain deltas to the previous value. In the default case the first received value is therefor not sent to AWS but only used as a baseline for follow up changes to this metric. This is fine for high throughput metrics with stable labels (e.g. `requests{code=200}`). In this case it does not matter if the first value of this metric is discarded. However when your metric describes infrequent events or events with high label cardinality, then the exporter in default configuration would still drop the first occurrence of this metric. With this configuration value set to `true` the first value of all metrics will instead be send to AWS. | false |
### metric_declaration
A metric_declaration section characterizes a rule to be used to set dimensions for exported metrics, filtered by the incoming metrics' labels and metric names.
| Name | Description | Default |
| :---------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ------- |
| `dimensions` | List of dimension sets to be exported. Dimension sets that include dimensions that are not labels are ignored. Use empty dimension set `[]` for metrics without labels. | [[ ]] |
| `metric_name_selectors` | List of regex strings to filter metric names by. | |
| [`label_matchers`](#label_matcher) | (Optional) list of label matching rules to filter metrics by their labels. This rule is applied to any metric that matches any of the label matchers. | [ ] |
#### label_matcher
A label_matcher section defines a matching rule against the labels of the incoming metric. Only metrics that match the rules will be used by the surrounding `metric_declaration`.
| Name | Description | Default |
| :---------------- | :--------------------------------------------------------------------- | ------- |
| `label_names` | List of label names to filter by. Their corresponding values are concatenated using the separator and matched against the configured regular expression. | |
| `separator` | (Optional) separator placed between concatenated label values. | ";" |
| `regex` | Regex string to be matched against concatenated label values. | |
### metric_descriptor
A metric descriptor section allows the schema of a metric to be overwritten before sending out to the CloudWatch backend service. Currently, we only support unit override.
| Name | Description | Default |
| :---------------- | :--------------------------------------------------------------------- | ------- |
| `metric_name` | The name of the metric to be overwritten. | |
| `unit` | The overwritten value of unit. The [MetricDatum](https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_MetricDatum.html) contains a full list of supported unit values. | |
| `overwrite` | `true` if the schema should be overwritten with the given specification, otherwise it will only be configured if empty. | false |
## AWS Credential Configuration
This exporter follows default credential resolution for the
[aws-sdk-go](https://docs.aws.amazon.com/sdk-for-go/api/index.html).
Follow the [guidelines](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html) for the
credential configuration.
## Metric Attributes
By setting attributes on your metrics you can change how individual metrics are sent to CloudWatch. Attributes can be set in code or using components like the [Attribute Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor).
The AWS EMF Exporter will interpret the following metric attributes to change how it publishes metrics to CloudWatch:
| Attribute Name | Description | Default |
| :---------------- | :--------------------------------------------------------------------- | ------- |
| `aws.emf.storage_resolution` | This attribute should be set to an integer value of `1` or `60`. When sending the metric value to CloudWatch use the specified storage resolution value. CloudWatch currently supports a storage resolution of `1` or `60` to indicate 1 second or 60 second resolution. | `aws.emf.storage_resolution = 60` |
## Configuration Examples
### Resource Attributes to Metric Labels
The `resource_to_telemetry_conversion` option can be used to convert resource attributes to metric labels. By default, this option is disabled.
You can specify `included` and `excluded` wildcard pattern lists to match attribute keys.
Legacy options `enabled` and `exclude_service_attributes` are **deprecated** (and can be disabled via the `exporter.awsemf.DisableLegacyResourceToTelemetryConversion` feature gate). When migrating from `enabled: true`, the equivalent configuration is setting `included: ["*"]`. See the config example below:
```yaml
exporters:
awsemf:
region: 'us-west-2'
resource_to_telemetry_conversion:
included:
- "*"
```
### Metric Declaration
The following is an example of how to use `metric_declaration` to select what metrics should be exported.
```yaml
exporters:
awsemf:
region: 'us-west-2'
output_destination: stdout
dimension_rollup_option: "NoDimensionRollup"
metric_declarations:
- dimensions: [[]]
metric_name_selectors:
# Metric without label
- "^node_load15$"
- dimensions: [[device, fstype], []]
metric_name_selectors:
- "^node_filesystem_readonly$"
```
---
### Overview (`exporter/awskinesisexporter/README.md`)
# AWS Kinesis Exporter
| Status | |
| ------------- |-----------|
| Stability | [beta]: traces, metrics, logs |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fawskinesis) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fawskinesis) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_awskinesis&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@Aneurysm9](https://www.github.com/Aneurysm9), [@MovieStoreGuy](https://www.github.com/MovieStoreGuy) |
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
The kinesis exporter currently exports dynamic encodings to the configured kinesis stream.
The exporter relies heavily on the kinesis.PutRecords api to reduce network I/O and reduces records into smallest atomic representation
to avoid hitting the hard limits placed on Records (No greater than 1Mb).
This producer will block until the operation is done to allow for retryable and queued data to help during high loads.
The following settings are required:
- `aws`
- `stream_name` (no default): The name of the Kinesis stream to export to.
The following settings can be optionally configured:
- `aws`
- `kinesis_endpoint` (no default)
- `region` (default = us-west-2): the region that the kinesis stream is deployed in
- `role` (no default): The role to be used in order to send data to the kinesis stream
- `encoding`
- `name` (default = otlp): defines the export type to be used to send to kinesis (available is `otlp_proto`, `otlp_json`, `zipkin_proto`, `zipkin_json`, `jaeger_proto`)
- **Note** : `otlp_json` is considered experimental and _should not_ be used for production environments.
- `compression` (default = none): allows to set the compression type (defaults BestSpeed for all) before forwarding to kinesis (available is `flate`, `gzip`, `zlib` or `none`)
- `max_records_per_batch` (default = 500, PutRecords limit): The number of records that can be batched together then sent to kinesis.
- `max_record_size` (default = 1Mb, PutRecord(s) limit on record size): The max allowed size that can be exported to kinesis
- `timeout` (default = 5s): Is the timeout for every attempt to send data to the backend.
- `retry_on_failure`
- `enabled` (default = true)
- `initial_interval` (default = 5s): Time to wait after the first failure before retrying; ignored if `enabled` is `false`
- `max_interval` (default = 30s): Is the upper bound on backoff; ignored if `enabled` is `false`
- `max_elapsed_time` (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if `enabled` is `false`
- `sending_queue`
- `enabled` (default = true)
- `num_consumers` (default = 10): Number of consumers that dequeue batches; ignored if `enabled` is `false`
- `queue_size` (default = 1000): Maximum number of batches kept in memory before dropping data; ignored if `enabled` is `false`;
User should calculate this as `num_seconds * requests_per_second` where:
- `num_seconds` is the number of seconds to buffer in case of a backend outage
- `requests_per_second` is the average number of requests per seconds.
Example Configuration:
```yaml
exporters:
awskinesis:
aws:
stream_name: raw-trace-stream
region: us-east-1
role: arn:test-role
```
---
### Overview (`exporter/awss3exporter/README.md`)
# AWS S3 Exporter
| Status | |
| ------------- |-----------|
| Stability | [alpha]: traces, metrics, logs |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fawss3) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fawss3) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_awss3&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@atoulme](https://www.github.com/atoulme), [@Erog38](https://www.github.com/Erog38) |
| Emeritus | [@pdelewski](https://www.github.com/pdelewski) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
## Schema supported
This exporter targets to support proto/json format.
## Exporter Configuration
The following exporter configuration parameters are supported.
| Name | Description | Default |
|:--------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------|
| `region` | AWS region. | "us-east-1" |
| `s3_bucket` | S3 bucket | |
| `s3_base_prefix` | root prefix for the S3 key applied to all files. | |
| `s3_prefix` | prefix for the S3 key that can be overridden dynamically by `resource_attrs_to_s3` parameter. | |
| `s3_partition_format` | filepath formatting for the partition; See [strftime](https://www.man7.org/linux/man-pages/man3/strftime.3.html) for format specification. | "year=%Y/month=%m/day=%d/hour=%H/minute=%M" |
| `s3_partition_timezone` | timezone used to format partition | Local |
| `role_arn` | the Role ARN to be assumed | |
| `access_key` | AWS access key ID for static credentials. Must be set together with `secret_key` and cannot be combined with `role_arn`. See [Static credentials](#static-credentials). | |
| `secret_key` | AWS secret access key for static credentials. Must be set together with `access_key` and cannot be combined with `role_arn`. See [Static credentials](#static-credentials). | |
| `file_prefix` | file prefix defined by user | |
| `marshaler` | marshaler used to produce output data | `otlp_json` |
| `encoding` | Encoding extension to use to marshal data. Overrides the `marshaler` configuration option if set. | |
| `encoding_file_extension` | file format extension suffix when using the `encoding` configuration option. May be left empty for no suffix to be appended. | |
| `endpoint` | (REST API endpoint) overrides the endpoint used by the exporter instead of constructing it from `region` and `s3_bucket` | |
| `storage_class` | [S3 storageclass](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-class-intro.html) | STANDARD |
| `acl` | [S3 Object Canned ACL](https://docs.aws.amazon.com/AmazonS3/latest/userguide/acl-overview.html#canned-acl) | none (does not set by default) |
| `s3_force_path_style` | [set this to `true` to force the request to use path-style addressing](http://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html) | false |
| `disable_ssl` | set this to `true` to disable SSL when sending requests | false |
| `compression` | should the file be compressed | none |
| `sending_queue` | [exporters common queuing](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md) | disabled |
| `timeout` | [exporters common timeout](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md) | 5s |
| `resource_attrs_to_s3` | determines the mapping of S3 configuration values to resource attribute values for uploading operations. | |
| `retry_mode` | The retryer implementation, the supported values are "standard", "adaptive" and "nop". "nop" will set the retryer as `aws.NopRetryer`, which effectively disable the retry. | standard |
| `retry_max_attempts` | The max number of attempts for retrying a request if the `retry_mode` is set. Setting max attempts to 0 will allow the SDK to retry all retryable errors until the request succeeds, or a non-retryable error is returned. | 3 |
| `retry_max_backoff` | the max backoff delay that can occur before retrying a request if `retry_mode` is set | 20s |
| `unique_key_func_name` | Name of the function to use for generating a unique portion of the key name, defaults to a random integer. Only supported value is `uuidv7`. | |
| `retry_on_failure` | see [Retry on Failure](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper#retry-on-failure) for the full set of available options. | |
### Marshaler
Marshaler determines the format of data sent to AWS S3. Currently, the following marshalers are implemented:
- `otlp_json` (default): the [OpenTelemetry Protocol format](https://github.com/open-telemetry/opentelemetry-proto), represented as json.
- `otlp_proto`: the [OpenTelemetry Protocol format](https://github.com/open-telemetry/opentelemetry-proto), represented as Protocol Buffers. A single protobuf message is written into each object.
- `sumo_ic`: the [Sumo Logic Installed Collector Archive format](https://help.sumologic.com/docs/manage/data-archiving/archive/).
- _sourceCategory, _sourceHost, and _sourceName is needed
```yaml
resource/add_source_category:
attributes:
- action: insert
key: _sourceCategory
value: "value"
- action: insert
key: _sourceHost
value: "value"
- action: insert
key: _sourceName
value: "value"
```
**This format is supported only for logs.**
- `body`: export the log body as string.
**This format is supported only for logs.**
### Encoding
Encoding overrides marshaler if present and sets to use an encoding extension defined in the collector configuration.
See https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding.
### Compression
- `none` (default): No compression will be applied
- `gzip`: Files will be compressed with gzip.
- `zstd`: Files will be compressed with zstd.
### resource_attrs_to_s3
- `s3_bucket`: Defines which resource attribute's value should be used as the S3 bucket.
When this option is set, it dynamically overrides `s3uploader/s3_bucket`.
If the specified resource attribute exists in the data,
its value will be used as the bucket; otherwise, `s3uploader/s3_bucket` will serve as the fallback.
- `s3_prefix`: Defines which resource attribute's value should be used as the S3 prefix.
When this option is set, it dynamically overrides `s3uploader/s3_prefix`.
If the specified resource attribute exists in the data,
its value will be used as the prefix; otherwise, `s3uploader/s3_prefix` will serve as the fallback.
# Example Configurations
Following example configuration defines to store output in 'eu-central' region and bucket named 'databucket'.
```yaml
exporters:
awss3:
s3uploader:
region: 'eu-central-1'
s3_bucket: 'databucket'
s3_prefix: 'metric'
# Optional (disabled by default)
sending_queue:
enabled: true
num_consumers: 10
queue_size: 100
# Optional (5s by default)
timeout: 20s
```
Logs and traces will be stored inside 'databucket' in the following path format.
```console
metric/year=YYYY/month=MM/day=DD/hour=HH/minute=mm
```
## Partition Formatting
By setting the `s3_partition_format` option, users can specify the file path for their logs.
See the [strftime](https://www.man7.org/linux/man-pages/man3/strftime.3.html) reference for more formatting options.
```yaml
exporters:
awss3:
s3uploader:
region: 'eu-central-1'
s3_bucket: 'databucket'
s3_prefix: 'metric'
s3_partition_format: '%Y/%m/%d/%H/%M'
```
In this case, logs and traces would be stored in the following path format.
```console
metric/YYYY/MM/DD/HH/mm
```
Optionally along with `s3_partition_format` you can provide `s3_partition_timezone` as name from IANA Time Zone
database to change default local timezone to custom, for example `UTC` or `Europe/London`.
## Base Path Configuration
The `s3_base_prefix` option allows you to specify a root path inside the bucket that is not overridden by `resource_attrs_to_s3`. If provided, `s3_prefix` will be appended to this base path.
```yaml
exporters:
awss3:
s3uploader:
region: 'eu-central-1'
s3_bucket: 'databucket'
s3_base_prefix: 'environment/prod'
s3_prefix: 'metric'
s3_partition_format: '%Y/%m/%d/%H/%M'
```
In this case, logs and traces would be stored in the following path format.
```console
environment/prod/metric/YYYY/MM/DD/HH/mm
```
## Data routing based on resource attributes
When `resource_attrs_to_s3/s3_bucket` or `resource_attrs_to_s3/s3_prefix` is configured, the S3 bucket and/or prefix are dynamically derived from specified resource attributes in your data.
If the attribute values are unavailable, the bucket and prefix will fall back to the values defined in `s3uploader/s3_bucket` and `s3uploader/s3_prefix` respectively.
```yaml
exporters:
awss3:
s3uploader:
region: 'eu-central-1'
s3_bucket: 'databucket'
s3_prefix: 'metric'
s3_partition_format: '%Y/%m/%d/%H/%M'
resource_attrs_to_s3:
s3_bucket: "com.awss3.bucket"
s3_prefix: "com.awss3.prefix"
```
In this case, metrics, logs and traces would be stored in the following path format examples:
```console
bucket1/prefix1/YYYY/MM/DD/HH/mm
bucket2/foo-prefix/YYYY/MM/DD/HH/mm
bucket3/prefix-bar/YYYY/MM/DD/HH/mm
databucket/metric/YYYY/MM/DD/HH/mm
...
```
## Base Path with Resource Attributes
When using both `s3_base_prefix` and `resource_attrs_to_s3/s3_prefix`, the `s3_base_prefix` is always used while `s3_prefix` can be dynamically overridden by resource attributes.
```yaml
exporters:
awss3:
s3uploader:
region: 'eu-central-1'
s3_bucket: 'databucket'
s3_base_prefix: 'environment/prod'
s3_prefix: 'default-metric'
s3_partition_format: '%Y/%m/%d/%H/%M'
resource_attrs_to_s3:
s3_prefix: "com.awss3.prefix"
```
In this configuration:
- **Base Prefix**: `environment/prod` (always included)
- **Prefix**: Dynamically set from resource attribute `com.awss3.prefix` if available, otherwise falls back to `default-metric`
**Path format examples:**
```console
# When resource attribute com.awss3.prefix = "service-a/metrics"
environment/prod/service-a/metrics/YYYY/MM/DD/HH/mm
# When resource attribute com.awss3.prefix = "service-b/logs"
environment/prod/service-b/logs/YYYY/MM/DD/HH/mm
# When resource attribute is unavailable (fallback)
environment/prod/default-metric/YYYY/MM/DD/HH/mm
```
This allows you to maintain consistent organizational structure (via base path) while dynamically routing different data types or services to specific subdirectories.
## Retry
Standard is the default retryer implementation used by service clients. See the [retry](https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/retry) package documentation for details on what errors are considered as retryable by the standard retryer implementation.
See also the [aws-sdk-go reference](https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/configure-retries-timeouts.html) for more information.
```yaml
exporters:
awss3:
s3uploader:
region: 'eu-central-1'
s3_bucket: 'databucket'
s3_prefix: 'metric'
retry_mode: "standard"
retry_max_attempts: 5
retry_max_backoff: "30s"
```
## AWS Credential Configuration
This exporter follows default credential resolution for the
[aws-sdk-go](https://docs.aws.amazon.com/sdk-for-go/api/index.html).
Follow the [guidelines](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html) for the
credential configuration.
### Static credentials
Static credentials can be set with `access_key` and `secret_key`. When set, they take precedence over the default
credential resolution. Both must be set together and cannot be combined with `role_arn`. Prefer referencing
environment variables over writing secrets inline:
```yaml
exporters:
awss3:
s3uploader:
region: "us-east-1"
s3_bucket: "my-bucket"
access_key: ${env:AWS_ACCESS_KEY_ID}
secret_key: ${env:AWS_SECRET_ACCESS_KEY}
```
Avoid combining static credentials with `disable_ssl: true`: requests are still signed, but the exported data is sent
over plain HTTP.
### OpenTelemetry Collector Helm Chart for Kubernetes
For example, when using OpenTelemetry Collector Helm Chart you could use `extraEnvs` in the values.yaml.
```yaml
extraEnvs:
- name: AWS_ACCESS_KEY_ID
value: "< YOUR AWS ACCESS KEY >"
- name: AWS_SECRET_ACCESS_KEY
value: "< YOUR AWS SECRET ACCESS KEY >"
```
---
### Overview (`exporter/awsxrayexporter/README.md`)
# AWS X-Ray Tracing Exporter
| Status | |
| ------------- |-----------|
| Stability | [beta]: traces |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fawsxray) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fawsxray) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_awsxray&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@wangzlei](https://www.github.com/wangzlei), [@srprash](https://www.github.com/srprash) |
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
This exporter converts OpenTelemetry spans to
[AWS X-Ray Segment Documents](https://docs.aws.amazon.com/xray/latest/devguide/xray-api-segmentdocuments.html)
and then sends them directly to X-Ray using the
[PutTraceSegments](https://docs.aws.amazon.com/xray/latest/api/API_PutTraceSegments.html) API.
## Data Conversion
Trace IDs and Span IDs are expected to be originally generated by either AWS API Gateway or AWS ALB and
propagated by them using the `X-Amzn-Trace-Id` HTTP header. However, other generation sources are
supported by replacing fully-random Trace IDs with X-Ray formatted Trace IDs where necessary:
> AWS X-Ray IDs in binary are 128 bits, the same size as W3C Trace Context IDs but the string is formatted to
> “1-{8 digit hex}-{24 digit hex}“. For example, The W3C format trace ID “4bf92f3577b34da6a3ce929d0e0e4736” is
> converted to the X-Ray format trace ID “1-4bf92f35-77b34da6a3ce929d0e0e4736".
The `http` object is populated when the `component` attribute value is `grpc` as well as `http`. Other
synchronous call types should also result in the `http` object being populated.
## AWS Specific Attributes
The following AWS-specific Span attributes are supported in addition to the standard names and values
defined in the OpenTelemetry Semantic Conventions.
| Attribute name | Notes and examples | Required? |
| :--------------- | :--------------------------------------------------------------------- | --------- |
| `aws.operation` | The name of the API action invoked against an AWS service or resource. | No |
| `aws.account_id` | The AWS account number if accessing resource in different account. | No |
| `aws.region` | The AWS region if accessing resource in different region from app. | No |
| `aws.request_id` | AWS-generated unique identifier for the request. | No |
| `aws.queue_url` | For operations on an Amazon SQS queue, the queue's URL. | No |
| `aws.table_name` | For operations on a DynamoDB table, the name of the table. | No |
| `aws.xray.annotations` | The attribute is a slice(list) attribute that contains each of the string keys. If found on the span, the `awsxrayexporter` will use them in addition to the `indexed_attributes` configuration field when categorizing which attributes to index. This can be configured with `"aws.xray.annotations"=["key1", "key2"]` (Java example: `span.setAttribute(stringArrayKey("aws.xray.annotations"), List.of("key1", "key2"))`) | No |
Any of these values supplied are used to populate the `aws` object in addition to any relevant data supplied
by the Span Resource object. X-Ray uses this data to generate inferred segments for the remote APIs.
## Exporter Configuration
The following exporter configuration parameters are supported. They mirror and have the same effect as the
comparable AWS X-Ray Daemon configuration values.
| Name | Description | Default |
|:-----------------------------|:-------------------------------------------------------------------------------------------------------------------| ------- |
| `num_workers` | Maximum number of concurrent calls to AWS X-Ray to upload documents. | 8 |
| `endpoint` | Optionally override the default X-Ray service endpoint. | |
| `request_timeout_seconds` | Number of seconds before timing out a request. | 30 |
| `max_retries` | Maximun number of attempts to post a batch before failing. | 2 |
| `no_verify_ssl` | Enable or disable TLS certificate verification. | false |
| `proxy_address` | Upload segments to AWS X-Ray through a proxy. | |
| `region` | Send segments to AWS X-Ray service in a specific region. | |
| `local_mode` | Local mode to skip EC2 instance metadata check. | false |
| `resource_arn` | Amazon Resource Name (ARN) of the AWS resource running the collector. | |
| `role_arn` | IAM role to upload segments to a different account. | |
| `external_id` | Shared identitier used when assuming an IAM role in an external AWS account. | |
| `indexed_attributes` | List of attribute names to be converted to X-Ray annotations. | |
| `index_all_attributes` | Enable or disable conversion of all OpenTelemetry attributes to X-Ray annotations. | false |
| `aws_log_groups` | List of log group names for CloudWatch. | [] |
| `telemetry.enabled` | Whether telemetry collection is enabled at all. | false |
| `telemetry.include_metadata` | Whether to include metadata in the telemetry (InstanceID, Hostname, ResourceARN) | false |
| `telemetry.contributors` | List of X-Ray component IDs contributing to the telemetry (ex. for multiple X-Ray receivers: awsxray/1, awsxray/2) | |
| `telemetry.hostname` | Sets the Hostname included in the telemetry. | |
| `telemetry.instance_id` | Sets the InstanceID included in the telemetry. | |
| `telemetry.resource_arn` | Sets the Amazon Resource Name (ARN) included in the telemetry. | |
## Traces and logs correlation
AWS X-Ray can be integrated with CloudWatch Logs to correlate traces with logs. For this integration to work, the X-Ray
segments must have the AWS Property `cloudwatch_logs` set. This property is set using the AWS X-Ray exporter with the
following values that are evaluated in this order:
1. `aws.log.group.arns` resource attribute.
2. `aws.log.group.names` resource attribute.
3. `aws_log_groups` configuration property.
In the case of multiple values are defined, the value with higher precedence will be used to set the `cloudwatch_logs` AWS Property.
`aws.log.group.arns` and `aws.log.group.names` are slice resource attributes that can be set programmatically.
Alternatively those resource attributes can be set using the [`OTEL_RESOURCE_ATTRIBUTES` environment variable](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/resource/sdk.md#specifying-resource-information-via-an-environment-variable). To set multiple log group names /log group arns, you can use `&`
to separate them. For example, 3 log groups `log-group1`, `log-group2`, and `log-group3` are set in the following command:
```
export OTEL_RESOURCE_ATTRIBUTES="aws.log.group.names=log-group1&log-group2&log-group3"
```
## AWS Credential Configuration
This exporter follows default credential resolution for the
[aws-sdk-go](https://docs.aws.amazon.com/sdk-for-go/api/index.html).
Follow the [guidelines](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html) for the
credential configuration.
[beta]:https://github.com/open-telemetry/opentelemetry-collector#beta
[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[AWS]:https://aws-otel.github.io/docs/getting-started/x-ray#configuring-the-aws-x-ray-exporter
---
### Overview (`exporter/azureblobexporter/README.md`)
# Azure Blob Storage Exporter
| Status | |
| ------------- |-----------|
| Stability | [alpha]: traces, metrics, logs |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fazureblob) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fazureblob) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_azureblob&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@hgaol](https://www.github.com/hgaol), [@MovieStoreGuy](https://www.github.com/MovieStoreGuy) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
## Configuration
The following settings are required:
- url: Must be specified if auth type is not connection_string. If auth type is connection_string, it's optional or will be override by the auth.connection_string. Azure storage account endpoint. This setting might be replaced with `endpoint` for future. e.g. https://.blob.core.windows.net/
- auth (no default): Authentication method for exporter to ingest data.
- type (no default): Authentication type for exporter. supported values are: connection_string, service_principal, system_managed_identity, user_managed_identity and workload_identity.
- tenand_id: Tenand Id for the client, only needed when type is service_principal and workload_identity.
- client_id: Client Id for the auth, only needed when type is service_principal, user_managed_identity and workload_identity.
- client_secret: Secret for the client, only needed when type is service_principal.
- connection_string: Connection string to the endpoint. Only needed for connection_string auth type. Once provided, it'll **override** the `url` parameter to the storage account.
- federated_token_file: The path of the projected service account token file, only needed when type is workload_identity.
The following settings can be optionally configured and have default values:
- container: container for metrics, logs and traces. A container organizes a set of blobs, similar to a directory in a file system. More details can refer [this](https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers).
- metrics (default `metrics`): container to store metrics. default value is `metrics`.
- logs (default `logs`): container to store logs. default value is `logs`.
- traces (default `traces`): container to store traces. default value is `traces`.
- blob_name_format: the final blob name will be blob_name
- template_enabled (default `false`): enables Go template parsing for blob name formats. If parsing fails, it will not throw an error but will log a warning and continue formatting the blob name using other rules.
- metrics_format (default `2006/01/02/metrics_15_04_05.json`): blob name format. The date format follows constants in Golang, refer [here](https://go.dev/src/time/format.go).
- logs_format (default `2006/01/02/logs_15_04_05.json`): blob name format.
- traces_format (default `2006/01/02/traces_15_04_05.json`): blob name format.
- timezone (default `""`): Timezone for blob name formatting. Local time is used if empty. Must be a valid IANA timezone identifier accepted by Golang's [`time.LoadLocation`](https://pkg.go.dev/time#LoadLocation), such as `UTC` or `America/New_York`.
- serial_num_enabled (default `true`): toggles whether a random serial number is appended to the blob name.
- serial_num_range (default `10000`): a range of random number to be appended after blob_name. e.g. `blob_name_{serial_num}`. the number will be in `[0, serial_num_range)`.
- serial_num_before_extension (default `false`): places the serial number before the file extension if there is one. e.g `blob_name_{serial_num}.json` instead of `blob_name.json_{serial_num}`
- time_parser_enabled (default `true`): controls whether the exporter interprets the format string as a Go time layout. When `false`, values such as `2006` or `15_04_05` are treated as literal text.
- time_parser_ranges (default `nil`): limits time formatting to specific parts of the blob name. When not set (nil), the entire blob name is time-formatted if `time_parser_enabled` is `true`. Provide a list of character ranges like `["0-10", "15-25"]` to only apply time formatting within those positions. For example, if your blob name is `prefix/2006/01/02/file.json` and you set `time_parser_ranges: ["7-17"]`, only the `2006/01/02` portion will be replaced with actual date values, while `prefix/` and `/file.json` remain unchanged. This is helpful when your blob name contains patterns like `2006` that you want to keep as literal text.
- compression (default `""`): sets the algorithm used to process the payload before uploading to Azure Blob Storage. Valid values are `gzip`, `zstd`, or no value set (uncompressed). The appropriate extension (`.gz` or `.zst`) is automatically appended to the blob name.
- format (default `json`): `json` or `proto`. which present otel json or otel protobuf format, the file extension will be `json` or `pb`.
- encodings (default using encoding specified in `format`, which is `json`): if specified, uses the encoding extension to encode telemetry data. Overrides format.
- logs (default `nil`): encoding component id.
- metrics (default `nil`): encoding component id.
- traces (default `nil`): encoding component id.
- append_blob: configures append blob behavior. When enabled, telemetry data is appended to a single blob instead of creating new blobs. This can be useful for aggregating data or reducing the number of blobs created.
- enabled (default `false`): determines whether to use append blob mode.
- separator (default `\n`): string to insert between appended data blocks.
- `retry_on_failure`
- `enabled` (default = true)
- `initial_interval` (default = 5s): Time to wait after the first failure before retrying; ignored if `enabled` is `false`
- `max_interval` (default = 30s): Is the upper bound on backoff; ignored if `enabled` is `false`
- `max_elapsed_time` (default = 120s): Is the maximum amount of time spent trying to send a batch; ignored if `enabled` is `false`
### Blob Name Templates
When `template_enabled` is `true`, you can use Go templates in `metrics_format`, `logs_format`, and `traces_format` to create dynamic blob names based on telemetry data. The root object for the template is the telemetry data itself (`pmetric.Metrics`, `plog.Logs`, or `ptrace.Traces`).
The following template functions are available:
| Function | Description | Example |
| ----------------------- | --------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| `getResourceMetricAttr` | Gets a resource attribute from metrics data. | `{{ getResourceMetricAttr . 0 "service.name" }}` |
| `getResourceLogAttr` | Gets a resource attribute from logs data. | `{{ getResourceLogAttr . 0 "service.name" }}` |
| `getResourceSpanAttr` | Gets a resource attribute from traces data. | `{{ getResourceSpanAttr . 0 "service.name" }}` |
| `getScopeMetricAttr` | Gets a scope attribute from metrics data. | `{{ getScopeMetricAttr . 0 0 "scope.name" }}` |
| `getScopeLogAttr` | Gets a scope attribute from logs data. | `{{ getScopeLogAttr . 0 0 "scope.name" }}` |
| `getScopeSpanAttr` | Gets a scope attribute from traces data. | `{{ getScopeSpanAttr . 0 0 "scope.name" }}` |
| `getMetric` | Gets a metric object. You can chain to access its fields. | `{{ (getMetric . 0 0 0).Name }}` |
| `getLogRecord` | Gets a log record object. You can chain to access its fields. | `{{ (getLogRecord . 0 0 0).TraceID }}` |
| `getSpan` | Gets a span object. You can chain to access its fields. | `{{ (getSpan . 0 0 0).Name }}` |
An example configuration is provided as follows:
```yaml
extensions:
zpages:
endpoint: localhost:55679
text_encoding:
encoding: utf8
marshaling_separator: "\n"
unmarshaling_separator: "\r?\n"
exporter:
azure_blob/1:
url: "https://.blob.core.windows.net/"
container:
logs: "logs"
metrics: "metrics"
traces: "traces"
blob_name_format:
template_enabled: true
metrics_format: `{{ getResourceMetricAttr . 0 "service.name" }}/2006/01/02/metrics.json`
logs_format: `{{ getScopeLogAttr . 0 0 "scope.name" }}/2006/01/02/logs.json`
traces_format: `{{ (getSpan . 0 0 0).Name }}/2006/01/02/traces.json`
serial_num_enabled: true
time_parser_enabled: true
auth:
type: "connection_string"
connection_string: "DefaultEndpointsProtocol=https;AccountName=;AccountKey=;EndpointSuffix=core.windows.net"
encodings:
logs: text_encoding
append_blob:
enabled: true
separator: "\n"
compression: gzip # or "zstd" or leave unset for no compression
```
### Append Blob
When `append_blob` is enabled:
- The exporter will create append blobs instead of block blobs
- New data will be appended to existing blobs rather than creating new ones
- The configured separator will be inserted between data blocks
- If the blob doesn't exist, it will be created automatically
---
### Overview (`exporter/azuredataexplorerexporter/README.md`)
# Azure Data Explorer Exporter
| Status | |
| ------------- |-----------|
| Stability | [unmaintained]: traces, metrics, logs |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fazuredataexplorer) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fazuredataexplorer) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_azuredataexplorer&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | |
| Emeritus | [@asaharn](https://www.github.com/asaharn), [@ag-ramachandran](https://www.github.com/ag-ramachandran) |
[unmaintained]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#unmaintained
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
This exporter sends metrics, logs and trace data to
[Azure Data Explorer](https://docs.microsoft.com/en-us/azure/data-explorer),
[Azure Synapse Data Explorer](https://docs.microsoft.com/en-us/azure/synapse-analytics/data-explorer/data-explorer-overview) and
[Real time analytics in Fabric](https://learn.microsoft.com/en-us/fabric/real-time-analytics/overview)
## Configuration
The following settings are required:
- `cluster_uri` (no default): The cluster name of the provisioned ADX cluster to ingest the data.
One authentication method is required:
- Service principal:
- `application_id` (no default): The client id to connect to the cluster and ingest data.
- `application_key` (no default): The cluster secret corresponding to the client id.
- `tenant_id` (no default): The tenant id where the application_id is referenced from.
- Managed identity:
- `managed_identity_id` (no default): The managed identity id to authenticate with. Set to "system" for system-assigned managed identity. Set the MI client ID (GUID) for user-assigned managed identity.
- Default authentication:
- `use_azure_auth` (default: false): Set to true to use the Azure [default authentication](https://learn.microsoft.com/en-us/azure/developer/go/azure-sdk-authentication?tabs=bash#2-authenticate-with-azure).
The following settings can be optionally configured and have default values:
> Note that the database tables are expected to be created upfront before the exporter is in operation , the definition of these are in the section [Database and Table definition scripts](#database-and-table-definition-scripts)
- `db_name` (default = "oteldb"): The ADX database where the tables are present to ingest the data.
- `metrics_table_name` (default = OTELMetrics): The target table in the database `db_name` that stores exported metric data.
- `logs_table_name` (default = OTELLogs): The target table in the database `db_name` that stores exported logs data.
- `traces_table_name` (default = OTELTraces): The target table in the database `db_name` that stores exported traces data.
Optionally the following table mappings can be specified if the data needs to be mapped to a different table on ADX. This uses json [table mapping](https://docs.microsoft.com/azure/data-explorer/kusto/management/mappings#json-mapping) that can be used to map [attributes](#attribute-mapping) to target tables
- `metrics_table_json_mapping` (optional, no default): The table mapping name to be used for the table `db_name`.`metrics_table_name`
- `logs_table_json_mapping` (optional, no default): The table mapping name to be used for the table `db_name`.`logs_table_name`
- `traces_table_json_mapping` (optional, no default): The table mapping name to be used for the table `db_name`.`traces_table_name`
- `ingestion_type` (possible values=`queued` / `managed`, default = queued): ADX ingest can happen in managed [streaming](https://docs.microsoft.com/azure/data-explorer/kusto/management/streamingingestionpolicy) or [queued](https://docs.microsoft.com/azure/data-explorer/kusto/management/batchingpolicy) modes.
> Note: [Streaming ingestion](https://docs.microsoft.com/azure/data-explorer/ingest-data-streaming?tabs=azure-portal%2Ccsharp) has to be enabled on ADX [configure the ADX cluster] in case of `streaming` option. Refer the query below to check if streaming is enabled
```kql
.show database policy streamingingestion
```
An example configuration is provided as follows:
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Attribute mapping
This exporter maps OpenTelemetry [trace](https://opentelemetry.io/docs/reference/specification/trace/sdk/), [metric](https://opentelemetry.io/docs/reference/specification/metrics/sdk/) and [log](https://opentelemetry.io/docs/reference/specification/logs/data-model/) attributes to specific structures in the ADX tables. These can then be extended by usage of [update policies](https://docs.microsoft.com/azure/data-explorer/kusto/management/updatepolicy) in ADX if needed
### Traces
| ADX Table column | Description / OpenTelemetry attribute |
| ----------------------------- |----------------------------------------------------------------------------------------------|
| TraceId | A valid trace identifier is a 16-byte array with at least one non-zero byte. |
| SpanId | A valid span identifier is an 8-byte array with at least one non-zero byte. |
| ParentId | A parent spanId, for the current span |
| SpanName | The span name |
| SpanStatus | [Status Code](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#set-status) of the Span. |
| SpanStatusMessage | [Status Message](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#set-status) of the Span |
| SpanKind | [SpanKind](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/trace/api.md#spankind) describes the relationship between the Span, its parents, and its children in a Trace |
| StartTime | A start timestamp |
| EndTime | An end timestamp |
| TraceAttributes | Custom metric [attributes](https://opentelemetry.io/docs/reference/specification/common/#attribute) set from the application. Also contains the [instrumentation scope](https://opentelemetry.io/docs/reference/specification/common/#attribute) name and version |
| ResourceAttributes | The resource attributes JSON map as specified in open telemetry [resource semantics](https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/) |
| Events | A list of timestamped [Events](https://opentelemetry.io/docs/reference/specification/trace/api/#add-events) |
| Links | A list of [Links](https://opentelemetry.io/docs/reference/specification/trace/api/#specifying-links) to other Spans |
### Metrics
| ADX Table column | Description / OpenTelemetry attribute
| ----------------------------- |---------------------------------------------------------------------------------------------------|
| Timestamp | The timestamp of the datapoint |
| MetricName | The name of the datapoint |
| MetricType | The type / datapoint type of the metric (e.g. Sum, Histogram, Gauge etc.) |
| MetricUnit | Unit of measure of the metric |
| MetricDescription | Description about the metric |
| MetricValue | The metric value measured for the datapoint |
| MetricAttributes | Custom metric [attributes](https://opentelemetry.io/docs/reference/specification/common/#attribute) set from the application. Also contains the [instrumentation scope](https://opentelemetry.io/docs/reference/specification/common/#attribute) name and version |
| Host | The host.name extracted from [Host resource semantics](https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/host/). If empty , the hostname of the exporter is used |
| ResourceAttributes | The resource attributes JSON map as specified in open telemetry [resource semantics](https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/) |
### Logs
| ADX Table column | Description / OpenTelemetry attribute |
| ----------------------------- |------------------------------------------------------ |
| Timestamp | The timestamp of the datapoint |
| ObservedTimestamp | Time when the event was observed. |
| TraceId | Request trace id. |
| SpanId | Request span id. |
| SeverityText | The severity text (also known as log level) |
| SeverityNumber | [Numerical value](https://opentelemetry.io/docs/reference/specification/logs/data-model/#field-severitynumber) of the severity. |
| Body | The body of the log record. |
| LogsAttributes | Custom metric [attributes](https://opentelemetry.io/docs/reference/specification/common/#attribute) set from the application. Also contains the [instrumentation scope](https://opentelemetry.io/docs/reference/specification/common/#attribute) name and version |
| ResourceAttributes | The resource attributes JSON map as specified in open telemetry [resource semantics](https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/) |
### Database and Table definition scripts
The following tables need to be created in the database specified in the configuration.
```kql
.create-merge table (Timestamp:datetime, ObservedTimestamp:datetime, TraceID:string, SpanID:string, SeverityText:string, SeverityNumber:int, Body:string, ResourceAttributes:dynamic, LogsAttributes:dynamic)
.create-merge table (Timestamp:datetime, MetricName:string, MetricType:string, MetricUnit:string, MetricDescription:string, MetricValue:real, Host:string, ResourceAttributes:dynamic,MetricAttributes:dynamic)
.create-merge table (TraceID:string, SpanID:string, ParentID:string, SpanName:string, SpanStatus:string, SpanKind:string, StartTime:datetime, EndTime:datetime, ResourceAttributes:dynamic, TraceAttributes:dynamic, Events:dynamic, Links:dynamic)
// This is an optional column to store the status code and message as a dynamic field. This augments the status field with Status code and status message
.alter-merge table (SpanStatusMessage:string)
//Enable streaming ingestion( for managed streaming) for the created tables using
.alter table policy streamingingestion enable
```
### Optional configurations/Enhancements suggestions
- Using update policies , the collected data can further be processed as per application need. The following is an example where histogram metrics are exported to a histo specific table (buckets and aggregates)
```
/* Detailed source-code truncated for AI context efficiency. */
```
### OpenTelemetry Exporter Helper Configurations
The ADX exporter now includes support for OpenTelemetry exporter helper configurations. This feature allows you to leverage the exporter helper capabilities(retries, timeout etc.) provided natively by Otel. Read more about the [exporterhelper](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md).
Please note that this configuration is not enabled by default. To utilize the OpenTelemetry exporter helper, you will need to add it manually to the configuration.
#### Example Configuration
```yaml
# Example OpenTelemetry Exporter Configuration
timeout: 10s
sending_queue:
enabled: true
num_consumers: 2
queue_size: 10
retry_on_failure:
enabled: true
initial_interval: 10s
max_interval: 60s
max_elapsed_time: 10m
---
### Overview (`exporter/azuremonitorexporter/README.md`)
# Azure Monitor Exporter
| Status | |
| ------------- |-----------|
| Stability | [beta]: traces, metrics, logs |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fazuremonitor) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fazuremonitor) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_azuremonitor&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@hgaol](https://www.github.com/hgaol) |
| Emeritus | [@pcwiese](https://www.github.com/pcwiese) |
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
This exporter sends logs, traces and metrics to [Azure Monitor](https://docs.microsoft.com/azure/azure-monitor/).
## Configuration
To configure the Azure Monitor Exporter, you must specify one of the following settings:
- `connection_string` (recommended): The Azure Application Insights Connection String is required to send telemetry data to the monitoring service. It is the recommended method for configuring the exporter, aligning with Azure Monitor's best practices. If you need guidance on creating Azure resources, please refer to the step-by-step guides to [Create an Application Insights resource](https://docs.microsoft.com/azure/azure-monitor/app/create-new-resource) and [find your connection string](https://docs.microsoft.com/azure/azure-monitor/app/sdk-connection-string?tabs=net#find-your-connection-string).
- `instrumentation_key`: Application Insights instrumentation key, which can be found in the Application Insights resource in the Azure Portal. While it is currently supported, its use is discouraged and it is slated for deprecation. It is highly encouraged to use the `connection_string` setting for new configurations and migrate existing configurations to use the `connection_string` as soon as possible.
### Environment Variable Support
In addition to the above configuration options, the Azure Monitor Exporter now supports setting the connection string via the `APPLICATIONINSIGHTS_CONNECTION_STRING` environment variable. This method is particularly useful for cloud or containerized environments where managing configuration through environment variables is standard practice.
**Note:** If both the environment variable and the `connection_string` configuration option are provided, the environment variable takes precedence.
### Configuration Options
**Important**: Only one of `connection_string` or `instrumentation_key` should be specified in your configuration. If both are provided, `connection_string` will be used as the priority setting.
The following settings can be optionally configured:
- `endpoint` (default = `https://dc.services.visualstudio.com/v2/track`): The endpoint URL where data will be submitted. While this option remains available, it is important to note that the use of the `connection_string` is recommended, as it encompasses the endpoint information. The direct configuration of the `endpoint` is considered to be on a deprecation path.
- `maxbatchsize` (default = 1024): The maximum number of telemetry items that can be submitted in each request. If this many items are buffered, the buffer will be flushed before `maxbatchinterval` expires.
- `maxbatchinterval` (default = 10s): The maximum time to wait before sending a batch of telemetry.
- `spaneventsenabled` (default = false): Enables export of span events.
- `sending_queue`
- `enabled` (default = false)
- `num_consumers` (default = 10): Number of consumers that dequeue batches; ignored if `enabled` is `false`
- `queue_size` (default = 1000): Maximum number of batches kept in memory before data; ignored if `enabled` is `false`
- `storage` (default = `none`): When set, enables persistence and uses the component specified as a storage extension for the persistent queue
- `shutdown_timeout` (default = 1s): Timeout to wait for graceful shutdown. Once exceeded, the component will shut down forcibly, dropping any element in queue.
- `custom_events_enabled` (default = `false`): Enables export log record to custom events when there's attribute `microsoft.custom_event.name` or `APPLICATION_INSIGHTS_EVENT_MARKER_ATTRIBUTE`.
- `telemetry_mappings.traces.http.success.server_policy` (default = unset): Set to `otel` to export HTTP server spans with 4xx response codes with Application Insights request `Success=true`, matching OpenTelemetry HTTP semantic conventions. HTTP client dependencies are unaffected.
- `telemetry_mappings.traces.http.success.additional_success_status_codes` (default = `[]`): Additional HTTP status codes to export with Application Insights `Success=true` for both HTTP server requests and HTTP client dependencies.
### Tag mappings (alpha)
The optional `tag_mappings:` block overrides how selected Application Insights envelope tags are populated from OpenTelemetry resource attributes. Each field takes an ordered list of sources; the first one that resolves to a non-empty string wins.
A source entry is interpreted as:
- a **resource attribute key** when it contains a `.` (e.g. `service.instance.id`, `host.name`)
- a **string literal terminal default** when it does not contain a `.` (e.g. `unknown-instance`)
Defaults preserve the historical hardcoded behavior — when a field is unset, zero-config users see no change.
| Mapping key | Application Insights tag | Default |
| --------------------- | ----------------------------- | ------------------------ |
| `cloud_role_instance` | `ai.cloud.roleInstance` | `[service.instance.id]` |
| `application_version` | `ai.application.ver` | `[service.version]` |
> The `cloud_role_name` tag (`ai.cloud.role`) is not yet configurable; its hardcoded `service.namespace`/`service.name` concatenation continues to apply.
Example — Azure Container Apps, where `service.instance.id` is set to a random uuid and operators want the revision/replica hostname instead:
```yaml
exporters:
azure_monitor:
connection_string: "InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://ingestion.azuremonitor.com/"
tag_mappings:
cloud_role_instance: [host.name, service.instance.id, unknown-instance]
```
Validation rejects an explicitly empty array (`cloud_role_instance: []`) at startup. The feature is alpha and the schema may evolve.
Example:
```yaml
# Recommended Configuration:
# It is highly recommended to use the connection string which includes the InstrumentationKey and IngestionEndpoint
# This is the preferred method over using 'instrumentation_key' alone.
exporters:
azure_monitor:
connection_string: "InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://ingestion.azuremonitor.com/"
```
(or)
```yaml
# Legacy Configuration:
# The use of 'instrumentation_key' alone is not recommended and will be deprecated in the future. It is advised to use the connection_string instead.
# This example is provided primarily for existing configurations that have not yet transitioned to the connection string.
exporters:
azure_monitor:
instrumentation_key: b1cd0778-85fc-4677-a3fa-79d3c23e0efd
```
Example using environment variable:
Ensure `APPLICATIONINSIGHTS_CONNECTION_STRING` is set in your environment, then configure the exporter without specifying a connection string or instrumentation key:
```yaml
exporters:
azure_monitor:
```
## Attribute mapping
### Traces
This exporter maps OpenTelemetry trace data to [Application Insights data model](https://docs.microsoft.com/azure/azure-monitor/app/data-model-dependency-telemetry) using the following schema.
The OpenTelemetry SpanKind determines the Application Insights telemetry type.
| OpenTelemetry SpanKind | Application Insights telemetry type |
| -------------------------------- | ----------------------------------- |
| `CLIENT`, `PRODUCER`, `INTERNAL` | Dependency |
| `SERVER`, `CONSUMER` | Request |
The exporter follows the semantic conventions to fill the Application Insights specific telemetry properties. The following table shows a basic mapping.
| Application Insights property | OpenTelemetry attribute | Default |
| ----------------------------- | ----------------------------------------------------- | --------- |
| Request.Name | `http.method`, `http.route` or `rpc.system` | span name |
| Request.Url | `http.scheme`, `http.host`, `http.target` | |
| Request.Source | `http.client_ip` or `net.peer.name` | |
| Request.ResponseCode | `http.status_code` or `status_code` | `"0"` |
| Request.Success | `http.status_code` or `status_code` | `true` |
| Dependency.Name | `http.method`, `http.route` | span name |
| Dependency.Data | `http.url` or span name or `db.statement` | |
| Dependency.Type | `"HTTP"` or `rpc.system` or `db.system` or `"InProc"` | |
| Dependency.Target | host of `http.url` or `net.peer.name` | |
| Dependency.ResultCode | `http.status_code` or `status_code` | `"0"` |
| Dependency.Success | `http.status_code` or `status_code` | `true` |
The exact mapping can be found in [trace_to_envelope.go](trace_to_envelope.go).
All attributes are also mapped to custom properties if they are booleans or strings and to custom measurements if they are ints or doubles.
All links are mapped to property `_MS.links` with JSON array string.
#### Span Events
Span events are optionally saved to the Application Insights `traces` table.
Exception events are saved to the Application Insights `exception` table.
### Logs
This exporter saves log records to Application Insights `traces` table.
[TraceId](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-traceid) is mapped to `operation_id` column and [SpanId](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#field-spanid) is mapped to `operation_parentId` column.
#### Custom Events
When `custom_events_enabled` = `true`, azure monitor exporter will export log record to custom events when there's attribute `microsoft.custom_event.name` or `APPLICATION_INSIGHTS_EVENT_MARKER_ATTRIBUTE`.
#### Exceptions
This exporter saves exception records to Application Insights `exceptions` table when log records indicate an exception [specification](https://opentelemetry.io/docs/specs/otel/trace/exceptions/).
When `exception_events_enabled` = `true`, azure monitor exporter will export log records to exceptions when either one of `exception.message` or `exception.type` attributes are set.
### Metrics
This exporter saves metrics to Application Insights `customMetrics` table.
## AAD/Entra Authentication
Details of how to use the Azure Monitor Exporter with AAD/Entra based identities can be found in the [Authentication](AUTHENTICATION.md) page.
---
### Overview (`exporter/bmchelixexporter/README.md`)
# BMC Helix Exporter
| Status | |
| ------------- |-----------|
| Stability | [alpha]: metrics |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fbmchelix) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fbmchelix) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_bmchelix&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@bertysentry](https://www.github.com/bertysentry), [@NassimBtk](https://www.github.com/NassimBtk), [@MovieStoreGuy](https://www.github.com/MovieStoreGuy) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
> **Note:** The exporter type has been renamed from `bmchelix` to `bmc_helix` to follow the
> snake_case naming convention. The old name `bmchelix` is preserved as a deprecated alias and
> will continue to work, but a deprecation warning will be logged at startup. Please update your
> configuration to use `bmc_helix:`.
This exporter supports sending metrics to [BMC Helix Operations Management](https://docs.bmc.com/xwiki/bin/view/IT-Operations-Management/Operations-Management/BMC-Helix-Operations-Management/bhom261/Getting-started/Product-overview/) through its [metric ingestion REST API](https://docs.bmc.com/docs/helixoperationsmanagement/244/en/metric-operation-management-endpoints-in-the-rest-api-1392780044.html).
## Getting Started
The following settings are **required**:
- `endpoint`: is the *BMC Helix Portal URL* of your environment, at **onbmc.com** for a BMC Helix SaaS tenant (e.g., `https://company.onbmc.com`), or your own Helix Portal URL for an on-prem instance.
- `api_key`: API key to authenticate the exporter. Connect to BMC Helix Operations Management, go to the Administration > Repository page, and click on the Copy API Key button to get your API Key. Alternatively, it is recommended to create and use a dedicated [authentication key for external integration](https://docs.bmc.com/xwiki/bin/view/Helix-Common-Services/BMC-Helix-Portal/BMC-Helix-Portal/helixportal261/Administering/Using-API-keys-for-external-integrations/).
Example:
```yaml
exporters:
bmc_helix/helix1:
endpoint: https://company.onbmc.com
api_key:
sending_queue:
batch:
```
### Optional Settings
The following settings can be **optionally configured**:
- `timeout`: (default = `10s`) Timeout for requests made to the BMC Helix.
- `enrich_metric_with_attributes`: (default = `true`) When enabled, creates enriched metrics by appending datapoint attribute values to the metric name. This provides more detailed identification in BMC Helix Operations Management but increases metric cardinality. Set to `false` to reduce the number of unique metric series.
- `retry_on_failure` [details here](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper#configuration)
- `enabled` (default = true)
- `initial_interval` (default = 5s) Time to wait after the first failure before retrying; ignored if `enabled` is false.
- `max_interval` (default = 30s) The upper bound on backoff; ignored if `enabled` is false.
- `max_elapsed_time` (default = 300s) The maximum amount of time spent trying to send a batch; ignored if `enabled` is false. If set to 0, the retries are never stopped.
Example:
```yaml
exporters:
bmc_helix/helix2:
endpoint: https://company.onbmc.com
api_key:
timeout: 20s
enrich_metric_with_attributes: false
sending_queue:
batch:
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 1m
max_elapsed_time: 8m
```
---
## Setting Required Attributes for Metrics
To ensure metrics are correctly populated in BMC Helix, the following attributes must be set either at the *Resource* level, or at the *Metric* level:
- `entityName`: Unique identifier for the entity. Used as display name if `instanceName` is missing.
- `entityTypeId`: Type identifier for the entity.
- `instanceName`: Display name of the entity.
> **Note:** If `entityName` or `entityTypeId` is missing, the metric will not be exported.
To ensure the necessary attributes are present, it is recommended to leverage the [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) with [OTTL](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl), and include it in the configuration of the telemetry pipeline.
The minimal pipeline most often looks like: `OTEL metrics --> (memory limit) --> transform processor --> bmc_helix exporter`.
### Transformer Example for Hardware Metrics
You can use the following OpenTelemetry Transformation Language (OTTL) configuration to map these attributes dynamically:
```
/* Detailed source-code truncated for AI context efficiency. */
```
This transformer dynamically sets the attributes required for BMC Helix based on metric names and resource attributes.
---
### Overview (`exporter/cassandraexporter/README.md`)
# Cassandra Exporter
| Status | |
| ------------- |-----------|
| Stability | [alpha]: traces, logs |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fcassandra) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fcassandra) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_cassandra&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@atoulme](https://www.github.com/atoulme), [@emreyalvac](https://www.github.com/emreyalvac) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
## Configuration options
The following settings can be optionally configured:
- `dsn` The Cassandra server DSN (Data Source Name), for example `127.0.0.1`.
reference: [github.com/apache/cassandra-gocql-driver/v2](https://github.com/apache/cassandra-gocql-driver/)
- `port` (default = 9042): The Cassandra server port
- `timeout` (default = 10s): The Cassandra server connection timeout
- `keyspace` (default = otel): The keyspace name.
- `trace_table` (default = otel_spans): The table name for traces.
- `replication` (default = class: SimpleStrategy, replication_factor: 1): The strategy of
replication. https://cassandra.apache.org/doc/4.1/cassandra/architecture/dynamo.html#replication-strategy
- `compression` (default = LZ4Compressor): https://cassandra.apache.org/doc/4.0/cassandra/operating/compression.html
- `auth` (default = username: "", password: "") Authorization for the Cassandra.
## Example
```yaml
exporters:
cassandra:
dsn: 127.0.0.1
port: 9042
timeout: 10s
keyspace: "otel"
trace_table: "otel_spans"
replication:
class: "SimpleStrategy"
replication_factor: 1
compression:
algorithm: "ZstdCompressor"
auth:
username: "your-username"
password: "your-password"
```
---
### Overview (`exporter/clickhouseexporter/README.md`)
# ClickHouse Exporter
| Status | |
| ------------- |-----------|
| Stability | [development]: profiles |
| | [alpha]: metrics |
| | [beta]: traces, logs |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fclickhouse) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fclickhouse) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_clickhouse&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@Frapschen](https://www.github.com/Frapschen), [@SpencerTorres](https://www.github.com/SpencerTorres) |
| Emeritus | [@dmitryax](https://www.github.com/dmitryax), [@hanjm](https://www.github.com/hanjm) |
[development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
This exporter supports sending OpenTelemetry data to [ClickHouse](https://clickhouse.com/).
> ClickHouse is an open-source, high performance columnar OLAP database management system for real-time analytics using
> SQL.
> Throughput can be measured in rows per second or megabytes per second.
> If the data is placed in the page cache, a query that is not too complex is processed on modern hardware at a speed of
> approximately 2-10 GB/s of uncompressed data on a single server.
> If 10 bytes of columns are extracted, the speed is expected to be around 100-200 million rows per second.
Note:
**Batching Recommendation**
For optimal performance, [ClickHouse recommends](https://clickhouse.com/docs/en/introduction/performance/#performance-when-inserting-data) inserting data in large batches:
> We recommend inserting data in packets of at least 5000 rows, or no more than a single request per second. When inserting to a MergeTree table from a tab-separated dump, the insertion speed can be from 50 to 200 MB/s.
To achieve this natively, enable batching within the exporter's `sending_queue` configuration. You do not need to add the external `batch` processor to your collector pipeline. Relying on the exporter's internal batching is the recommended approach to avoid data-loss issues associated with the external processor.
Enable it by adding a `batch` block inside `sending_queue`:
```yaml
exporters:
clickhouse:
endpoint: tcp://127.0.0.1:9000
sending_queue:
# num_consumers controls how many batches are inserted into ClickHouse
# concurrently.
num_consumers: 10
batch:
min_size: 5000 # rows per INSERT (items sizer); tune to your workload
flush_timeout: 5s # flush a partial batch after this delay
```
If you are migrating from a pipeline that uses the standalone `batch` processor, remove `batch` from the pipeline's `processors` list and configure `sending_queue.batch` instead. For durability across restarts, also set `sending_queue.storage` to a [storage extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage) so queued batches survive a crash (at-least-once delivery).
## Visualization Tools
#### Official ClickHouse Plugin for Grafana
The official [ClickHouse Datasource for Grafana](https://grafana.com/grafana/plugins/grafana-clickhouse-datasource/) contains features that integrate directly with this exporter.
You can view associated [logs](https://clickhouse.com/docs/en/integrations/grafana/query-builder#logs) and [traces](https://clickhouse.com/docs/en/integrations/grafana/query-builder#traces), as well as visualize other queries such as tables and time series graphs. Learn [how to configure the OpenTelemetry integration](https://clickhouse.com/docs/en/integrations/grafana/config#opentelemetry).
#### Altinity's ClickHouse Plugin for Grafana
If the official plugin doesn't meet your needs, you can try the [Altinity plugin for ClickHouse](https://grafana.com/grafana/plugins/vertamedia-clickhouse-datasource/), which also supports a wide range of features.
### Logs
- Get log severity count time series.
```sql
SELECT toDateTime(toStartOfInterval(Timestamp, INTERVAL 60 second)) as time, SeverityText, count() as count
FROM otel_logs
WHERE toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
AND Timestamp >= NOW() - INTERVAL 1 HOUR
GROUP BY SeverityText, time
ORDER BY time;
```
The default logs table is ordered by `(toStartOfFiveMinutes(Timestamp), ServiceName, Timestamp)`.
For time range queries, filter on both `toStartOfFiveMinutes(Timestamp)` and `Timestamp`, and order by the tuple `(toStartOfFiveMinutes(Timestamp), Timestamp)` to use the primary key's read-in-order optimization.
Apply `toStartOfFiveMinutes` to the range bound as well (e.g. `toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)`) so the time bucket bounds are not truncated off the scan.
- Find any log.
```sql
SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;
```
- Find log with specific service.
```sql
SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE ServiceName = 'clickhouse-exporter'
AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;
```
- Find log with specific attribute.
```sql
SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE LogAttributes['container_name'] = '/example_flog_1'
AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;
```
- Find log with body contain string token.
```sql
SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE hasToken(Body, 'http')
AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;
```
- Find log with body contain string.
```sql
SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE Body like '%http%'
AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;
```
- Find log with body regexp match string.
```sql
SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE match(Body, 'http')
AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;
```
- Find log with body json extract.
```sql
SELECT Timestamp as log_time, Body
FROM otel_logs
WHERE JSONExtractFloat(Body, 'bytes') > 1000
AND toStartOfFiveMinutes(Timestamp) >= toStartOfFiveMinutes(NOW() - INTERVAL 1 HOUR)
AND Timestamp >= NOW() - INTERVAL 1 HOUR
ORDER BY (toStartOfFiveMinutes(Timestamp), Timestamp) DESC
LIMIT 100;
```
### Traces
- Find spans with specific attribute.
```sql
SELECT Timestamp,
TraceId,
SpanId,
ParentSpanId,
SpanName,
SpanKind,
ServiceName,
Duration,
StatusCode,
StatusMessage,
toString(SpanAttributes),
toString(ResourceAttributes),
toString(Events.Name),
toString(Links.TraceId)
FROM otel_traces
WHERE ServiceName = 'clickhouse-exporter'
AND SpanAttributes['peer.service'] = 'telemetrygen-server'
AND Timestamp >= NOW() - INTERVAL 1 HOUR
Limit 100;
```
- Find traces with traceID (using time primary index and TraceID skip index).
```sql
WITH
'391dae938234560b16bb63f51501cb6f' as trace_id,
(SELECT min(Start) FROM otel_traces_trace_id_ts WHERE TraceId = trace_id) as start,
(SELECT max(End) + 1 FROM otel_traces_trace_id_ts WHERE TraceId = trace_id) as end
SELECT Timestamp,
TraceId,
SpanId,
ParentSpanId,
SpanName,
SpanKind,
ServiceName,
Duration,
StatusCode,
StatusMessage,
toString(SpanAttributes),
toString(ResourceAttributes),
toString(Events.Name),
toString(Links.TraceId)
FROM otel_traces
WHERE TraceId = trace_id
AND Timestamp >= start
AND Timestamp <= end
Limit 100;
```
- Find spans is error.
```sql
SELECT Timestamp,
TraceId,
SpanId,
ParentSpanId,
SpanName,
SpanKind,
ServiceName,
Duration,
StatusCode,
StatusMessage,
toString(SpanAttributes),
toString(ResourceAttributes),
toString(Events.Name),
toString(Links.TraceId)
FROM otel_traces
WHERE ServiceName = 'clickhouse-exporter'
AND StatusCode = 'Error'
AND Timestamp >= NOW() - INTERVAL 1 HOUR
Limit 100;
```
- Find slow spans.
```sql
SELECT Timestamp,
TraceId,
SpanId,
ParentSpanId,
SpanName,
SpanKind,
ServiceName,
Duration,
StatusCode,
StatusMessage,
toString(SpanAttributes),
toString(ResourceAttributes),
toString(Events.Name),
toString(Links.TraceId)
FROM otel_traces
WHERE ServiceName = 'clickhouse-exporter'
AND Duration > 1 * 1e9
AND Timestamp >= NOW() - INTERVAL 1 HOUR
Limit 100;
```
### Metrics
Metrics data is stored in different clickhouse tables depending on their types. The tables will have a suffix to
distinguish which type of metrics data is stored.
| Metrics Type | Metrics Table |
| --------------------- | ---------------------- |
| sum | _sum |
| gauge | _gauge |
| histogram | _histogram |
| exponential histogram | _exponential_histogram |
| summary | _summary |
Before you make a metrics query, you need to know the type of metric you wish to use. If your metrics come from
Prometheus(or someone else uses OpenMetrics protocol), you also need to know the
[compatibility](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/compatibility/prometheus_and_openmetrics.md#prometheus-and-openmetrics-compatibility)
between Prometheus(OpenMetrics) and OTLP Metrics.
- Find a sum metrics with name
```sql
select TimeUnix,MetricName,Attributes,Value from otel_metrics_sum
where MetricName='calls' limit 100
```
- Find a sum metrics with name, attribute.
```sql
select TimeUnix,MetricName,Attributes,Value from otel_metrics_sum
where MetricName='calls' and Attributes['service_name']='featureflagservice'
limit 100
```
The OTLP Metrics [define two type value for one datapoint](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/metrics/v1/metrics.proto#L358),
clickhouse only use one value of float64 to store them.
### Profiles
> [!IMPORTANT]
> Profiles support is at `development` stability. The OpenTelemetry profiling signal itself is
> pre-GA (the OTLP profiles protocol is in `v1development`), so the schema and behavior may change
> in a backwards-incompatible way. To send profiles through a collector pipeline you must enable the
> `service.profilesSupport` feature gate (`--feature-gates=+service.profilesSupport`).
>
> **The profiles table requires ClickHouse 26.2 or newer.** It always uses `text` (full-text-search)
> indexes and does not fall back to `bloom_filter` on older server versions. If you manage the schema
> yourself (`create_schema: false`), you can adapt the DDL for an older version.
Profiles are stored as one denormalized row per OTLP `Sample`. The interned `ProfilesDictionary`
(strings, functions, locations, mappings, links, attributes) is resolved at write time so each row
is self-contained and can be queried without joins.
## Performance Guide
A single ClickHouse instance with 32 CPU cores and 128 GB RAM can handle around 20 TB (20 Billion) logs per day,
the data compression ratio is 7 ~ 11, the compressed data store in disk is 1.8 TB ~ 2.85 TB,
add more clickhouse node to cluster can increase linearly.
The otel-collector with `otlp receiver/clickhouse tcp exporter` (with `sending_queue` batching enabled) can process
around 40k/s logs entry per CPU cores, add more collector node can increase linearly.
### Reading a shared table by a resource attribute
The default schemas order primarily by time and `ServiceName`, and store resource-level
labels (for example a `tenant` or `namespace` attribute) in the `ResourceAttributes` map
rather than in the sort key. A skip index on the map values lets a query that filters on a
single attribute value prune granules, but because every value shares the same
time-partitioned parts, a wide time-range read filtered to one value still scans granules
that also contain other values' rows — the filter is applied after the granule is read.
For a low-cardinality attribute this is negligible. If you query a shared table by a
*high-cardinality* attribute (for example one tenant out of hundreds or thousands, each
reading concurrently), that shared-parts scan can become the read bottleneck. In that case,
set [`create_schema: false`](#schema-management) and manage the DDL yourself so the
attribute participates in the primary key — for example by adding it early in `ORDER BY` or
via a projection — accepting that this diverges from the default schema.
## Configuration options
The following settings are required:
- `endpoint` (no default): The ClickHouse server address, support multi host with port, for example:
- tcp protocol `tcp://addr1:port,tcp://addr2:port` or TLS `tcp://addr1:port,addr2:port?secure=true`
- http protocol `http://addr1:port,addr2:port` or https `https://addr1:port,addr2:port`
- clickhouse protocol `clickhouse://addr1:port,addr2:port` or TLS `clickhouse://addr1:port,addr2:port?secure=true`
When multiple endpoints are provided, the driver handles load balancing and automatic failover. By default, it uses `in_order` strategy (tries endpoints in the order specified). Alternatively, use `connection_open_strategy=round_robin` (distributes connections evenly) or `connection_open_strategy=random` (randomly selects endpoints) in `connection_params`. See [connection_open_strategy documentation](https://pkg.go.dev/github.com/ClickHouse/clickhouse-go/v2#readme-connection-strategy).
Many other ClickHouse specific options can be configured through query parameters e.g. `addr?dial_timeout=5s&compress=lz4`. For a full list of options see the [ClickHouse driver documentation](https://pkg.go.dev/github.com/ClickHouse/clickhouse-go/v2#readme-connection-settings-reference)
Connection options:
- `username` (default = ): The authentication username.
- `password` (default = ): The authentication password.
- `ttl` (default = 0): The data time-to-live example 30m, 48h. Also, 0 means no ttl.
- `database` (default = default): The database name. Overrides the database defined in `endpoint` when this setting is not equal to `default`.
- `connection_params` (default = {}). Extra connection parameters with map format. Query parameters provided in `endpoint` will be individually overwritten if present in this map. Parameters can be either driver parameters (e.g., `connection_open_strategy`, `max_open_conns`) that control client-side behavior, or ClickHouse session settings (e.g., `max_execution_time`) that are passed to the server. See the [driver parameters list](https://pkg.go.dev/github.com/ClickHouse/clickhouse-go/v2#Options) for recognized driver options; all others are treated as session settings.
- `create_schema` (default = true): When set to true, will run DDL to create the database and tables. (See [schema management](#schema-management))
- `compress` (default = lz4): Controls the compression algorithm. Valid options: `none` (disabled), `zstd`, `lz4` (default), `gzip`, `deflate`, `br`, `true` (lz4). Ignored if `compress` is set in the `endpoint` or `connection_params`.
- `async_insert` (default = true): Enables [async inserts](https://clickhouse.com/docs/en/optimize/asynchronous-inserts). Ignored if async inserts are configured in the `endpoint` or `connection_params`. Async inserts may still be overridden server-side.
- `tls` Advanced TLS configuration (See [TLS](#tls)).
Additional DSN features:
The underlying `clickhouse-go` module offers additional configuration. These can be set in the exporter's `endpoint` or `connection_params` config values.
- `client_info_product` Must be in `productName/version` format with comma separated entries. By default the exporter will append its binary build information. You can use this information to track the origin of `INSERT` statements in the `system.query_log` table.
ClickHouse tables:
- `logs_table_name` (default = otel_logs): The table name for logs.
- `traces_table_name` (default = otel_traces): The table name for traces.
- `profiles_table_name` (default = otel_profiles): The table name for profiles.
- `metrics_tables`
- `gauge`
- `name` (default = "otel_metrics_gauge")
- `sum`
- `name` (default = "otel_metrics_sum")
- `summary`
- `name` (default = "otel_metrics_summary")
- `histogram`
- `name` (default = "otel_metrics_histogram")
- `exponential_histogram`
- `name` (default = "otel_metrics_exp_histogram")
Cluster definition:
- `cluster_name` (default = ): Optional. If present, will include `ON CLUSTER cluster_name` when creating the database and tables.
Table engine:
- `table_engine`
- `name` (default = MergeTree)
- `params` (default = )
Modifies `ENGINE` definition when table is created. If not set then `ENGINE` defaults to `MergeTree()`.
Can be combined with `cluster_name` to enable [replication for fault tolerance](https://clickhouse.com/docs/en/architecture/replication).
Database engine:
- `database_engine`
- `name` (default = )
- `params` (default = )
Modifies `ENGINE` definition when the database is created. If not set then the server default is used. Not used for the `default` database.
With `Replicated`, `ON CLUSTER` is only used for the database since a [Replicated database](https://clickhouse.com/docs/engines/database-engines/replicated) replicates table DDL itself.
Processing:
- `timeout` (default = 5s): The timeout for every attempt to send data to the backend.
- `sending_queue`
- `enabled` (default = true)
- `num_consumers` (default = 10): Number of concurrent consumers that dequeue and insert data into ClickHouse. Enabling `batch` does not reduce this parallelism, only the (cheap) queue reader becomes single-threaded, while inserts still run on up to `num_consumers` workers. Ignored if `enabled` is `false`.
- `queue_size` (default = 1000): Maximum size of the queue, measured in `sizer` units. Data is dropped when the queue is full unless `block_on_overflow` is enabled.
- `sizer` (default = `requests`): How `queue_size` is measured. One of `requests`, `items`, or `bytes`.
- `block_on_overflow` (default = false): If `true`, waits for space when the queue is full instead of dropping data (applies backpressure to the pipeline).
- `storage` (default = none): Name of a [storage extension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/storage/filestorage) for a persistent, crash-safe queue. When unset, the queue is in-memory and is lost on restart.
- `batch` (disabled by default): Batches data inside the sending queue. Add an empty `batch: {}` to enable it with the defaults below. This replaces the standalone `batch` processor; see the batching recommendation near the top of this document.
- `flush_timeout` (default = 200ms): Time after which a batch is sent regardless of size.
- `min_size` (default = 8192): Minimum batch size before it is sent, in `batch.sizer` units.
- `max_size` (default = 0): Maximum batch size; `0` means no limit. When set, larger batches are split, and it must be `>= min_size`.
- `sizer` (default = `items`): How batch size is measured. One of `items` or `bytes` (not `requests`). If unset, inherits `sending_queue.sizer`.
- `retry_on_failure`
- `enabled` (default = true)
- `initial_interval` (default = 5s): The Time to wait after the first failure before retrying; ignored if `enabled`
is `false`
- `max_interval` (default = 30s): The upper bound on backoff; ignored if `enabled` is `false`
- `max_elapsed_time` (default = 300s): The maximum amount of time spent trying to send a batch; ignored if `enabled`
is `false`
## TLS
The exporter supports TLS. To enable TLS, you must specify the `secure=true` query parameter in the `endpoint` URL or use the `https` scheme.
You may also use certificate authentication with the `tls` setting:
```yaml
exporters:
clickhouse:
endpoint: . . .
tls:
insecure: false
insecure_skip_verify: false
ca_file: CAroot.crt
cert_file: client.crt
key_file: client.key
```
The available `tls` options are inherited from [OpenTelemetry's TLS config structure](https://pkg.go.dev/go.opentelemetry.io/collector/config/configtls#ClientConfig), more options are available than shown in this example.
## Schema management
By default, the exporter will create the database and tables under the names defined in the config. This is fine for simple deployments, but for production workloads, it is recommended that you manage your own schema by setting `create_schema` to `false` in the config.
This prevents each exporter process from racing to create the database and tables, and makes it easier to upgrade the exporter in the future.
In this mode, the only SQL sent to your server will be for `INSERT` statements.
The default DDL used by the exporter can be found in `internal/sqltemplates`.
Be sure to customize the indexes, TTL, and partitioning to fit your deployment.
Column names and types must be the same to preserve compatibility with the exporter's `INSERT` statements.
As long as the column names/types match the `INSERT` statement, you can create whatever kind of table you want.
See [ClickHouse's LogHouse](https://clickhouse.com/blog/building-a-logging-platform-with-clickhouse-and-saving-millions-over-datadog#schema) as an example of this flexibility.
### Upgrading existing tables
Sometimes new columns are added to the exporter in a backwards compatible way.
The exporter runs a `DESC TABLE` command on startup to determine which of these new columns are available on the table schema.
If you already have tables created by a previous version of the exporter, you will need to add these new columns manually.
Here is an example of a command you can use to update your existing table (adjust database and table names as needed):
```sql
ALTER TABLE otel.otel_logs ADD COLUMN IF NOT EXISTS EventName String CODEC(ZSTD(1));
```
To find the newest columns available check the `internal/sqltemplates` folder.
The `CREATE TABLE` statements will always have the latest columns.
In some cases the table changes will not be backwards compatible. Be sure to check the changelog for breaking changes before upgrading your collector.
### Optional table upgrades
As mentioned in the previous section, the exporter is able to detect which columns are present on the schema for backwards compatibility.
Here are some columns you can add to your table to update the schema:
```sql
-- EventName
ALTER TABLE otel.otel_logs ADD COLUMN IF NOT EXISTS EventName String CODEC(ZSTD(1));
-- JSON tables only. These 3 columns are part of one feature, you must add all 3 at once.
ALTER TABLE otel.otel_logs
ADD COLUMN IF NOT EXISTS ResourceAttributesKeys Array(LowCardinality(String)) CODEC(ZSTD(1)),
ADD COLUMN IF NOT EXISTS ScopeAttributesKeys Array(LowCardinality(String)) CODEC(ZSTD(1)),
ADD COLUMN IF NOT EXISTS LogAttributesKeys Array(LowCardinality(String)) CODEC(ZSTD(1)),
-- Optional indices
ADD INDEX IF NOT EXISTS idx_res_attr_keys ResourceAttributesKeys TYPE bloom_filter(0.01) GRANULARITY 1,
ADD INDEX IF NOT EXISTS idx_scope_attr_keys ScopeAttributesKeys TYPE bloom_filter(0.01) GRANULARITY 1,
ADD INDEX IF NOT EXISTS idx_log_attr_keys LogAttributesKeys TYPE bloom_filter(0.01) GRANULARITY 1;
-- JSON tables only. These 2 columns are part of one feature, you must add all 2 at once.
ALTER TABLE otel.otel_traces
ADD COLUMN IF NOT EXISTS ResourceAttributesKeys Array(LowCardinality(String)) CODEC(ZSTD(1)),
ADD COLUMN IF NOT EXISTS SpanAttributesKeys Array(LowCardinality(String)) CODEC(ZSTD(1)),
-- Optional indices
ADD INDEX IF NOT EXISTS idx_res_attr_keys ResourceAttributesKeys TYPE bloom_filter(0.01) GRANULARITY 1,
ADD INDEX IF NOT EXISTS idx_span_attr_keys SpanAttributesKeys TYPE bloom_filter(0.01) GRANULARITY 1;
```
## Example Config
This example shows how to configure the exporter to send data to a ClickHouse server.
It uses the native protocol without TLS. The exporter will create the database and tables if they don't exist.
The data is stored for 72 hours (3 days).
```yaml
receivers:
examplereceiver:
exporters:
clickhouse:
endpoint: tcp://127.0.0.1:9000?dial_timeout=10s
database: otel
async_insert: true
ttl: 72h
compress: lz4
create_schema: true
logs_table_name: otel_logs
traces_table_name: otel_traces
timeout: 5s
metrics_tables:
gauge:
name: "otel_metrics_gauge"
sum:
name: "otel_metrics_sum"
summary:
name: "otel_metrics_summary"
histogram:
name: "otel_metrics_histogram"
exponential_histogram:
name: "otel_metrics_exp_histogram"
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
# cluster_name: my_cluster
# database_engine:
# name: Replicated
# params: "'/clickhouse/databases/otel'"
# table_engine:
# name: ReplicatedMergeTree
# params:
service:
pipelines:
logs:
receivers: [ examplereceiver ]
exporters: [ clickhouse ]
```
## Experimental JSON support
JSON column types can be enabled per-exporter using the `json` config option:
```yaml
exporters:
clickhouse:
endpoint: clickhouse://localhost:9000?enable_json_type=1
json: true
```
Previously, the `clickhouse.json` feature gate was used to enable JSON for all
ClickHouse exporter instances. This feature gate is now deprecated. Use the `json`
config option instead, which allows per-pipeline control.
You may also need to add `enable_json_type=1` to your endpoint or `connection_params`.
DDL has been updated, but feel free to tune the schema as needed.
DDL can be found in the `internal/sqltemplates` package.
All `Map` columns have been replaced with `JSON`.
ClickHouse v25+ is recommended for reliable JSON support.
## Contributing
Before contributing, review the contribution guidelines in [CONTRIBUTING.md](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md).
#### Integration tests
Integration tests can be run with the following command (includes unit tests):
```sh
go test -tags integration
```
*Note: Make sure integration tests pass after making changes to SQL.*
---
### Overview (`exporter/coralogixexporter/README.md`)
# Coralogix Exporter
| Status | |
| ------------- |-----------|
| Stability | [alpha]: profiles |
| | [beta]: traces, metrics, logs |
| Distributions | [contrib] |
| Warnings | [Authentication issues in v0.127.0](#warnings) |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fcoralogix) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fcoralogix) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_coralogix&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@povilasv](https://www.github.com/povilasv), [@iblancasa](https://www.github.com/iblancasa), [@douglascamata](https://www.github.com/douglascamata) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
The Coralogix exporter sends traces, metrics and logs to [Coralogix](https://coralogix.com/).
> Please review the Collector's [security
> documentation](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md),
> which contains recommendations on securing sensitive information such as the
> API key required by this exporter.
## Configuration
Example configuration:
```yaml
exporters:
coralogix:
# The Coralogix domain
domain: "coralogix.com"
# Your Coralogix private key is sensitive
private_key: "xxx"
# (Optional) Protocol to use for communication: "grpc" (default) or "http"
protocol: "grpc"
# (Optional) Ordered list of Resource attributes that are used for Coralogix
# AppName and SubSystem values. The first non-empty Resource attribute is used.
# Example: application_name_attributes: ["k8s.namespace.name", "service.namespace"]
# Example: subsystem_name_attributes: ["k8s.deployment.name", "k8s.daemonset.name", "service.name"]
application_name_attributes:
- "service.namespace"
subsystem_name_attributes:
- "service.name"
# Traces, Metrics and Logs emitted by this OpenTelemetry exporter
# are tagged in Coralogix with the default application and subsystem constants.
application_name: "MyBusinessEnvironment"
subsystem_name: "MyBusinessSystem"
# (Optional) Configure the sending queue for batching capabilities
sending_queue:
sizer: bytes
batch:
min_size: 4194304
max_size: 8388608
# (Optional) Timeout is the timeout for every attempt to send data to the backend.
timeout: 30s
# (Optional) Use AWS PrivateLink for private connectivity. When true, data is sent to
# ingress.private.. See "Coralogix's Domain" section below for details.
# private_link: true
```
### Transport Protocol
The Coralogix exporter supports two transport protocols:
- **gRPC** (default): Uses gRPC for efficient binary communication
- **HTTP**: Uses HTTP with protobuf encoding, useful for proxy support or environments where gRPC is restricted
To use HTTP protocol:
```yaml
exporters:
coralogix:
protocol: "http"
domain: "coralogix.com"
```
#### Using HTTP Protocol with Proxy
When using HTTP protocol, you can configure proxy settings:
```yaml
exporters:
coralogix:
protocol: "http"
domain: "coralogix.com"
private_key: "xxx"
application_name: "MyApp"
subsystem_name: "MySubsystem"
domain_settings:
proxy_url: "http://proxy.example.com:8080"
timeout: 30s
```
**Notes**:
- Proxy support (`proxy_url`) is only available when using the HTTP protocol. gRPC protocol does not support this setting.
- Signal-specific settings (logs, traces, metrics) take precedence over `domain_settings`.
- **The profiles signal is not supported when using HTTP protocol**. Use gRPC protocol (default) if you need to send profiles data.
```
### Compression
By default, the Coralogix exporter uses gzip compression. Alternatively, you can use zstd compression, for example:
```yaml
exporters:
coralogix:
domain_settings:
compression: "zstd"
```
### v0.76.0 Coralogix Domain
Since v0.76.0 you can specify Coralogix domain in the configuration file instead of specifying different endpoints for traces, metrics and logs. For example, the configuration below, can be replaced with domain field:
Old configuration:
```yaml
exporters:
coralogix:
traces:
endpoint: "ingress.coralogix.com:443"
metrics:
endpoint: "ingress.coralogix.com:443"
logs:
endpoint: "ingress.coralogix.com:443"
```
New configuration with domain field:
```yaml
exporters:
coralogix:
domain: "coralogix.com"
```
### Coralogix's Domain
Depending on your region and, you might need to use a different domain. For an up-to-date list of domains, please refer to [the official Coralogix's Domain documentation](https://coralogix.com/docs/user-guides/account-management/account-settings/coralogix-domain/#domains).
Additionally, Coralogix supports AWS PrivateLink, which provides private connectivity between virtual private clouds (VPCs), supported AWS services, and your on-premises networks without exposing your traffic to the public internet. For an up-to-date list of AWS PrivateLink domains, please refer to [Coralogix's official AWS PrivateLink documentation](https://coralogix.com/docs/integrations/aws/aws-privatelink/aws-privatelink/#privatelink-endpoints).
To automatically use the PrivateLink endpoint that corresponds to the configured domain, you can set the `private_link` configuration field to `true`. For example:
```yaml
exporters:
coralogix:
domain: "eu2.coralogix.com"
private_link: true
```
### Application and SubSystem attributes
v0.62.0 release of OpenTelemetry Collector allows you to map Application name and Subsystem name to Resource attributes.
You need to set `application_name_attributes` and `subsystem_name_attributes` fields with a list of potential Resource attributes for the AppName and Subsystem values. The first not-empty Resource attribute is going to be used. If multiple resource attributes are available, **the order of the attributes in the list determines their priority.**
### Kubernetes attributes
When using OpenTelemetry Collector with [k8sattribute](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/k8sattributesprocessor) processor, you can use attributes coming from Kubernetes, such as `k8s.namespace.name` or `k8s.deployment.name`. The following example shows recommended list of attributes:
```yaml
exporters:
coralogix:
domain: "coralogix.com"
application_name_attributes:
- "k8s.namespace.name"
- "service.namespace"
subsystem_name_attributes:
- "k8s.deployment.name"
- "k8s.statefulset.name"
- "k8s.daemonset.name"
- "k8s.cronjob.name"
- "service.name"
```
### Host Attributes
OpenTelemetry Collector [Resource Detection](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor) processor can discover Host Resource attributes, such as `host.name` and provide Resource attributes using environment variables, which can be used for setting AppName and SubSystem fields in Coralogix.
Example:
```yaml
processors:
resource_detection/system:
detectors: ["system", "env"]
system:
hostname_sources: ["os"]
```
And setting environment variable such as:
```
OTEL_RESOURCE_ATTRIBUTES="env=production"
```
You can configure Coralogix Exporter:
```yaml
exporters:
coralogix:
domain: "coralogix.com"
application_name_attributes:
- "env"
subsystem_name_attributes:
- "host.name"
```
### EC2 Attributes
OpenTelemetry Collector [Resource Detection](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor) processor can discover EC2 Resource attributes, such as EC2 tags as resource attributes.
Example:
```yaml
processors:
resource_detection/ec2:
detectors: ["ec2"]
ec2:
# A list of regex's to match tag keys to add as resource attributes can be specified
tags:
- ^ec2.tag.name$
- ^ec2.tag.subsystem$
```
**_NOTE:_** In order to fetch EC2 tags, the IAM role assigned to the EC2 instance must have a policy that includes the `ec2:DescribeTags` permission.
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ec2:DescribeTags",
"Resource": "*"
}
]
}
```
You can configure Coralogix Exporter:
```yaml
exporters:
coralogix:
domain: "coralogix.com"
application_name_attributes:
- "ec2.tag.name"
subsystem_name_attributes:
- "ec2.tag.subsystem"
```
### Custom Attributes
You can combine and create custom Resource attributes using [transform](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) processor. For example:
```yaml
processors:
transform:
error_mode: ignore
log_statements:
- context: resource
statements:
- set(attributes["applicationName"], Concat(["development-environment", attributes["k8s.namespace.name"]], "-"))
```
Then you can use the custom Resource attribute in Coralogix exporter:
```yaml
exporters:
coralogix:
domain: "coralogix.com"
application_name_attributes:
- "applicationName"
subsystem_name_attributes:
- "host.name"
```
### Exporting to multiple teams based on attributes
You can export the signals based on your business logic (attributes) to different Coralogix teams. To achieve this, you'll need to use the [`filter`](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/filterprocessor/README.md) processor and setup one pipeline per team. You can setup your `filter` processors as following (example with metrics):
```yaml
processors:
filter/teamA:
metrics:
datapoint:
- 'attributes["your_label"] != "teamA"'
filter/teamB:
metrics:
datapoint:
- 'attributes["your_label"] != "teamB"'
```
This configuration ensures separate processor per each team. Any data points without an attribute for a particular team will be dropped from exporting.
Secondly, set up an individual exporter per each team:
```yaml
exporters:
coralogix/teamA:
metrics:
endpoint: "otel-metrics.coralogix.com:443"
private_key:
application_name: "MyBusinessEnvironment"
subsystem_name: "MyBusinessSystem"
coralogix/teamB:
metrics:
endpoint: "otel-metrics.coralogix.com:443"
private_key:
application_name: "MyBusinessEnvironment"
subsystem_name: "MyBusinessSystem"
```
Finally, join each processor and exporter (and any other components you wish) in the pipelines. Here is an example with a Prometheus receiver:
```yaml
service:
pipelines:
metrics/1:
receivers: [prometheus]
processors: [filter/teamA]
exporters: [coralogix/teamA]
metrics/2:
receivers: [prometheus]
processors: [filter/teamB]
exporters: [coralogix/teamB]
```
### Custom application and subsystem name
You can pass custom application and subsystem name via the following resource attributes:
- `cx.subsystem.name`
- `cx.application.name`
For example:
```yaml
receivers:
file_log/nginx:
include:
- '/tmp/tmp.log'
include_file_path: true
include_file_name: false
start_at: end
resource:
cx.subsystem.name: nginx
file_log/access-log:
include:
- '/tmp/access.log'
include_file_path: true
include_file_name: false
resource:
cx.subsystem.name: access-log
exporters:
coralogix:
domain: 'coralogix.com'
private_key: "XXX"
application_name: 'app_name'
timeout: 30s
service:
pipelines:
logs:
receivers: [file_log/nginx, file_log/access-log]
exporters: [coralogix]
```
## Warnings
### Authentication issues in v0.127.0
Version 0.127.0 introduced a regression in the Coralogix exporter. As a consequence, it requires an updated authentication configuration to ensure proper telemetry data transmission to Coralogix. If you're using this version, please modify your configuration to include the authentication headers as shown below:
```yaml
coralogix:
traces:
headers:
"Authorization": "Bearer ${env:CORALOGIX_PRIVATE_KEY}"
metrics:
headers:
"Authorization": "Bearer ${env:CORALOGIX_PRIVATE_KEY}"
logs:
headers:
"Authorization": "Bearer ${env:CORALOGIX_PRIVATE_KEY}"
```
This configuration ensures proper authentication with the Coralogix backend.
Prior versions (v0.126.0 and earlier) and subsequent versions (v0.128.0 and later) are not affected by this authentication issue.
### Need help?
Our world-class customer success team is available 24/7 to walk you through the setup for this exporter and answer any questions that may come up.
Feel free to reach out to us **via our in-app chat** or by sending us an email to [support@coralogix.com](mailto:support@coralogix.com).
---
### Overview (`exporter/datadogexporter/README.md`)
# Datadog Exporter
| Status | |
| ------------- |-----------|
| Stability | [beta]: traces, metrics, logs |
| Unsupported Platforms | aix, freebsd, solaris |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fdatadog) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fdatadog) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_datadog&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@mx-psi](https://www.github.com/mx-psi), [@dineshg13](https://www.github.com/dineshg13), [@liustanley](https://www.github.com/liustanley), [@songy23](https://www.github.com/songy23), [@mackjmr](https://www.github.com/mackjmr), [@jade-guiton-dd](https://www.github.com/jade-guiton-dd), [@IbraheemA](https://www.github.com/IbraheemA) |
| Emeritus | [@gbbr](https://www.github.com/gbbr), [@jackgopack4](https://www.github.com/jackgopack4), [@ankitpatel96](https://www.github.com/ankitpatel96) |
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
> Please review the Collector's [security documentation](https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md), which contains recommendations on securing sensitive information such as the API key required by this exporter.
> The Datadog Exporter now skips APM stats computation by default. It is recommended to only use the [Datadog Connector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/connector/datadogconnector) in order to compute APM stats.
> To temporarily revert to the previous behavior, disable the `exporter.datadogexporter.DisableAPMStats` feature gate. Example: `otelcol --config=config.yaml --feature-gates=-exporter.datadogexporter.DisableAPMStats`
Find the full configs of Datadog exporter and their usage in [collector.yaml](./examples/collector.yaml). More example configs can be found in the [official documentation](https://docs.datadoghq.com/opentelemetry/setup/collector_exporter/).
## FAQs
### Why am I getting errors 413 - Request Entity Too Large, how do I fix it?
This error indicates the payload size sent by the Datadog exporter exceeds the size limit (see previous examples https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/16834, https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/17566).
This is usually caused by the pipeline batching too many telemetry data before sending to the Datadog API intake. To fix that, prefer using the Datadog exporter `sending_queue::batch` section instead of the batch processor:
```yaml
exporters:
datadog:
api:
key: ${env:DD_API_KEY}
sending_queue:
batch:
min_size: 10
max_size: 100
flush_timeout: 10s
```
If you are using the batch processor instead, try lowering `send_batch_size` and `send_batch_max_size` in your config. You might want to have a separate batch processor dedicated for datadog exporter if other exporters expect a larger batch size, e.g.
```
processors:
batch: # To be used by other exporters
timeout: 1s
# Default value for send_batch_size is 8192
batch/datadog:
send_batch_max_size: 100
send_batch_size: 10
timeout: 10s
...
service:
pipelines:
metrics:
receivers: ...
processors: [batch/datadog]
exporters: [datadog]
```
The exact values for `send_batch_size` and `send_batch_max_size` depends on your specific workload. Also note that, Datadog intake has different payload size limits for the 3 signal types:
- Trace intake: 3.2MB
- Log intake: https://docs.datadoghq.com/api/latest/logs/
- Metrics V2 intake: https://docs.datadoghq.com/api/latest/metrics/#submit-metrics
### Fall back to the Zorkian metric client with feature gate
Support for Zorkian is now deprecated, please use the metrics export serializer. See https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/tag/v0.122.0 and #37930 for more info about Metrics Export Serializer.
### Remap OTel’s service.name attribute to service for logs
**NOTE** this workaround is only needed when feature gate `exporter.datadogexporter.UseLogsAgentExporter` is disabled. This feature gate is enabled by default starting v0.108.0.
For Datadog Exporter versions 0.83.0 - v0.107.0, the `service` field of OTel logs is populated as [OTel semantic convention](https://opentelemetry.io/docs/specs/semconv/resource/#service) `service.name`. However, `service.name` is not one of the default [service attributes](https://docs.datadoghq.com/logs/log_configuration/pipelines/?tab=service#service-attribute) in Datadog’s log preprocessing.
To get the service field correctly populated in your logs, you can specify service.name to be the source of a log’s service by setting a [log service remapper processor](https://docs.datadoghq.com/logs/log_configuration/pipelines/?tab=service#service-attribute).
[beta]:https://github.com/open-telemetry/opentelemetry-collector#beta
[alpha]:https://github.com/open-telemetry/opentelemetry-collector#alpha
[contrib]:https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[AWS]:https://aws-otel.github.io/docs/partners/datadog
### How to add custom log source
In order to add a custom source to your OTLP logs, set resource attribute `datadog.log.source`. This feature requires `exporter.datadogexporter.UseLogsAgentExporter` feature flag to be enabled (now enabled by default).
Example:
```
processors:
transform/logs:
log_statements:
- context: resource
statements:
- set(attributes["datadog.log.source"], "otel")
```
### My Collector K8s pod is getting rebooted on startup when I don't manually set a hostname under `exporters::datadog::hostname`
This is due to a bug with underlying hostname detection blocking the `health_check` extension from responding to liveness/readiness probes on startup. To fix, either set `hostname_detection_timeout` to be less than the pod/daemonset `livenessProbe: failureThreshold * periodSeconds` so that the timeout for hostname detection on startup takes less time than the control plane waits before restarting the pod, or leave `hostname_detection_timeout` at the default `25s` value and double-check the `livenessProbe` and `readinessProbe` settings and ensure that the control plane will in fact wait long enough for startup to complete before restarting the pod.
Hostname detection is currently required to initialize the Datadog Exporter, unless a hostname is specified manually under `hostname`.
---
### Overview (`exporter/datasetexporter/README.md`)
# DataSet Exporter
| Status | |
| ------------- |-----------|
| Stability | [alpha]: logs, traces |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fdataset) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fdataset) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_dataset&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@atoulme](https://www.github.com/atoulme) |
| Emeritus | [@zdaratom-s1](https://www.github.com/zdaratom-s1), [@tomaz-s1](https://www.github.com/tomaz-s1), [@martin-majlis-s1](https://www.github.com/martin-majlis-s1) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
This exporter sends logs to [DataSet](https://www.dataset.com/).
See the [Getting Started](https://app.scalyr.com/help/getting-started) guide.
## Configuration
### Required Settings
- `dataset_url` (no default): The URL of the DataSet API that ingests the data. Most likely https://app.scalyr.com.
- `api_key` (no default): The "Log Write" API Key required to use API. Instructions how to get [API key](https://app.scalyr.com/help/api-keys).
If you do not want to specify `api_key` in the file, you can use the [builtin functionality](https://opentelemetry.io/docs/collector/configuration/#configuration-environment-variables) and use `api_key: ${env:DATASET_API_KEY}`.
### Server Host Settings
Specifying the server host is crucial for ensuring the correct functionality of DataSet.
DataSet expects the server host value to be provided in the `serverHost` attribute.
If the server host value is stored in a different attribute, you can use the [resourceprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/resourceprocessor/README.md) or [attributesprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor) to copy it into the `serverHost` attribute.
You can also utilize the `server_host` settings (described below) to populate the serverHost attribute with different values.
The process of populating the serverHost attribute works as follows:
* If the `serverHost` attribute is specified and not empty in the log or trace, then it is used.
* If the `serverHost` attribute is specified and not empty in the [resource](https://opentelemetry.io/docs/specs/otel/resource/sdk/), then it is used.
* If the `host.name` attribute is specified and not empty in the [resource](https://opentelemetry.io/docs/specs/otel/resource/sdk/), then it is used.
* If the `server_host.server_host` setting is specified and not empty, then it is used.
* If `server_host.use_host_name` setting is set to `true`, the `hostname` of the node is used.
Make sure to provide the appropriate server host value in the `serverHost` attribute to ensure the proper functionality of DataSet and accurate handling of events.
### Optional Settings
- `debug` (default = false): Adds `session_key` to the server fields. It's useful for debugging throughput issues.
- `buffer`:
- `max_lifetime` (default = 5s): The maximum delay between sending batches from the same session.
- `purge_older_than` (default = 30s): The maximum delay between receiving data for the same session after which resources associated with it are purged.
- `group_by` (default = []): The list of attributes based on which events should be grouped. They are moved from the event attributes to the session info and shown as server fields in the UI.
- `retry_initial_interval` (default = 5s): Time to wait after the first failure before retrying.
- `retry_max_interval` (default = 30s): Is the upper bound on backoff.
- `retry_max_elapsed_time` (default = 300s): Is the maximum amount of time spent trying to send a buffer.
- `retry_shutdown_timeout` (default = 30s): The maximum time for which it will try to send data to the DataSet during shutdown. This value should be shorter than container's grace period.
- `max_parallel_outgoing` (default = 100): The maximum number of parallel outgoing requests.
- `logs`:
- `export_resource_info_on_event` (default = false): Include LogRecord resource information (if available) on the DataSet event.
- `export_resource_prefix` (default = 'resource.attributes.'): A prefix string for the resource, if `export_resource_info_on_event` is enabled.
- `export_scope_info_on_event` (default = true): Include LogRecord scope information (if available) on the DataSet event.
- `export_scope_prefix` (default = 'scope.attributes.'): A prefix string for the scope, if `export_scope_info_on_event` is enabled.
- `export_separator` (default = '.'): The separator to add between keys when flattening nested structures (maps, arrays).
- `export_distinguishing_suffix` (default = '_'): A suffix string to resolve naming collisions when flattening.
- `decompose_complex_message_field` (default = false): Decompose complex body / message field types (e.g. a maps, arrays) into separate fields.
- `decomposed_complex_message_prefix` (default = 'body.map.'): A prefix string to use when a complex message is decomposed.
- `traces`:
- `export_separator` (default = '.'): The separator to add between keys when flattening nested structures (maps, arrays).
- `export_distinguishing_suffix` (default = '_'): A suffix string to resolve naming collisions when flattening.
- `server_host`:
- `server_host` (default = ''): Specifies the server host to be used for the events.
- `use_hostname` (default = true): Determines whether the `hostname` of the node should be used as the server host for the events. When set to `true`, the node's `hostname` is automatically used.
- `retry_on_failure`: See [retry_on_failure](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
- `sending_queue`: See [sending_queue](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
- `timeout`: See [timeout](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
### Attributes
Enabled attributes are exported in the order:
1. Log properties
2. Body
3. Resource attributes
4. Scope attributes
5. Log attributes
If there is a name conflict, the `export_distinguishing_suffix` value is appended to the later attribute's name. If the `export_distinguishing_suffix` value is an empty string, then the value from the last attribute is used.
#### Example
Example LogRecord:
```
Log
- body:
- b: 1
- x: "b"
- resource:
- r: 2
- x: "r"
- scope:
- s: 3
- x: "s"
- attribute:
- a: 4
- x: "a"
- map:
- m1: 5
- m2: 6
```
Then the event will look like:
* Default settings for `logs`:
* Event:
```
- message: "{\"b\": 1, \"x\": \"b\"}"
- scope.attributes.s: 3
- scope.attributes.x: "s"
- a: 4
- x: "a"
- map.m1: 5
- map.m2: 6
```
* Everything enabled:
* Configuration:
```
logs:
export_resource_info_on_event: true
export_resource_prefix: "r."
export_scope_info_on_event: true
export_scope_prefix: "s."
decompose_complex_message_field: true
decomposed_complex_message_prefix: "m."
export_separator: "-"
export_distinguishing_suffix: "_"
```
* Event:
```
- message: "{\"b\": 1, \"x\": \"b\"}"
- m.b: 1
- m.x: "b"
- r.r: 2
- r.x: "r"
- s.s: 3
- s.x: "s"
- a: 4
- x: "a"
- map-m1: 5
- map-m2: 6
```
* Everything enabled, prefixes are empty strings:
* Configuration:
```
logs:
export_resource_info_on_event: true
export_resource_prefix: ""
export_scope_info_on_event: true
export_scope_prefix: ""
decompose_complex_message_field: true
decomposed_complex_message_prefix: ""
export_separator: "-"
export_distinguishing_suffix: "_"
```
* Event:
```
- message: "{\"b\": 1, \"x\": \"b\"}"
- b: 1
- x: "b"
- r: 2
- x_: "r"
- s: 3
- x__: "s"
- a: 4
- x___: "a"
- map-m1: 5
- map-m2: 6
```
* Everything enabled, prefixes are empty strings, suffix is empty string:
* Configuration:
```
logs:
export_resource_info_on_event: true
export_resource_prefix: ""
export_scope_info_on_event: true
export_scope_prefix: ""
decompose_complex_message_field: true
decomposed_complex_message_prefix: ""
export_separator: "-"
export_distinguishing_suffix: ""
```
* Event:
```
- message: "{\"b\": 1, \"x\": \"b\"}"
- b: 1
- r: 2
- s: 3
- a: 4
- x: "a"
- map-m1: 5
- map-m2: 6
```
Field names can have `.` dots, `_` underscores, and `-` hyphens. You must escape slashes in Search and PowerQueries. For example, search the field name `app.kubernetes.io/component` as `app.kubernetes.io\/component`.
### Example
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Handling `serverHost` Attribute
Based on the given configuration and scenarios, here's the expected behavior:
1. Resource: `{'node_id:' 'node-pay-01', 'host.name': 'host-pay-01'}`, Log: `{'container_id': 'cont-pay-01'}`, Env: `SERVER_HOST='server-pay-01'`, Hostname: `ip-172-31-27-19`
* Since the attribute `container_id` is set, `attributesprocessor` will copy this value to the `serverHost`.
* Used `serverHost` will be `cont-pay-01`.
2. Resource: `{'node_id': 'node-pay-01', 'host.name': 'host-pay-01'}`, Log: `{'attribute.foo': 'Bar'}`, Env: `SERVER_HOST='server-pay-01'`, Hostname: `ip-172-31-27-19`
* Since the resource attribute `node_id` is set, `resourceprocessor` will copy this value to the `serverHost`.
* Used `serverHost` will be `node-pay-01`.
3. Resource: `{'host.name': 'host-pay-01'}`, Log: `{'attribute.foo': 'Bar'}`, Env: `SERVER_HOST='server-pay-01'`, Hostname: `ip-172-31-27-19`
* Since the resource attribute `host.name` is set, it will be used.
* Used `serverHost` will be `host-pay-01`.
4. Resource: `{}`, Log: `{'attribute.foo': 'Bar'}`, Env: `SERVER_HOST='server-pay-01'`, Hostname: `ip-172-31-27-19`
* Since the attribute `container_id` is not set, the value from the environmental variable `SERVER_HOST` will be copied to the `serverHost`.
* Used `serverHost` will be `server-pay-01`.
5. Resource: `{}`, Log: `{'attribute.foo': 'Bar'}`, Env: `SERVER_HOST=''`, Hostname: `ip-172-31-27-19`
* Since the attribute `container_id` is not set and the environmental variable `SERVER_HOST` is empty, the `hostname` of the node (`ip-172-31-27-19`) will be used as the fallback value for `serverHost`.
* Used `serverHost` will be `ip-172-31-27-19`.
## Metrics
To enable metrics you have to:
1. Run collector with enabled feature gate `telemetry.useOtelForInternalMetrics`. This can be done by executing it with one additional parameter - `--feature-gates=telemetry.useOtelForInternalMetrics`.
2. Enable metrics scraping as part of the configuration and add receiver into services:
```yaml
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 5s
static_configs:
- targets: ['0.0.0.0:8888']
...
service:
pipelines:
metrics:
# add prometheus among metrics receivers
receivers: [prometheus]
exporters: [otlp_http/prometheus, debug]
```
### Available Metrics
Available metrics contain `dataset` in their name. There are counters related to the
number of processed events (`events`), buffers (`buffer`), sessions (`sessions`), and transferred bytes (`bytes`).
There are also histograms related to response times (`responseTime`) and payload size (`payloadSize`).
There are several counters related to events/buffers:
* `enqueued` - the number of received entities
* `processed` - the number of entities that were accepted by the next layer
* `dropped` - the number of entities that were not accepted by the next layer
* `broken` - the number of entities that were somehow corrupted during processing (should be 0)
The number of entities, that are still in the queue can be computed as `enqueued - (processed + dropped + broken)`.
---
### Overview (`exporter/dorisexporter/README.md`)
# Apache Doris Exporter
| Status | |
| ------------- |-----------|
| Stability | [alpha]: traces, metrics, logs |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Fdoris) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Fdoris) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_doris&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@joker-star-l](https://www.github.com/joker-star-l) |
| Emeritus | [@atoulme](https://www.github.com/atoulme) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
This exporter supports sending traces, metrics, and logs data to [Apache Doris](https://doris.apache.org/) (version >= 2.1.1).
## Configuration
The following configuration options are supported:
* `endpoint` The http stream load address.
* `database` (default = otel) The database name.
* `username` The authentication username.
* `password` The authentication password.
* `table`
* `logs` (default = otel_logs) The table name for logs.
* `traces` (default = otel_traces) The table name for traces.
* `metrics` (default = otel_metrics) The table name for metrics.
* `create_schema` (default = true) Whether databases and tables are created automatically in doris.
* `mysql_endpoint` The mysql protocol address of doris. Only use to create the schema; ignored if `create_schema` is false.
* `history_days` (default = 0) Data older than these days will be deleted; ignored if `create_schema` is false. If set to 0, historical data will not be deleted.
* `create_history_days` (default = 0) The number of days in the history partition that was created when the table was created; ignored if `create_schema` is false. If `history_days` is not 0, `create_history_days` needs to be less than or equal to `history_days`.
* `replication_num` (default = 1) The number of replicas of the table; ignored if `create_schema` is false.
* `timezone` (default is the time zone of the opentelemetry collector if IANA Time Zone Database is found, else is UTC) The time zone of doris, e.g. Asia/Shanghai.
* `log_response` (default = false) Whether to log the response of doris stream load.
* `label_prefix` (default = open_telemetry) the prefix of the label in doris stream load. The final generated label is {label_prefix}{db}{table}{yyyyMMddHHmmss}{uuid}.
* `headers` (default is empty map) The headers of doris stream load. Details: [header parameters](https://doris.apache.org/docs/data-operate/import/import-way/stream-load-manual#load-configuration-parameters) and [group commit](https://doris.apache.org/docs/data-operate/import/group-commit-manual#stream-load).
* `log_progress_interval` (default = 10) The interval, in seconds, between statistical logs. When it is less than or equal to 0, the statistical log is not printed.
* `sending_queue` [details here](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper#configuration)
* `enabled` (default = true)
* `num_consumers` (default = 10) Number of consumers that dequeue batches; ignored if `enabled` is false.
* `queue_size` (default = 1000) Maximum number of batches kept in memory before dropping; ignored if `enabled` is false.
* `retry_on_failure` [details here](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper#configuration)
* `enabled` (default = true)
* `initial_interval` (default = 5s) Time to wait after the first failure before retrying; ignored if `enabled` is false.
* `max_interval` (default = 30s) The upper bound on backoff; ignored if `enabled` is false.
* `max_elapsed_time` (default = 300s) The maximum amount of time spent trying to send a batch; ignored if `enabled` is false. If set to 0, the retries are never stopped.
The Doris exporter supports common [HTTP Configuration Settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md#http-configuration-settings), except for compression (all requests are uncompressed). As a consequence of supporting [confighttp](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md#http-configuration-settings), the Doris exporter also supports common [TLS Configuration Settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#tls-configuration-settings).
The Doris exporter sets `timeout` (HTTP request timeout) to 60s by default. All other defaults are as defined by [confighttp](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md#http-configuration-settings).
Example:
```yaml
exporters:
doris:
endpoint: http://localhost:8030
database: otel
username: admin
password: admin
table:
logs: otel_logs
traces: otel_traces
metrics: otel_metrics
create_schema: true
mysql_endpoint: localhost:9030
history_days: 0
create_history_days: 0
replication_num: 1
timezone: Asia/Shanghai
timeout: 5s
sending_queue:
enabled: true
num_consumers: 10
queue_size: 1000
retry_on_failure:
enabled: true
initial_interval: 5s
max_interval: 30s
max_elapsed_time: 300s
```
## Notes
1. Time Zone
The Doris exporter uses IANA Time Zone Database (known as tzdata) to handle time zones, so make sure tzdata is on your system.
For example, when you use docker, you should add option `-v your/path/to/tzdata:/usr/share/zoneinfo` when running the container.
2. Traces schema
Besides the span columns, the traces table has an `is_root` column (1 for spans without a parent) and two
synchronous materialized views:
* `_services` groups service, instance and span name, answering catalog queries such as
`SELECT DISTINCT service_name`.
* `_summary` aggregates each trace by `trace_id` and day (`s_start_time`, `s_end_time`,
`s_span_count`, `s_error_count`, `s_first_root` = earliest root span as
`||`, `s_max_span_duration`). Trace-list queries that group by
`trace_id, date_trunc(timestamp, 'day')` are rewritten to this view by Doris.
Doris builds a synchronous view in the background and rejects a second one while the first is building,
so the exporter creates them one after the other. Tables created by older versions are not altered.
---
### Overview (`exporter/elasticsearchexporter/README.md`)
# Elasticsearch Exporter
| Status | |
| ------------- |-----------|
| Stability | [development]: profiles |
| | [beta]: traces, logs, metrics |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Felasticsearch) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Felasticsearch) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_elasticsearch&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@JaredTan95](https://www.github.com/JaredTan95), [@blakerouse](https://www.github.com/blakerouse), [@carsonip](https://www.github.com/carsonip), [@lahsivjar](https://www.github.com/lahsivjar) |
[development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development
[beta]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#beta
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
This exporter supports sending logs, metrics, traces and profiles to [Elasticsearch](https://www.elastic.co/elasticsearch).
The Exporter is API-compatible with Elasticsearch 7.17.x, 8.x, and 9.x. Certain features of the exporter,
such as the `otel` mapping mode, may require newer versions of Elasticsearch. Limited effort will
be made to support EOL versions of Elasticsearch -- see https://www.elastic.co/support/eol.
## Configuration options
Exactly one of the following settings is required:
- `endpoint` (no default): The target Elasticsearch URL to which data will be sent
(e.g. `https://elasticsearch:9200`)
- `endpoints` (no default): A list of Elasticsearch URLs to which data will be sent,
attempted in round-robin order
- `cloudid` (no default): The [Elastic Cloud ID](https://www.elastic.co/guide/en/cloud/current/ec-cloud-id.html)
of the Elastic Cloud Cluster to which data will be sent (e.g. `foo:YmFyLmNsb3VkLmVzLmlvJGFiYzEyMyRkZWY0NTY=`)
When the above settings are missing, `endpoints` will default to the
comma-separated `ELASTICSEARCH_URL` environment variable.
Elasticsearch credentials may be configured via [Authentication configuration][configauth] settings.
As a shortcut, the following settings are also supported:
- `user` (optional): Username used for HTTP Basic Authentication.
- `password` (optional): Password used for HTTP Basic Authentication.
- `api_key` (optional): [Elasticsearch API Key] in "encoded" format (e.g. `VFR2WU41VUJIbG9SbGJUdVFrMFk6NVVhVDE3SDlSQS0wM1Rxb24xdXFldw==`).
Example:
```yaml
exporters:
elasticsearch:
endpoint: https://elastic.example.com:9200
auth:
authenticator: basicauth
extensions:
basicauth:
client_auth:
username: elastic
password: changeme
······
service:
extensions: [basicauth]
pipelines:
logs:
receivers: [otlp]
exporters: [elasticsearch]
traces:
receivers: [otlp]
exporters: [elasticsearch]
```
## Advanced configuration
### HTTP settings
The Elasticsearch exporter supports common [HTTP Configuration Settings][confighttp]. Gzip compression is enabled by default. To disable compression, set `compression` to `none`. Default Compression Level is set to 1 (gzip.BestSpeed).
As a consequence of supporting [confighttp], the Elasticsearch exporter also supports common [TLS Configuration Settings][configtls].
```yaml
timeout: 90s
```
The Elasticsearch exporter sets `timeout` (HTTP request timeout) to 90s by default.
Note that this is per request and is independent between HTTP request retries.
All other defaults are as defined by [confighttp].
### Queuing and batching
The Elasticsearch exporter supports the common [`sending_queue` settings][exporterhelper] which
supports both queueing and batching. The default sending queue is configured to do async batching
with the following configuration:
```yaml
sending_queue:
enabled: true
sizer: requests
num_consumers: 10
queue_size: 10
batch:
flush_timeout: 10s
min_size: 1e+6 // 1MB
max_size: 5e+6 // 5MB
sizer: bytes
```
The default configurations are chosen to be closer to the defaults with the exporter's previous inbuilt batching feature. The [`exporterhelper` documentation][exporterhelper] provides more details on the `sending_queue` settings.
### Elasticsearch document routing
Documents are statically or dynamically routed to the target index / data stream in the following order. The first routing mode that applies will be used.
1. "Static mode": Route to `logs_index` for log records, `metrics_index` for data points and `traces_index` for spans, if these configs are not empty respectively. [^3]
2. "Dynamic - Index attribute mode": Route to index name specified in `elasticsearch.index` attribute (precedence: log record / data point / span attribute > scope attribute > resource attribute) if the attribute exists. [^3]
3. "Dynamic - Data stream routing mode": Route to data stream constructed from `${data_stream.type}-${data_stream.dataset}-${data_stream.namespace}`,
where `data_stream.type` is `logs` for log records, `metrics` for data points, and `traces` for spans, and is static. [^3]
In a special case with `mapping::mode: bodymap`, `data_stream.type` field (valid values: `logs`, `metrics`, `traces`, `profiles`, `synthetics`) can be dynamically set from attributes.
The resulting documents will contain the corresponding `data_stream.*` fields, see restrictions applied to [Data Stream Fields](https://www.elastic.co/guide/en/ecs/current/ecs-data_stream.html).
1. `data_stream.dataset` or `data_stream.namespace` in attributes (precedence: log record / data point / span attribute > scope attribute > resource attribute)
2. Otherwise, if a scope attribute with the name `encoding.format` exists and contains a string value, `data_stream.dataset` will be set to this value.
Note that while enabled by default, this behaviour is considered experimental. Some encoding extensions set this field (e.g. [awslogsencodingextension](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/encoding/awslogsencodingextension)), but it is not yet part of Semantic Conventions. There is the potential that the name of this routing field evolves as the [discussion progresses in SemConv](https://github.com/open-telemetry/semantic-conventions/issues/2854).
3. Otherwise, if scope name matches regex `/receiver/(\w*receiver)` or `/connector/(\w*connector)`, `data_stream.dataset` will be capture group #1
4. Otherwise, `data_stream.dataset` falls back to `generic` and `data_stream.namespace` falls back to `default`.
[^3]: See additional handling in [Document routing exceptions for OTel data mode](#document-routing-exceptions-for-otel-data-mode)
This can be customised through the following settings:
- `logs_index` (optional): The [index] or [data stream] name to publish logs (and span events in OTel mapping mode) to. `logs_index` should be empty unless all logs should be sent to the same index.
- `logs_dynamic_index` (optional): uses resource, scope, or log record attributes to dynamically construct index name.
- `enabled`(DEPRECATED): No-op. Documents are now always routed dynamically unless `logs_index` is not empty. Will be removed in a future version.
- `metrics_index` (optional): The [index] or [data stream] name to publish metrics to. `metrics_index` should be empty unless all metrics should be sent to the same index. Note that metrics support is currently in development.
- `metrics_dynamic_index` (optional): uses resource, scope or data point attributes to dynamically construct index name.
- `enabled`(DEPRECATED): No-op. Documents are now always routed dynamically unless `metrics_index` is not empty. Will be removed in a future version.
- `traces_index` (optional): The [index] or [data stream] name to publish traces to. `traces_index` should be empty unless all traces should be sent to the same index.
- `traces_dynamic_index` (optional): uses resource, scope, or span attributes to dynamically construct index name.
- `enabled`(DEPRECATED): No-op. Documents are now always routed dynamically unless `traces_index` is not empty. Will be removed in a future version.
- `logstash_format` (optional): Logstash format compatibility. Logs, metrics and traces can be written into an index in Logstash format.
- `enabled`(default=false): Enable/disable Logstash format compatibility. When `logstash_format::enabled` is `true`, the index name is composed using the above dynamic routing rules as prefix and the date as suffix,
e.g: If the computed index name is `logs-generic-default`, the resulting index will be `logs-generic-default-YYYY.MM.DD`.
The last string appended belongs to the date when the data is being generated.
- `prefix_separator`(default=`-`): Set a separator between logstash_prefix and date.
- `date_format`(default=`%Y.%m.%d`): Time format (based on strftime) to generate the second part of the Index name.
- `logs_dynamic_id` (optional): Dynamically determines the document ID to be used in Elasticsearch based on a log record attribute.
- `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document when the `otel` mapping mode is used. See [Setting a document id dynamically](#setting-a-document-id-dynamically).
- `traces_dynamic_id` (optional): Dynamically determines the document ID to be used in Elasticsearch based on a span attribute.
- `enabled`(default=false): Enable/Disable dynamic ID for spans. If `elasticsearch.document_id` exists and is not an empty string in the span attributes, it will be used as the document ID. For span events, this only applies when using `otel` mapping mode (where span events are stored as separate documents). Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document when the `otel` mapping mode is used. See [Setting a document id dynamically](#setting-a-document-id-dynamically).
#### Document routing exceptions for OTel data mode
In OTel mapping mode (`mapping::mode: otel`), there is special handling in addition to the above document routing rules in [Elasticsearch document routing](#elasticsearch-document-routing).
The order to determine the routing mode is the same as [Elasticsearch document routing](#elasticsearch-document-routing).
1. "Static mode": Span events are separate documents routed to `logs_index` if non-empty.
2. "Dynamic - Index attribute mode": Span events are separate documents routed using attribute `elasticsearch.index` (precedence: span event attribute > scope attribute > resource attribute) if the attribute exists.
3. "Dynamic - Data stream routing mode":
- For all documents, `data_stream.dataset` will always be appended with `.otel`.
- A special case to (3)(1) in [Elasticsearch document routing](#elasticsearch-document-routing), span events are separate documents that have `data_stream.type: logs` and are routed using data stream attributes (precedence: span event attribute > scope attribute > resource attribute)
### Elasticsearch document mapping
The Elasticsearch exporter supports several document schemas and preprocessing
behaviours, which may be configured through the following settings:
- `mapping`:
- `mode` (DEPRECATED): The mapping mode if supplied via config file is ignored. Use the `X-Elastic-Mapping-Mode` client metadata key or the `elastic.mapping.mode` scope attribute instead. If not specified via these methods, the default mapping mode is `otel`.
- `allowed_modes` (defaults to all mapping modes): A list of allowed mapping modes.
If `otel` is included in the list, it is used as the default mapping mode.
Otherwise, the first entry in the list is used as the default.
The mapping mode can be controlled via the client metadata key `X-Elastic-Mapping-Mode`,
e.g. via HTTP headers, gRPC metadata.
It is possible to restrict which mapping modes may be requested by configuring
`mapping::allowed_modes`, which defaults to all mapping modes. Keep in mind that not all
processors or exporter configurations will maintain client metadata.
The mapping mode can also be controlled via the scope attribute `elastic.mapping.mode`.
If specified, this takes precedence over the `X-Elastic-Mapping-Mode` client metadata.
If any scope has an invalid mapping mode, the exporter will reject the entire batch.
The attribute will be excluded from the final document.
Valid mapping modes are:
- `none`
- `ecs`
- `otel`
- `raw`
- `bodymap`
See below for a description of each mapping mode.
#### Migration: Setting mapping mode via scope attribute
Since the `mapping::mode` config option is deprecated, use the following method to set the mapping mode:
**Use scope attribute via transform processor**
This approach sets the `elastic.mapping.mode` scope attribute on the telemetry data.
```yaml
processors:
transform:
log_statements:
- context: scope
statements:
- set(attributes["elastic.mapping.mode"], "otel")
trace_statements:
- context: scope
statements:
- set(attributes["elastic.mapping.mode"], "otel")
metric_statements:
- context: scope
statements:
- set(attributes["elastic.mapping.mode"], "otel")
exporters:
elasticsearch:
endpoint: https://elasticsearch:9200
service:
pipelines:
logs:
receivers: [otlp]
processors: [transform]
exporters: [elasticsearch]
```
> [!NOTE]
> The scope attribute `elastic.mapping.mode` takes precedence over the `X-Elastic-Mapping-Mode` client metadata.
> The attribute will be excluded from the final document sent to Elasticsearch.
> [!NOTE]
> `otel` and `ecs` mapping modes require Elasticsearch 8.12 or above[^1].
> `otel` mode works best with Elasticsearch 8.16 or above[^2].
[^1]: as OTel and ECS mapping modes rely on the `require_data_stream` bulk action metadata, available since Elasticsearch 8.12
[^2]: Elasticsearch 8.16 contains a built-in `otel-data` plugin
#### OTel mapping mode
The default and recommended "OTel-native" mapping mode.
In `otel` mapping mode, the Elasticsearch Exporter stores documents in Elastic's preferred
"OTel-native" schema. In this mapping mode, documents use the original attribute names and
closely follows the event structure from the OTLP events.
There is special treatment for the following attributes: `data_stream.type`, `data_stream.dataset`,
and `data_stream.namespace`. Instead of serializing these values under the `*attributes.*` namespace,
they are put at the root of the document, to conform with the conventions of the data stream naming
scheme that maps these as `constant_keyword` fields.
`data_stream.dataset` will always be appended with `.otel` if [dynamic data stream routing mode](#elasticsearch-document-routing) is active.
Span events are stored in separate documents. They will be routed with `data_stream.type` set to
`logs` if [dynamic data stream routing mode](#elasticsearch-document-routing) is active.
Attribute `elasticsearch.index` will be removed from the final document if exists.
| Signal | Supported |
| --------- | ------------------ |
| Logs | :white_check_mark: |
| Traces | :white_check_mark: |
| Metrics | :white_check_mark: |
| Profiles | :white_check_mark: |
#### ECS mapping mode
> [!WARNING]
> The ECS mode mapping mode is currently undergoing changes, and its behaviour is unstable.
In `ecs` mapping mode, the Elasticsearch Exporter maps fields from
[OpenTelemetry Semantic Conventions][SemConv] (version 1.22.0) to [Elastic Common Schema][ECS] where possible.
This mode may be used for compatibility with existing dashboards that work with ECS.
| Signal | `ecs` |
| --------- | ------------------ |
| Logs | :white_check_mark: |
| Traces | :white_check_mark: |
| Metrics | :white_check_mark: |
| Profiles | :no_entry_sign: |
In ECS mapping mode, span events are extracted as separate ECS-formatted log documents following
the APM data stream convention:
- Span events named `exception` with `exception.type` or `exception.message` present are routed
to `logs-apm.error-`.
- All other span events are routed to `logs-apm.app.-`.
#### Bodymap mapping mode
> [!WARNING]
> The Bodymap mode mapping mode is currently undergoing changes, and its behaviour is unstable.
In `bodymap` mapping mode, the Elasticsearch Exporter supports only logs and will take the "body"
of a log record as the exact content of the Elasticsearch document without any transformation.
This mapping mode is intended for use cases where the client wishes to have complete control over
the Elasticsearch document structure.
| Signal | `bodymap` |
| --------- | ------------------ |
| Logs | :white_check_mark: |
| Traces | :no_entry_sign: |
| Metrics | :no_entry_sign: |
| Profiles | :no_entry_sign: |
#### Default (none) mapping mode
In the `none` mapping mode the Elasticsearch Exporter produces documents with the original
field names of from the OTLP data structures.
| Signal | `none` |
| --------- | ------------------ |
| Logs | :white_check_mark: |
| Traces | :white_check_mark: |
| Metrics | :no_entry_sign: |
| Profiles | :no_entry_sign: |
#### Raw mapping mode
The `raw` mapping mode is identical to `none`, except for two differences:
- In `none` mode attributes are mapped with an `Attributes.` prefix,
while in `raw` mode they are not.
- In `none` mode span events are mapped with an `Events.` prefix,
while in `raw` mode they are not.
| Signal | `raw ` |
| --------- | ------------------ |
| Logs | :white_check_mark: |
| Traces | :white_check_mark: |
| Metrics | :no_entry_sign: |
| Profiles | :no_entry_sign: |
### Elasticsearch ingest pipeline
Documents may be optionally passed through an [Elasticsearch Ingest pipeline] prior to indexing.
This can be configured through the following settings:
- `pipeline` (optional): ID of an [Elasticsearch Ingest pipeline] used for processing documents published by the exporter.
- `logs_dynamic_pipeline` (optional): Dynamically determines the ingest pipeline to be used in Elasticsearch based on attributes in the log signal.
- `enabled`(default=false): Enable/Disable dynamic pipeline. If `elasticsearch.ingest_pipeline` attribute exists in the log record attributes and is not an empty string, it will be used as the Elasticsearch ingest pipeline. This currently only applies to the log signal. The attribute `elasticsearch.ingest_pipeline` is removed from the final document when the `otel` mapping mode is used.
### Elasticsearch bulk indexing
The Elasticsearch exporter uses the [Elasticsearch Bulk API] for indexing documents.
The behaviour of this bulk indexing can be configured with the following settings:
The deprecated `num_workers` and `flush` settings have been removed. Existing configurations should use these replacements:
| Removed setting | Replacement |
| --- | --- |
| `num_workers` | `sending_queue::num_consumers` |
| `flush::interval` | `sending_queue::batch::flush_timeout` |
| `flush::bytes` | `sending_queue::batch::max_size` with `sending_queue::batch::sizer` set to `bytes` |
- `retry`: Elasticsearch bulk request retry settings
- `enabled` (default=true): Enable/Disable request retry on error. Failed requests are retried with exponential backoff.
- `max_requests` (DEPRECATED, use retry::max_retries instead): Number of HTTP request retries including the initial attempt. If used, `retry::max_retries` will be set to `max_requests - 1`.
- `max_retries` (default=2): Number of HTTP request retries. To disable retries, set `retry::enabled` to `false` instead of setting `max_retries` to `0`.
- `initial_interval` (default=100ms): Initial waiting time if a HTTP request failed.
- `max_interval` (default=1m): Max waiting time if a HTTP request failed.
- `retry_on_status` (default=[429]): Status codes that trigger request level retries. To avoid duplicates, it defaults to `[429]`.
- `retry_on_document_status` (default=same as `retry_on_status`): Status codes that trigger document level retries for failed documents in successful bulk HTTP responses. Set to `[]` to disable document level retries by status code while keeping request level retries configured through `retry_on_status`.
- `sending_queue`: Configures the queueing and batching behaviour. Below are the defaults (which may vary from standard defaults), for full configuration check the [`exporterhelper` docs][exporterhelper].
- `enabled` (default=true): Enable queueing and batching behaviour.
- `num_consumers` (default=10): Number of consumers that dequeue batches.
- `wait_for_result` (default=false): If `true`, blocks incoming requests until processed.
- `block_on_overflow` (default=false): If `true`, blocks the request until the queue has space.
- `sizer` (default=requests): Measure queueing by requests.
- `queue_size` (default=10): Maximum size the queue can accept.
- `batch`:
- `flush_timeout` (default=10s): Time after which batch is exported irrespective of other settings.
- `sizer` (default=bytes): Size batches by bytes. Note that bytes here are based on the pdata model and not on the NDJSON docs that will constitute the bulk indexer requests. To address this discrepancy, the bulk indexers could also flush when their size exceeds the configured max_size due to size of pdata model being smaller than their corresponding NDJSON encoding.
- `min_size` (default=1MB): Min size of the batch.
- `max_size` (default=5MB): Max size of the batch. This value should be much lower than [Elasticsearch's `http.max_content_length`](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#http-settings) config to avoid HTTP 413 Entity Too Large error. It is recommended to keep this value under 5MB.
#### Bulk indexing error response
With Elasticsearch 8.18+, a new [query parameter `include_source_on_error`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk#operation-bulk-include_source_on_error)
allows users to receive the source document in the error response, if there were any parsing errors in the bulk request.
In the exporter, the equivalent configuration is also named `include_source_on_error`.
- `include_source_on_error`:
- `true`: Enables bulk index responses to include source document on error. Requires Elasticsearch 8.18+. WARNING: the exporter may log error responses containing request payload, causing potential sensitive data to be exposed in logs.
- `false`: Disables including source document on bulk index error responses. Requires Elasticsearch 8.18+.
- `null` (default): Backward-compatible option for older Elasticsearch versions. By default, the error reason is discarded from bulk index responses entirely, i.e. only error type is returned.
### Elasticsearch node discovery
The Elasticsearch Exporter will regularly check Elasticsearch for available nodes.
Newly discovered nodes will automatically be used for load balancing.
Settings related to node discovery are:
- `discover`:
- `on_start` (optional): If enabled the exporter queries Elasticsearch
for all known nodes in the cluster on startup.
- `interval` (optional): Interval to update the list of Elasticsearch nodes.
Node discovery can be disabled by setting `discover.interval` to 0.
### Elasticsearch version detection
On startup, the Elasticsearch Exporter queries each configured endpoint for its
version and build flavor and logs the result. This information is used to
automatically enable features supported by the connected Elasticsearch version,
so no per-feature configuration is required.
- `version_detection`:
- `enabled` (default `true`): If enabled, the exporter queries Elasticsearch
at startup to detect its version and build flavor. When disabled, the
exporter does not query Elasticsearch and assumes only the baseline
capability set is available.
The query is best-effort: if it fails (for example, when Elasticsearch is
temporarily unreachable), startup is not blocked. Version detection cannot be
performed when the endpoint is resolved at request time (for example, when a
routing middleware rewrites the destination per request).
### Telemetry settings
The Elasticsearch Exporter's own telemetry settings for testing and debugging purposes.
⚠️ This is experimental and may change at any time.
- `telemetry`:
- `log_request_body` (default=false): Logs Elasticsearch client request body as a field in a log line at DEBUG level. It requires `service::telemetry::logs::level` to be set to `debug`. WARNING: Enabling this config may expose sensitive data.
- `log_response_body` (default=false): Logs Elasticsearch client response body as a field in a log line at DEBUG level. It requires `service::telemetry::logs::level` to be set to `debug`. WARNING: Enabling this config may expose sensitive data.
- `log_failed_docs_input` (default=false): Include the input (action line and document line) causing indexing error under `input` field in a log line at DEBUG level. It requires `service::telemetry::logs::level` to be set to `debug`. WARNING: Enabling this config may expose sensitive data.
- `log_failed_docs_input_rate_limit` (default="1s"): Rate limiting of logs emitted by `log_failed_docs_input` config, e.g. "1s" means roughly 1 log line per second. A zero or negative value disables rate limiting.
- `include_destination_index` (default=false): Adds the `db.collection.name` attribute to the `otelcol.elasticsearch.docs.processed` metric, based on the `_index` of each document in the bulk response. For backing and failure store indices, the `.ds-`/`.fs-` prefix and the date and generation suffix are stripped, leaving the data stream name. WARNING: it multiplies the number of metric series by the number of destination indices, which can be high with dynamic index names.
### Metadata keys
Metadata keys are a list of client metadata keys that the exporter uses to partition batches
when `sending_queue` is enabled with batching support and enrich internal telemetry.
⚠️ This is experimental and may change at any time.
- `metadata_keys` (optional): List of metadata keys that will be used to partition the data
into batches if [sending_queue][exporterhelper] is enabled with batching support. With
batching enabled only these metadata keys are guaranteed to be propagated. The keys will also
be used to enrich the exporter's internal telemetry if defined. The keys are extracted from
the client metadata available via the context and added to the internal telemetry as attributes.
NOTE: The metadata keys are converted to lower case as key lookups for client metadata is case insensitive. This means that the metric produced by internal telemetry will also have the attribute in lower case.
## Exporting metrics
Metrics support is currently in development.
The metric types supported are:
- Gauge
- Sum
- Histogram (Delta temporality only)
- Exponential histogram (Delta temporality only)
- Summary
### Histogram storage
In OTel mapping mode, OTLP exponential histograms are stored in Elasticsearch's
native [`exponential_histogram`](https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/exponential-histogram)
field when connected to Elasticsearch 9.4.0 or later. This preserves the
exponential bucket layout (scale, zero bucket, positive/negative buckets, sum,
min, max) losslessly.
This is enabled automatically via [version detection](#elasticsearch-version-detection): when
version detection is disabled (`version_detection.enabled: false`) or the
connected Elasticsearch is older than 9.4.0, exponential histograms fall back to
a T-Digest `histogram` field. Explicit-bucket histograms are always stored as a
`histogram` field.
Explicit-bounds histograms are converted to exponential histograms
when the Elasticsearch version is 9.4.0 or later. This can be disabled via `version_detection.enabled: false`, which stores
explicit-bounds histograms as `histogram` field.
Per-data-point mapping hints still apply and take precedence: `histogram:raw`
stores a `histogram` field using raw bucket bounds, and `aggregate_metric_double`
stores a `{sum, value_count}` summary. The native `exponential_histogram` field
is only used in OTel mapping mode.
### Metrics dynamic templates
For metrics, the exporter sends **per-document `dynamic_templates`** with each bulk index action so that Elasticsearch can apply the correct mapping to metric fields. It uses the [bulk API `dynamic_templates` parameter](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk):
> A map from the full name of fields to the name of dynamic templates. It defaults to an empty map. If a name matches a dynamic template, that template will be applied regardless of other match predicates defined in the template. If a field is already defined in the mapping, then this parameter won't be used.
The index template must define dynamic templates whose names match the values sent by the exporter. Behavior depends on the mapping mode:
| Mapping mode | Field path in document | Template names sent | Notes |
| ------------ | ---------------------- | -------------------- | ----- |
| **OTel** | `metrics.` | `histogram`, `exponential_histogram`, `summary`, `gauge_double`, `gauge_long`, `counter_double`, `counter_long` | The OTel data plugin defines more specific templates. `exponential_histogram` requires Elasticsearch 9.4.0 or later. |
| **ECS** | `metric.` | `histogram_metrics`, `summary_metrics`, `double_metrics` | Relies on core templates in [metrics@mappings](https://github.com/elastic/elasticsearch/blob/8.15/x-pack/plugin/core/template-resources/src/main/resources/metrics%40mappings.json). Intended to match the [APM metrics ingest pipeline](https://github.com/elastic/elasticsearch/blob/b34960a2b450869aee2866e91c647e0026dd6953/x-pack/plugin/apm-data/src/main/resources/ingest-pipelines/metrics-apm%40pipeline.yaml). |
- **OTel**: Each metric is written under the `metrics` object; the bulk action maps full field names (e.g. `metrics.my_metric`) to one of the OTel template names above based on metric type (histogram, summary, gauge, or counter) and value type.
- **ECS**: Each metric is written as a top-level field `metric.`; the bulk action maps that field name to one of the ECS/APM template names (`histogram_metrics`, `summary_metrics`, or `double_metrics` for gauges and counters).
### Bulk Response Filter Path
The Elasticsearch bulk API accepts a [filter_path](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/common-options#common-options-response-filtering) parameter. This can be used to reduce the response returned by Elasticsearch. The exporter uses a default `filter_path` that is set by the [go-docappender](https://github.com/elastic/go-docappender). The default is currently `items.*._index,items.*.status,items.*.failure_store,items.*.error.type,items.*.error.reason` and is defined by the `DefaultFilterPath` in the [go-docappnder](https://github.com/elastic/go-docappender) package.
If you want to change the `filter_path` you may do so by setting `bulk_response_filter_path` to the desired string in the configuration.
> [!NOTE]
> If `items.*._index.items` is not in the BulkResponseFilterPath than for any failed documents, the exporter will not be able to log the index to which the document was being written to.
> [!NOTE]
> If `items.*._index.items` is not in the BulkResponseFilterPath than the exporter will log rejection of duplicates to ".profiling-stackframes" which were previously suppressed.
## Exporting profiles
Profiles support is currently in development, and should not be used in
production. Profiles only support the OTel mapping mode.
Example:
```yaml
exporters:
elasticsearch:
endpoint: https://elastic.example.com:9200
mapping:
mode: otel
```
> [!IMPORTANT]
> For the Elasticsearch Exporter to be able to export Profiles data, Universal Profiling needs to be installed in the database.
> See [the Universal Profiling getting started documentation](https://www.elastic.co/guide/en/observability/current/profiling-get-started.html)
> You will need to use the Elasticsearch endpoint, with an [Elasticsearch API key](https://www.elastic.co/guide/en/kibana/current/api-keys.html).
### OTel profiling datastreams
In `otel` mapping mode, profiling signals are ingested into OTel-native Elasticsearch datastreams.
Each profiling signal type is written to a dedicated backing index:
| Signal type | Index pattern |
| -------------- | --------------------------------------------- |
| Stack traces | `profiling-stacktraces.otel-default` |
| Stack frames | `profiling-stackframes.otel-default` |
| Executables | `profiling-executables.otel-default` |
| Trace events | `profiling-events-all.otel-default` |
| Downsampled trace events | `profiling-events-5powNN.otel-default` (`NN` from `01` to `11`) |
| Host metadata | `profiling-hosts.otel-default` |
> [!NOTE]
> Symbolization (resolving unsymbolized stack frames to human-readable function names and file locations)
> is not yet supported in OTel profiling datastream mode.
> [!WARNING]
> The `.otel-default` profiling datastream index templates are only available in **Elasticsearch 9.6.0 and later**.
> If you send profiles with the default OTel mapping mode to an older cluster, documents will be rejected
> with `index_not_found_exception` (HTTP 404) and profiling data will be lost.
>
> To continue sending profiles to Elasticsearch < 9.6.0, restrict the exporter to ECS mode:
>
> ```yaml
> mapping:
> allowed_modes: [ecs]
> ```
>
> This is a **breaking change** for existing OTel-mode profiling users (profiles are still in tech preview):
> the backing indices are different from the ECS-schema indices used by prior versions,
> and existing profiling data in the old indices is not migrated automatically.
[confighttp]: https://github.com/open-telemetry/opentelemetry-collector/tree/main/config/confighttp/README.md#http-configuration-settings
[configtls]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md#tls-configuration-settings
[configauth]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configauth/README.md#authentication-configuration
[exporterhelper]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md
[Elasticsearch Ingest pipeline]: https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html
[Elasticsearch Bulk API]: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html
[Elasticsearch API Key]: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html
[index]: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices.html
[data stream]: https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html
[ecs]: https://www.elastic.co/guide/en/ecs/current/index.html
[SemConv]: https://github.com/open-telemetry/semantic-conventions
## ECS Mapping
`elasticsearchexporter` follows ECS mapping defined here: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model-appendix.md#elastic-common-schema
When `mode` is set to `ecs`, `elasticsearchexporter` performs conversions for resource-level and record-level (log or trace) attributes from their Semantic Conventions (SemConv) names to equivalent Elastic Common Schema (ECS) names.
If the target ECS field name is specified as an empty string (`""`), the converter will neither convert the SemConv key to the equivalent ECS name nor pass through the SemConv key as-is to become the ECS name.
When "Preserved" is true, the attribute will be preserved in the payload and duplicated as mapped to its ECS equivalent.
When more than one SemConv attribute maps to the same ECS attribute, the converter will map all attributes to the same ECS name.
This is mean to support backwards compatibility for SemConv attributes that have been renamed/deprecated.
The value of the last-mapped attribute will take precedence.
It is recommended to enrich events using the [elasticapmprocessor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elasticapmprocessor) to ensure index documents contain all required Elastic fields to power the Kibana UI.
### Resource attribute mapping
| Semantic Convention Name | ECS Name | Preserve | Skip if exists |
|-----------------------------|-----------------------------|----------|----------------|
| client.address | client.ip | false | false |
| cloud.platform | cloud.service.name | false | false |
| container.image.tags | container.image.tag | false | false |
| deployment.environment | service.environment | false | false |
| deployment.environment.name | service.environment | false | false |
| faas.instance | faas.id | false | false |
| faas.trigger | faas.trigger.type | false | false |
| host.arch | host.architecture | false | false |
| host.hostname | host.hostname | true | true |
| k8s.cluster.name | orchestrator.cluster.name | false | false |
| k8s.container.name | kubernetes.container.name | false | false |
| k8s.cronjob.name | kubernetes.cronjob.name | false | false |
| k8s.daemonset.name | kubernetes.daemonset.name | false | false |
| k8s.deployment.name | kubernetes.deployment.name | false | false |
| k8s.job.name | kubernetes.job.name | false | false |
| k8s.namespace.name | kubernetes.namespace | false | false |
| k8s.node.name | kubernetes.node.name | false | false |
| k8s.pod.name | kubernetes.pod.name | false | false |
| k8s.pod.uid | kubernetes.pod.uid | false | false |
| k8s.replicaset.name | kubernetes.replicaset.name | false | false |
| k8s.statefulset.name | kubernetes.statefulset.name | false | false |
| os.description | host.os.full | false | false |
| os.name | host.os.name | false | false |
| os.type | host.os.platform | false | false |
| os.version | host.os.version | false | false |
| process.command_line | process.args | false | false |
| process.executable.name | process.title | false | false |
| process.executable.path | process.executable | false | false |
| process.parent.pid | process.parent.pid | false | false |
| process.runtime.name | service.runtime.name | false | false |
| process.runtime.version | service.runtime.version | false | false |
| service.instance.id | service.node.name | false | false |
| source.address | source.ip | false | false |
| telemetry.distro.name | "" | false | false |
| telemetry.distro.version | "" | false | false |
| telemetry.sdk.language | service.language.name | false | false |
| telemetry.sdk.name | "" | false | false |
| telemetry.sdk.version | service.language.version | false | false |
### Log record attribute mapping
| Semantic Convention Name | ECS Name | Preserve |
|--------------------------|---------------------------------|----------|
| event.name | event.action | false |
| exception.message | error.message | false |
| exception.stacktrace | error.stacktrace | false |
| exception.type | error.type | false |
| exception.escaped | event.error.exception.handled | false |
| http.response.body.size | http.response.encoded_body_size | false |
### Span attribute mapping
| Semantic Convention Name | ECS Name | Preserve |
|----------------------------|-----------------------------------------------------------|----------|
| db.system | span.db.type | false |
| db.namespace | span.db.instance | false |
| db.query.text | span.db.statement | false |
| http.response.body.size | http.response.encoded_body_size | false |
### Compound Mapping
There are ECS fields that are not mapped easily 1 to 1 but require more advanced logic.
#### `host.name` and `host.hostname`
Maintains the SemConv Value `host.name` as ECS Value `host.name` and maps it to ECS Value `host.hostname`, if this does not already exist.
#### `@timestamp`
In case the record contains `timestamp`, this value is used. Otherwise, the `observed timestamp` is used.
## Setting a document id dynamically
The `logs_dynamic_id` and `traces_dynamic_id` settings allow users to set the document ID dynamically based on log record, span, or span event attributes.
Besides the ability to control the document ID, these settings also work as a deduplication mechanism, as Elasticsearch will refuse to index a document with the same ID.
For logs, the log record attribute `elasticsearch.document_id` can be set explicitly by a processor based on the log record.
For traces, the span attribute `elasticsearch.document_id` (or span event attribute for span events) can be set explicitly by a processor based on the span or span event.
As an example, the `transform` processor can create this attribute dynamically for logs:
```yaml
processors:
transform/es-doc-id:
error_mode: ignore
log_statements:
- context: log
condition: attributes["event_name"] != null && attributes["event_creation_time"] != null
statements:
- set(attributes["elasticsearch.document_id"], Concat(["log", attributes["event_name"], attributes["event_creation_time"], "-"))
```
For traces, you can use the `transform` processor to set the document ID based on trace and span IDs to ensure uniqueness:
```yaml
exporters:
elasticsearch:
mapping:
mode: otel # Required for span events to be separate documents
traces_dynamic_id:
enabled: true
processors:
transform/es-doc-id-traces:
error_mode: ignore
trace_statements:
- context: span
statements:
# Set ID for spans
- set(attributes["elasticsearch.document_id"], Concat([trace_id.string, span_id.string], "-"))
- context: spanevent
statements:
# Set ID for span events (only works in otel mapping mode)
- set(attributes["elasticsearch.document_id"], Concat([trace_id.string, span_id.string, name], "-"))
```
**Note**: Span events are only stored as separate documents in `otel` mapping mode. In other mapping modes (ecs, bodymap, raw), span events are embedded within the span document and will not have separate document IDs.
## Known issues
### version_conflict_engine_exception
Symptom: `elasticsearchexporter` logs an error "failed to index document" with `error.type` "version_conflict_engine_exception" and `error.reason` containing "version conflict, document already exists".
This happens when the target data stream is a TSDB metrics data stream (e.g. using OTel mapping mode sending to a 8.16+ Elasticsearch, or ECS mapping mode sending to system integration data streams).
Elasticsearch [Time Series Data Streams](https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html) requires that there must only be one document per timestamp with the same dimensions.
The purpose is to avoid duplicate data when re-trying a batch of metrics that were previously sent but failed to be indexed.
The dimensions are mostly made up of resource attributes, scope attributes, scope name, attributes, and the unit.
The exporter can only group metrics with the same dimensions into the same document if they arrive in the same batch.
To ensure metrics are not dropped even if they arrive in different batches in the exporter, the exporter adds a fingerprint of the metric names to the document in the `otel` mapping mode.
Note that this functionality requires both
- minimum Elasticsearch Exporter version 0.121.0
- minimum Elasticsearch version 8.17.6, 8.18.1, 8.19.0, 9.0.1, or 9.1.0
If you are on an earlier version of Elasticsearch, either update your cluster or install this custom component template:
```shell
PUT _component_template/metrics-otel@custom
{
"template": {
"mappings": {
"properties": {
"_metric_names_hash": {
"type": "keyword",
"time_series_dimension": true
}
}
}
}
}
```
After installing this component template, if you've previously ingested data, you'll need to wait until the old index of the time series data stream reaches its `end_time`. This can take up to 30 minutes by default. See [time series index look-ahead time](https://www.elastic.co/docs/reference/elasticsearch/index-settings/time-series) for more information.
While in most situations, this error is just a sign that Elasticsearch's duplicate detection is working as intended, the data may be classified as a duplicate while it was not.
This implies data is lost.
1. If the data is not sent in `otel` mapping mode to `metrics-*.otel-*` data streams, the metrics name fingerprint is not applied.
This can happen for OTel host and k8s metrics that the [`elasticinframetricsprocessor`](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elasticinframetricsprocessor) has translated to the format the host and k8s dashboards in Kibana can consume.
If these metrics arrive in the `elasticsearchexporter` in different batches, they will not be grouped to the same document.
This can cause the `version_conflict_engine_exception` error.
Try to remove the `batchprocessor` from the pipeline (or set `send_batch_max_size: 0`) to ensure metrics are not split into different batches.
This gives the exporter the opportunity to group all related metrics into the same document.
2. Otherwise, check your metrics pipeline setup for misconfiguration that causes an actual violation of the [single writer principle](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#single-writer).
This means that the same metric with the same dimensions is sent from multiple sources, which is not allowed in the OTel metrics data model.
### flush failed (400) illegal_argument_exception
Symptom: bulk indexer logs an error that indicates "bulk indexer flush error" with bulk request returning HTTP 400 and an error type of `illegal_argument_exception`, similar to the following.
```
error elasticsearchexporter@v0.120.1/bulkindexer.go:343 bulk indexer flush error
{
"otelcol.component.id": "elasticsearch",
"otelcol.component.kind": "Exporter",
"otelcol.signal": "logs",
"error": "flush failed (400): {\"error\":{\"type\":\"illegal_argument_exception\",\"caused_by\":{}}}"
}
```
In this scenario, Elasticsearch may reject the bulk request because the `require_data_stream` bulk action metadata is not supported.
This may happen when you use [OTel mapping mode](#otel-mapping-mode) (the default mapping mode from v0.122.0, or explicitly by configuring `mapping::mode: otel`) or [ECS mapping mode](#ecs-mapping-mode), and send data to Elasticsearch version < 8.12.
To resolve this, upgrade Elasticsearch to 8.12+; for OTel mapping mode, 8.16+ is recommended.
Alternatively, try other mapping modes, but the document structure will be different.
### "dropping cumulative temporality histogram" and "dropping cumulative temporality exponential histogram"
Symptom: `elasticsearchexporter` logs a warning `dropping cumulative temporarily histogram` similar to:
```
warn elasticsearchexporter@v0.132.0/exporter.go:340 validation errors
{
"resource": {
"service.instance.id": "33ffe7e8-e944-4f92-8fce-9094f4b61d1d",
"service.name": "./elastic-agent",
"service.version": "9.1.5"
},
"otelcol.component.id": "elasticsearch/otel",
"otelcol.component.kind": "exporter",
"otelcol.signal": "metrics",
"error": "dropping cumulative temporality histogram \"http.client.request.duration\""
}
```
This issue occurs because Elasticsearch does not support **cumulative temporality** for histograms.
As a workaround, you can either:
- Export histogram metrics using **delta temporality**, or
- Apply a `cumulativetodelta` processor.
For more details, see [Metrics data ingestion](https://www.elastic.co/docs/reference/opentelemetry/compatibility/limitations#metrics-data-ingestion).
---
### Overview (`exporter/faroexporter/README.md`)
# Faro Exporter
| Status | |
| ------------- |-----------|
| Stability | [alpha]: logs, traces |
| Distributions | [contrib] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Ffaro) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Ffaro) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_faro&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@dehaansa](https://www.github.com/dehaansa), [@rlankfo](https://www.github.com/rlankfo), [@mar4uk](https://www.github.com/mar4uk) |
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
The Faro exporter sends telemetry data to a [Faro](https://grafana.com/oss/faro/) endpoint.
## Configuration
The following settings are required:
- `endpoint` (no default): The URL to send telemetry data to (e.g., https://faro.example.com/collect).
The following settings can be optionally configured:
- `timeout` (default = 30s), `sending_queue` and `retry_on_failure` settings as provided by [Exporter Helper](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper#configuration).
- `read_buffer_size` (default = 0): Size of the buffer used to read the response body.
- `write_buffer_size` (default = 512 KiB): Size of the buffer used to write the request body.
- `headers` (default = `{}`): Additional headers to send with the request.
- `compression` (default = gzip): Compression method to use for the request body. Supported values: `none`, `gzip`.
Example:
```yaml
exporters:
faro:
endpoint: https://faro.example.com/collect
timeout: 10s
headers:
X-API-Key: "my-api-key"
```
The full list of settings exposed for this exporter are documented [here](./config.go) with detailed sample configurations [here](./testdata/config.yaml).
## Getting Started
The following settings are required:
- `endpoint` (no default): URL to which the exporter is going to send Faro telemetry data. For example: `https://faro.example.com/collect`.
To use TLS, specify `https://` as the protocol scheme in the URL passed to the `endpoint` property.
See [Advanced Configuration](#advanced-configuration) for more TLS options.
Example:
```yaml
exporters:
faro:
endpoint: "https://faro.example.com/collect"
faro/tlsnoverify:
endpoint: "https://faro.example.com/collect"
tls:
insecure_skip_verify: true
```
## Propagating the `X-Faro-Session-Id` header
If using the faro exporter component to send telemetry data to the Grafana Cloud Collector Endpoint for [Frontend Observability](https://grafana.com/products/cloud/frontend-observability/), make sure to propagate the `X-Faro-Session-Id` header.
Grafana Cloud's Faro collector validates the session at the HTTP level and rejects requests that do
not carry an `X-Faro-Session-Id` header (HTTP 400, `missing X-Faro-Session-Id header`). The Faro Web
SDK sends this header on every request to the [Faro receiver][faroreceiver], but the exporter does
not forward incoming request headers automatically.
To propagate the session id from the incoming request onto the outgoing export, use the
[headers_setter extension][headerssetter] with `from_context`, and set `include_metadata: true` on
the receiver so the header is available to the pipeline:
```yaml
receivers:
faro:
include_metadata: true
cors:
allowed_origins:
- '*'
allowed_headers:
- 'X-Faro-Session-Id'
- 'Content-Type'
extensions:
headers_setter:
headers:
- action: insert
key: X-Faro-Session-Id
from_context: X-Faro-Session-Id
exporters:
faro:
endpoint: 'https://faro-collector-.grafana.net/collect/'
auth:
authenticator: headers_setter
service:
extensions: [headers_setter]
pipelines:
traces:
receivers: [faro]
exporters: [faro]
logs:
receivers: [faro]
exporters: [faro]
```
If a [batch processor][batchprocessor] is present in the pipeline, add `X-Faro-Session-Id` to its
`metadata_keys` so the metadata is preserved through batching.
When using [Grafana Alloy][alloy], the equivalent is the [`otelcol.auth.headers`][alloyauthheaders]
component with `from_context = "X-Faro-Session-Id"` and `include_metadata = true` on
`otelcol.receiver.faro`.
For a complete preprocessing example (including PII redaction and event filtering), see the Grafana
Cloud [Process Faro telemetry][processfaro] documentation.
[faroreceiver]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/faroreceiver
[headerssetter]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/headerssetterextension
[batchprocessor]: https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor#batching-and-client-metadata
[alloy]: https://grafana.com/docs/alloy/latest/
[alloyauthheaders]: https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.auth.headers/
[processfaro]: https://grafana.com/docs/grafana-cloud/observe-and-act/monitor-applications/frontend-observability/configure/process-faro-telemetry/
## Advanced Configuration
Several helper files are leveraged to provide additional capabilities automatically:
- [HTTP client settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/README.md#client-configuration)
- [TLS and mTLS settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/configtls/README.md)
- [Queuing, retry and timeout settings](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)
---
### Overview (`exporter/fileexporter/README.md`)
# File Exporter
| Status | |
| ------------- |-----------|
| Stability | [development]: profiles |
| | [alpha]: traces, metrics, logs |
| Distributions | [core], [contrib], [k8s] |
| Issues | [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aexporter%2Ffile) [](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aexporter%2Ffile) |
| Code coverage | [](https://app.codecov.io/gh/open-telemetry/opentelemetry-collector-contrib/tree/main/?components%5B0%5D=exporter_file&displayType=list) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@paulojmdias](https://www.github.com/paulojmdias) \| Seeking more code owners! |
| Emeritus | [@atingchen](https://www.github.com/atingchen) |
[development]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#development
[alpha]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md#alpha
[core]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol
[contrib]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib
[k8s]: https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-k8s
Writes telemetry data to files on disk.
Use the [OTLP JSON File receiver](../../receiver/otlpjsonfilereceiver/README.md) to read the data back into the collector. It only reads uncompressed, newline-delimited OTLP JSON (the default `format: json` without `compression` or `encoding`, see [File Format](#file-format)).
The exporter supports the following features:
+ Writing pipeline data to a file.
+ Rotation of telemetry files.
+ Compressing the telemetry data before exporting.
+ Writing into multiple files, where the file path is determined by a resource attribute.
Please note that there is no guarantee that exact field names will remain stable.
The official [opentelemetry-collector-contrib container](https://hub.docker.com/r/otel/opentelemetry-collector-contrib/tags#!) does not have a writable filesystem by default since it's built on the `scratch` layer.
As such, you will need to create a writable directory for the path. You could do this by [mounting a volume](https://docs.docker.com/storage/volumes/#choose-the--v-or---mount-flag) with flags such as `rw` or `rwZ`.
On Linux, and given a `otel-collector-config.yaml` with a `file` exporter whose path is prefixed with `/file-exporter`,
```bash
# linux needs +x to list a directory. You can use a+ instead of o+ for the mode if you want to ensure your user and group has access.
mkdir --mode o+rwx file-exporter
# z is an SELinux construct that is ignored on other systems
docker run -v "./file-exporter:/file-exporter:rwz" -v "otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml" otel/opentelemetry-collector-contrib:latest
```
Note this same syntax for volumes will work with docker-compose.
You could also modify the base image and manually build your own container to have a writeable directory or change the runas uid if needed, but this is more involved.
## Configuration options:
The following settings are required:
- `path` [no default]: where to write information.
The following settings are optional:
- `rotation` settings to rotate telemetry files. Rotation is only enabled when `rotation:` is present in the config.
- max_megabytes: [default: 100]: the maximum size in megabytes of the telemetry file before it is rotated.
- max_days: [no default (unlimited)]: the maximum number of days to retain rotated telemetry files based on the timestamp encoded in their filename. This only controls retention, it does not trigger rotation.
- max_backups: [default: 100]: the maximum number of rotated telemetry files to retain.
- localtime : [default: false (use UTC)] whether or not the timestamps in backup files is formatted according to the host's local time.
- `format`[default: json]: the data format of encoded telemetry data, either `json` or `proto`.
- `encoding`[default: none]: if specified, uses an encoding extension to encode telemetry data instead of the built-in `format` marshaler. How records are delimited in the file still depends on `format`, see [File Format](#file-format).
- `append`[default: `false`] defines whether append to the file (`true`) or truncate (`false`). `append: true` cannot be combined with `rotation`.
- `compression`[no default]: the compression algorithm used when exporting telemetry data to file. Supported compression algorithms:`zstd`
- `compression_params`
- `level` (default = 0, which uses the zstd default level `3`): the compression level used when exporting telemetry data. Only applied when the `exporter.file.nativeCompression` feature gate is enabled, see [Known limitations](#known-limitations).
- The following are valid combinations of `compression` and `level`:
- `zstd`
- SpeedFastest: `1`
- SpeedDefault: `3`
- SpeedBetterCompression: `6`
- SpeedBestCompression: `11`
- `flush_interval`[default: 1s]: `time.Duration` interval between flushes of the write buffer. See [time.ParseDuration](https://pkg.go.dev/time#ParseDuration) for valid formats.
- A value without a unit is in nanoseconds, so always specify one (for example `5s`).
- `0` is treated as the default (`1s`); buffering cannot be disabled.
- Ignored when `rotation` is set and the `exporter.file.nativeCompression` feature gate is disabled, as writes are not buffered in that case.
- With the `exporter.file.nativeCompression` feature gate enabled and without `rotation`, each flush finalizes a zstd frame. With `rotation`, each batch is written as its own frame and flushing is a no-op.
- `create_directory`[default: false]: when set, the exporter will create the parent directory of the configured `path` if it does not exist.
- `directory_permissions`[default: 0755]: file mode (octal string) used when creating directories, minus the process umask. Can only be set when `create_directory` is `true`. This also applies to directories created by `group_by`.
- `group_by` enables writing to separate files based on a resource attribute.
- enabled: [default: false] enables group_by. When enabled, `path` must contain exactly one `*` and must not start with it.
- resource_attribute: [default: fileexporter.path_segment]: specifies the name of the resource attribute that contains the path segment of the file to write to. The final path will be the `path` config value, with the `*` replaced with the value of this resource attribute.
- max_open_files: [default: 100]: specifies the maximum number of open file descriptors for the output files.
## File Rotation
Telemetry data is exported to a single file by default.
`fileexporter` only enables file rotation when the user specifies `rotation:` in the config. However, if specified, related default settings would apply.
Telemetry is first written to a file that exactly matches the `path` setting.
When writing the next batch would make the file exceed `max_megabytes`, the file is rotated. Rotation is size-based only; `max_days` and `max_backups` only control how many rotated files are kept.
When a file is rotated, **it is renamed by putting the current time and the rotation reason**
in the name immediately before the file's extension (or the end of the filename if there's no extension).
**A new telemetry file will be created at the original `path`.**
For example, if your `path` is `data.json` and rotation is triggered, this file will be renamed to `data-2022-09-14T05-02-14.173-size.json`, and a new telemetry file created with `data.json`.
Rotated files from older collector versions (named without the `-size` suffix, e.g. `data-2022-09-14T05-02-14.173.json`) are renamed to the new format on startup, so `max_days` and `max_backups` keep applying to them.
## File Compression
Telemetry data is compressed according to the `compression` setting.
`fileexporter` does not compress data by default.
> [!NOTE]
> An alpha feature gate `exporter.file.nativeCompression` is available that switches from
> per-message compression to native file-level compression, producing standard `.zst` files
> compatible with tools like `zstd -d`. See [Feature Gates](documentation.md) for details.
Currently, `fileexporter` support the `zstd` compression algorithm, and we will support more compression algorithms in the future.
## File Format
Telemetry data is encoded according to the `format` (or `encoding`) setting and then written to the file. How each batch is delimited depends on `format`, `encoding`, `compression` and the `exporter.file.nativeCompression` feature gate:
| `exporter.file.nativeCompression` | `compression` | `format` / `encoding` | Written to the file |
| --------------------------------- | ------------- | --------------------- | ------------------- |
| any | none | `json`, no `encoding` | Newline-delimited OTLP JSON, one line per batch |
| any | none | `json` + `encoding` | Encoder output followed by `\n` per batch (no length prefix) |
| any | none | `proto`, with or without `encoding` | Length-prefixed messages |
| disabled | `zstd` | any | Each batch compressed on its own, then length-prefixed. Not readable by `zstd -d` |
| enabled | `zstd` | `json`, no `encoding` | Newline-delimited OTLP JSON inside a standard zstd stream |
| enabled | `zstd` | `proto`, or any `encoding` | Length-prefixed messages inside a standard zstd stream |
Length-prefixed means each encoded message is preceded by 4 bytes (a big-endian unsigned 32 bit integer) holding the number of bytes in the message. To read the messages back, read the size, then read that many bytes into a separate buffer and parse from that buffer.
When using an `encoding` extension that produces binary output without compression, set `format: proto` to get length-prefixed framing; otherwise records can't be reliably split back apart.
## Group by attribute
By specifying `group_by.resource_attribute` in the config, the exporter will determine a filepath for each telemetry record, by substituting the value of the resource attribute into the `path` configuration value.
The final path is guaranteed to start with the prefix part of the `path` config value (the part before the `*` character). For example if `path` is "/data/*.json", and the resource attribute value is "../etc/my_config", then the final path will be sanitized to "/data/etc/my_config.json".
The final path can contain path separators (`/`). The exporter will create missing directories recursively (similarly to `mkdir -p`), regardless of `create_directory`.
Resources that do not have the attribute, or where its value is not a string, are dropped.
When more than `max_open_files` files are in use, the least recently used file is closed and reopened on its next write. Unless `append: true` or `rotation` is set, reopening truncates the file and its earlier content is lost, so keep `max_open_files` above the number of distinct attribute values or set `append: true`.
Grouping by attribute currently only supports a **single** **resource** attribute. If you would like to use multiple attributes, please use [Transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor) create a routing key. If you would like to use a non-resource level (eg: Log/Metric/DataPoint) attribute, please use [Group by Attributes processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/groupbyattrsprocessor) first.
## Known limitations
- `append` and `rotation` cannot be used together.
- With `rotation` and the `exporter.file.nativeCompression` feature gate disabled:
- writes are not buffered and `flush_interval` has no effect;
- a single batch larger than `max_megabytes` is rejected with an error. With the feature gate enabled, such a batch is split across multiple zstd frames instead.
- `group_by` drops resources without the configured attribute (or with a non-string value), logging this only at `debug` level.
- `group_by` always creates missing directories, whatever the value of `create_directory`.
- With `group_by`, a file evicted because of `max_open_files` is truncated when reopened, unless `append: true` or `rotation` is set.
- Without the `exporter.file.nativeCompression` feature gate, `compression_params.level` is ignored and the zstd default level is used.
- Files are not fsynced. Written data may be lost if the host crashes before the OS flushes it to disk.
- The exporter does not support `sending_queue`, `retry_on_failure` or `timeout`.
## Telemetry
The exporter does not emit metrics of its own. Use the standard exporter metrics to monitor it, such as `otelcol_exporter_sent_spans`, `otelcol_exporter_sent_metric_points`, `otelcol_exporter_sent_log_records` and their `otelcol_exporter_send_failed_*` counterparts.
## Example:
```yaml
exporters:
file/no_rotation:
path: ./foo
file/rotation_with_default_settings:
path: ./foo
rotation:
file/rotation_with_custom_settings:
path: ./foo
rotation:
max_megabytes: 10
max_days: 3
max_backups: 3
localtime: true
format: proto
compression: zstd
compression_params:
level: 6
file/flush_every_5_seconds:
path: ./foo
flush_interval: 5s
# Writes one file per service, e.g. ./data/checkout.json, ./data/cart.json
file/group_by_service:
path: ./data/*.json
group_by:
enabled: true
resource_attribute: service.name
max_open_files: 50
```
Native file-level compression, readable with `zstd -d` (requires starting the collector with `--feature-gates=exporter.file.nativeCompression`):
```yaml
exporters:
file/native_zstd:
path: ./foo.jsonl.zst
compression: zstd
compression_params:
level: 3
```
## Get Started in an existing cluster
We will follow the [documentation](https://opentelemetry.io/docs/k8s-operator/) to first install the operator in an existing cluster
and then create an OpenTelemetry Collector (otelcol) instance,
mounting an additional volume under `/data` under which the file exporter will write `metrics.json`:
``` shell
kubectl apply -f - <Alternatives
If you obtained OS-specific packages or built your own binary in step 1, you'll need to follow the appropriate conventions for running the collector.
5. **Gather telemetry.** Run an application that can submit OTLP-formatted metrics and traces, and configure it to send them to `127.0.0.1:4317` (for gRPC) or `127.0.0.1:55681` (for HTTP).
Alternatives
* Set up the host metrics receiver, which will gather telemetry from the host without needing an external application to submit telemetry.
* Set up an application-specific receiver, such as the Nginx receiver, and run the corresponding application.
* Set up a receiver for some other protocol (such Prometheus, StatsD, Zipkin or Jaeger), and run an application that speaks one of those protocols.
6. **View telemetry in GCP.** Use the GCP [metrics explorer](https://console.cloud.google.com/monitoring/metrics-explorer) and [trace overview](https://console.cloud.google.com/traces) to view your newly submitted telemetry.
## Configuration reference
The following configuration options are supported:
- `project` (default = Fetch from Credentials): GCP project identifier.
- `destination_project_quota` (optional, default = false): Counts quota against the project to which the data is sent (as opposed to the project associated with the Collector's service account. For example, when setting `project_id` or using [multi-project export](#multi-project-exporting).
- `user_agent` (default = `collector description/version os/arch`, i.e. `opentelemetry-collector-contrib/v0.139.0 linux/amd64`): Override the user agent string sent on requests to Cloud Monitoring (currently only applies to metrics). Specify `{{version}}` to include the application version number.
- `timeout` (default = `12s`) The timeout for requests to Google Cloud Platform APIs, specified in Go Time Duration format.
- `impersonate` (optional): Configuration for service account impersonation
- `target_principal`: TargetPrincipal is the email address of the service account to impersonate.
- `subject`: (optional) Subject is the sub field of a JWT. This field should only be set if you wish to impersonate as a user. This feature is useful when using domain wide delegation.
- `delegates`: (default = []) Delegates are the service account email addresses in a delegation chain. Each service account must be granted roles/iam.serviceAccountTokenCreator on the next service account in the chain.
- `metric` (optional): Configuration for sending metrics to Google Cloud Monitoring.
- `prefix` (default = `workload.googleapis.com`): The prefix to add to metrics.
- `endpoint` (default = `monitoring.googleapis.com`): Endpoint where metric data is going to be sent to.
- `use_insecure` (default = false): If true, disables gRPC client transport security. Only has effect if Endpoint is not "".
- `compression` (optional, supported values: [`gzip`]): Compression format for Metrics gRPC requests. Defaults to no compression.
- `grpc_pool_size` (optional): Sets the size of the connection pool in the GCP client. Defaults to a single connection.
- `known_domains` (default = [googleapis.com, kubernetes.io, istio.io, knative.dev]): If a metric belongs to one of these domains it does not get a prefix.
- `skip_create_descriptor` (default = false): If set to true, do not send metric descriptors to Google Cloud Monitoring.
- `instrumentation_library_labels` (default = true): If true, the exporter will copy the OTLP `InstrumentationScope.Name` to a label `instrumentation_source` and `InstrumentationScope.Version` to a label `instrumentation_version` labels on metrics.
- `service_resource_labels` (default = true): If true, the exporter will copy the Semantic Conventions `service.name`, `service.namespace`, and `service.instance.id` from OTLP Resource Attributes into the Google Cloud Monitoring timeseries metric labels. These labels will be the same as the Semantic Conventions with `.` replaced by `_`.
- `create_metric_descriptor_buffer_size` (default = 10): Buffer size for the channel which asynchronously calls CreateMetricDescriptor.
- `resource_filters` (default = []): If provided, resource attributes matching any filter will be included in metric labels. Can be defined by `prefix`, `regex`, or `prefix` AND `regex`.
- `prefix`: Match resource keys by prefix.
- `regex`: Match resource keys by regex.
- `cumulative_normalization` (default = true): If true, normalizes cumulative metrics without start times or with explicit reset points by subtracting subsequent points from the initial point. It is enabled by default. Since it caches starting points, it may result in increased memory usage.
- `sum_of_squared_deviation` (default = false): If true, enables calculation of an estimated sum of squared deviation. It is an estimate, and is not exact.
- `create_service_timeseries` (default = false): If true, this will send all timeseries using `CreateServiceTimeSeries`. Implicitly, this sets `skip_create_descriptor` to true.
- `experimental_wal` (default = []): If provided, enables use of a write ahead
log for time series requests.
- `directory` (default = `./`): Path to local directory for WAL file.
- `max_backoff` (default = `1h`): Max duration to retry requests on network errors (`UNAVAILABLE` or `DEADLINE_EXCEEDED`).
- `trace` (optional): Configuration for sending traces to Google Cloud Trace.
- `endpoint` (default = `cloudtrace.googleapis.com`): Endpoint where trace data is going to be sent to.
- `use_insecure` (default = false): If true, disables gRPC client transport security. Only has effect if Endpoint is not "".
- `grpc_pool_size` (optional): Sets the size of the connection pool in the GCP client. Defaults to a single connection.
- `attribute_mappings` (optional): AttributeMappings determines how to map from OpenTelemetry attribute keys to Google Cloud Trace keys. By default, it changes `http` and `service` keys so that they appear more prominently in the UI.
- `key`: The OpenTelemetry attribute key
- `replacement`: The replacement attribute sent to Google Cloud Trace
- `log` (optional): Configuration for sending logs to Google Cloud Logging.
- `endpoint` (default = `logging.googleapis.com`): Endpoint where log data is going to be sent to.
- `use_insecure` (default = false): If true, disables gRPC client transport security. Only has effect if Endpoint is not "".
- `compression` (optional, Supported values: [`gzip`]): Compression format for Logs gRPC requests. Defaults to no compression.
- `grpc_pool_size` (optional): Sets the size of the connection pool in the GCP client. Defaults to a single connection.
- `default_log_name` (optional): Defines a default name for log entries. If left unset, and a log entry does not have the `gcp.log_name` attribute set, the exporter will return an error processing that entry.
- `resource_filters` (default = []): If provided, Resource Attributes matching any filter will be included in log entry labels. Can be defined by `prefix`, `regex`, or `prefix` AND `regex`.
- `prefix`: Match resource keys by prefix.
- `regex`: Match resource keys by regex.
- `sending_queue` (optional): Configuration for how to buffer data before sending. Note: The `sending_queue` is provided (and documented) by the [Exporter Helper](https://github.com/open-telemetry/opentelemetry-collector/tree/main/exporter/exporterhelper#sending-queue)
Beyond standard YAML configuration as outlined in the section above,
exporters that leverage the net/http package (all do today) also respect the
following proxy environment variables:
* HTTP_PROXY
* HTTPS_PROXY
* NO_PROXY
If set at Collector start time then exporters, regardless of protocol,
will or will not proxy traffic as defined by these environment variables.
### Monitored Resources
For metrics and logs, this exporter maps the OpenTelemetry Resource to a Google
Cloud [Logging](https://cloud.google.com/logging/docs/api/v2/resource-list) or
[Monitoring](https://cloud.google.com/monitoring/api/resources) Monitored Resource.
The complete mapping logic can be found in [resourcemapping.go](https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/main/internal/resourcemapping/resourcemapping.go).
That may be the most helpful reference if you want to map to a specific monitored
resource.
#### On GCP
If running on GCP, using the GCP resource detector, as shown above, will populate
the resource attributes required to map to the appropriate monitored resource.
#### Off GCP
If you are not running on GCP, you still need to choose a [GCP zone or
region](https://cloud.google.com/compute/docs/regions-zones) to send telemetry to
by setting `cloud.availability_zone` or `cloud.region`. In addition, you should use the detector associated with other cloud providers, if applicable.
If running on Kubernetes, it is recommended to additionally set `k8s.pod.name`,
`k8s.namespace.name`, and `k8s.container.name` using the `k8sattributes` processor.
If you are getting "duplicate timeseries encountered" errors, it is likely because
you are missing a required resource attribute, causing a metric from two different
instances of an application to end up with the same monitored resource.
### Preventing metric label collisions
The metrics exporter can add metric labels to timeseries, such as when setting
`metric.service_resource_labels`, `metric.instrumentation_library_labels` (both
on by default), or when using `metric.resource_filters` to convert resource
attributes to metric labels.
However, if your metrics already contain any of these labels they will fail to
export to Google Cloud with a `Duplicate label key encountered` error. Such
labels from the default features above include:
* `service_name`
* `service_namespace`
* `service_instance_id`
* `instrumentation_source`
* `instrumentation_version`
*(Note that these are the sanitized versions of OpenTelemetry attributes, with `.` replaced by `_` to be compatible with Cloud Monitoring. For example, `service_name` comes from the [`service.name` resource attribute](https://github.com/open-telemetry/opentelemetry-specification/blob/dc78006c12d9767fd2e35b691706c7572a76fd43/specification/resource/semantic_conventions/README.md#service).)*
To prevent this, it's recommended to use the [transform processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/c7bd50ce773e66be327ef7618775a884a774e5d1/processor/transformprocessor) in your collector config to rename existing metric labels to preserve them, for example:
```yaml
processors:
transform:
metric_statements:
- context: datapoint
statements:
- set(attributes["exported_service_name"], attributes["service_name"])
- delete_key(attributes, "service_name")
- set(attributes["exported_service_namespace"], attributes["service_namespace"])
- delete_key(attributes, "service_namespace")
- set(attributes["exported_service_instance_id"], attributes["service_instance_id"])
- delete_key(attributes, "service_instance_id")
- set(attributes["exported_instrumentation_source"], attributes["instrumentation_source"])
- delete_key(attributes, "instrumentation_source")
- set(attributes["exported_instrumentation_version"], attributes["instrumentation_version"])
- delete_key(attributes, "instrumentation_version")
```
**Note** It is not recommended to use these transformations with the googlecloud
exporter in a logging or trace pipeline.
The same method can be used for any resource attributes being filtered to metric
labels, or metric labels which might collide with the GCP monitored resource
used with resource detection.
Keep in mind that your conflicting attributes may contain dots instead of
underscores (eg, `service.name`), but these will still collide once all
attributes are normalized to metric labels. In this case you will need to update
the collector config above appropriately.
### Logging Example
The logging exporter processes OpenTelemetry log entries and exports them to GCP Cloud Logging. Logs can be collected using one
of the opentelemetry-collector-contrib log receivers, such as the [filelogreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver).
Log entries must contain any Cloud Logging-specific fields as a matching OpenTelemetry attribute (as shown in examples from the
[logs data model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md#google-cloud-logging)).
These attributes can be parsed using the various [log operators](../../pkg/stanza/docs/operators/README.md#what-operators-are-available) available upstream.
For example, the following config parses the [HTTPRequest field](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest) from Apache log entries saved in `/var/log/apache.log`.
It also parses out the `timestamp` and inserts a non-default `log_name` attribute and GCP [MonitoredResource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource) attribute.
```yaml
receivers:
file_log:
include: [ /var/log/apache.log ]
start_at: beginning
operators:
- id: http_request_parser
type: regex_parser
regex: '(?m)^(?P[^ ]*) (?P[^ ]*) (?P[^ ]*) \[(?P
