# Technical Documentation: open-telemetry/opentelemetry-specification > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/open-telemetry/opentelemetry-specification) ### Overview (`README.md`) # OpenTelemetry Specification [](https://cloud-native.slack.com/archives/C01N7PP1THC) [](https://github.com/open-telemetry/opentelemetry-specification/releases/) [](https://scorecard.dev/viewer/?uri=github.com/open-telemetry/opentelemetry-specification) [](https://www.bestpractices.dev/projects/11058) The **[OpenTelemetry][] specification** describes the cross-language requirements and expectations for all OpenTelemetry implementations. - The [latest release][] is hosted at [opentelemetry.io/docs/specs/otel][] - Markdown sources are under [specification](./specification/README.md) ## Change / contribution process For details, see [CONTRIBUTING.md](CONTRIBUTING.md), in particular read [Proposing a change](CONTRIBUTING.md#proposing-a-change) before submitting a PR. ## Questions New to CNCF Slack? [Create an account](https://slack.cncf.io/) first, then join the [`#otel-specification`](https://cloud-native.slack.com/archives/C01N7PP1THC) channel for questions and discussion. Questions that need additional attention can be brought to the regular specifications meeting. EU and US timezone friendly meeting is held every Tuesday at 8 AM Pacific time. Meeting notes are held in the [Google doc](https://docs.google.com/document/d/1pdvPeKjA8v8w_fGKAN68JjWBmVJtPCpqdi9IZrd6eEo/edit). APAC timezone friendly meetings are held on request. See [OpenTelemetry calendar](https://github.com/open-telemetry/community#calendar). Escalations to technical committee may be made over the [email](https://github.com/open-telemetry/community/blob/main/README.md#mailing-lists). Technical committee holds regular meetings, notes are held [here](https://docs.google.com/document/d/1hOHPCu5TGenqTeWPB9qQB_qd33uITZBcvK1FnWxYJAw/edit?usp=sharing). ## Specification compliance matrix by language See [Compliance of Implementations with Specification](./spec-compliance-matrix.md). ## Project Timeline The current project status as well as information on notable past releases is found at [the OpenTelemetry project page](https://opentelemetry.io/status/). Information about current work and future development plans is found at the [specification development milestones](https://github.com/open-telemetry/opentelemetry-specification/milestones). ## Versioning the Specification Changes to the [specification](./specification/overview.md) are versioned according to [Semantic Versioning 2.0](https://semver.org/spec/v2.0.0.html) and described in [CHANGELOG.md](CHANGELOG.md). Layout changes are not versioned. Specific implementations of the specification should specify which version they implement. Changes to the change process itself are not currently versioned but may be independently versioned in the future. ## License By contributing to OpenTelemetry Specification repository, you agree that your contributions will be licensed under its [Apache 2.0 License](LICENSE). [OpenTelemetry]: https://opentelemetry.io [latest release]: https://github.com/open-telemetry/opentelemetry-specification/releases/latest [opentelemetry.io/docs/specs/otel]: https://opentelemetry.io/docs/specs/otel/ ## Maintainers The maintainers of the specification are a superset of the Technical Committee. - [OpenTelemetry Technical Committee](https://github.com/open-telemetry/community/blob/main/community-members.md#technical-committee) - [Josh Suereth](https://github.com/jsuereth), Google - [Robert Pająk](https://github.com/pellared), Splunk For more information about the maintainer role, see the [community repository](https://github.com/open-telemetry/community/blob/main/guides/contributor/membership.md#maintainer). ## Approvers [Specification sponsors](https://github.com/open-telemetry/community/blob/main/guides/contributor/membership.md#specification-sponsor) represent the approver role for the specification, along with additional responsibilities and privileges. - [Alex Boten](https://github.com/codeboten), Grafana Labs - [Christian Neumüller](https://github.com/Oberon00), Dynatrace - [Cijo Thomas](https://github.com/cijothomas), Microsoft - [Daniel Dyla](https://github.com/dyladan), Dynatrace - [Josh Suereth](https://github.com/jsuereth), Google - [Juraci Paixão Kröhling](https://github.com/jpkrohling), OllyGarden - [Leighton Chen](https://github.com/lzchen), Microsoft - [Marc Alff](https://github.com/marcalff), Oracle - [Severin Neumann](https://github.com/svrnm), Independent - [Ted Young](https://github.com/tedsuo), Grafana Labs - [Tristan Sloughter](https://github.com/tsloughter), MyDecisiveAI - [Tyler Yahn](https://github.com/MrAlias), Splunk For more information about the approver role, see the [community repository](https://github.com/open-telemetry/community/blob/main/guides/contributor/membership.md#approver). ## Triagers - [OpenTelemetry Governance Committee](https://github.com/open-telemetry/community/blob/main/community-members.md#governance-committee) For more information about the triager role, see the [community repository](https://github.com/open-telemetry/community/blob/main/guides/contributor/membership.md#triager). ## Emeritus - [Christian Beedgen](https://github.com/kumoroku), Approver - [Daniel Jaglowski](https://github.com/djaglowski), Approver - [David Poncelow](https://github.com/zenmoto), Approver - [Nikita Salnikov-Tarnovski](https://github.com/iNikem), Approver --- ### Overview (`development/README.md`) # Features in Development This folder is used for features in [Development](../specification/maturity-levels.md#development). Development status is for components that are on track to become part of the specification, but that require a faster cadence of merges and collaboration. Features in Development must be: - Implementable as a plugin to OpenTelemetry components (API, SDK, collector, etc.). - Be in active development or testing. - Approved as a general direction via OTEP process. To avoid any confusion, all files in this directory must have a note about its Development status. Development status precedes the [Alpha](../specification/maturity-levels.md#alpha) maturity level. All changes in the `development` folder go through the regular review process. Changes are allowed to be merged faster as completeness of a solution is not a requirement. Approval means that proposed changes are OK for experimentation. When the feature or parts of it are developed far enough to declare them as an alpha version of a main project and move out of the Development status, it must go through a **new** OTEP PR and it must be expected that design and APIs will be changed. In fact, the same people who approved the experiment may likely be the most critical reviewers. It demonstrates an interest and involvement, not critique. --- ### Overview (`oteps/README.md`) # OpenTelemetry Enhancement Proposal (OTEP) ## Evolving OpenTelemetry at the speed of Markdown OpenTelemetry uses an "OTEP" (similar to a RFC) process for proposing changes to the OpenTelemetry Specification. > [!NOTE] > OTEPs are documents of intent. They become requirements only after their contents have been > [integrated](#integrating-the-otep-into-the-spec) into the actual [Specification](../specification). ### Table of Contents - [OpenTelemetry Enhancement Proposal (OTEP)](#opentelemetry-enhancement-proposal-otep) - [Evolving OpenTelemetry at the speed of Markdown](#evolving-opentelemetry-at-the-speed-of-markdown) - [Table of Contents](#table-of-contents) - [What changes require an OTEP](#what-changes-require-an-otep) - [Extrapolating cross-cutting changes](#extrapolating-cross-cutting-changes) - [OTEP scope](#otep-scope) - [Writing an OTEP](#writing-an-otep) - [Submitting the OTEP](#submitting-the-otep) - [Integrating the OTEP into the Spec](#integrating-the-otep-into-the-spec) - [Implementing the OTEP](#implementing-the-otep) - [Changes to the OTEP process](#changes-to-the-otep-process) - [Background on the OpenTelemetry OTEP process](#background-on-the-opentelemetry-otep-process) ### What changes require an OTEP The OpenTelemetry OTEP process is intended for changes that are **cross-cutting** - that is, applicable across *languages* and *implementations* - and either **introduce new behaviour**, **change desired behaviour**, or otherwise **modify requirements**. In practice, this means that OTEPs should be used for such changes as: - New tracer configuration options - Additions to span data - New metric types - Modifications to extensibility requirements On the other hand, they do not need to be used for such changes as: - Bug fixes - Rephrasing, grammatical fixes, typos, etc. - Refactoring - Things that affect only a single language or implementation **Note:** The above lists are intended only as examples and are not meant to be exhaustive. If you don't know whether a change requires an OTEP, please feel free to ask! #### Extrapolating cross-cutting changes Sometimes, a change that is only immediately relevant within a single language or implementation may be indicative of a problem upstream in the specification. We encourage you to add an OTEP if and when you notice such cases. ### OTEP scope While OTEPs are intended for "significant" changes, we recommend trying to keep each OTEP's scope as small as makes sense. A general rule of thumb is that if the core functionality proposed could still provide value without a particular piece, then that piece should be removed from the proposal and used instead as an *example* (and, ideally, given its own OTEP!). For example, an OTEP proposing configurable sampling *and* various samplers should instead be split into one OTEP proposing configurable sampling as well as an OTEP per sampler. ### Writing an OTEP - First, [fork](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/fork-a-repo) this repository. - Copy [`0000-template.md`](./0000-template.md) to `0000-my-OTEP.md`, where `my-OTEP` is a title relevant to your proposal, and `0000` is the OTEP ID. Leave the number as is for now. Once a Pull Request is made, update this ID to match the PR ID. - Fill in the template. Put care into the details: It is important to present convincing motivation, demonstrate an understanding of the design's impact, and honestly assess the drawbacks and potential alternatives. ### Submitting the OTEP - An OTEP is `proposed` by posting it as a PR. Once the PR is created, update the OTEP filename to use the PR ID as the OTEP ID. - An OTEP is `approved` when four reviewers github-approve the PR. The OTEP is then merged. - If an OTEP is `rejected` or `withdrawn`, the PR is closed. Note that these OTEPs submissions are still recorded, as GitHub retains both the discussion and the proposal, even if the branch is later deleted. - If an OTEP discussion becomes long, and the OTEP then goes through a major revision, the next version of the OTEP can be posted as a new PR, which references the old PR. The old PR is then closed. This makes OTEP review easier to follow and participate in. ### Integrating the OTEP into the Spec - Once an OTEP is `approved`, an issue is created in this repository to integrate the OTEP into the actual spec. - When reviewing the spec PR for the OTEP, focus on whether the spec is written clearly, and reflects the changes approved in the OTEP. Please abstain from relitigating the approved OTEP changes at this stage. - An OTEP is `integrated` when four reviewers github-approve the spec PR. The PR is then merged, and the spec is versioned. ### Implementing the OTEP - Once an OTEP is `integrated` into the spec, an issue is created in the backlog of every relevant OpenTelemetry implementation. - PRs are made until the all the requested changes are implemented. - The status of the OpenTelemetry implementation is updated to reflect that it is implementing a new version of the spec. ## Changes to the OTEP process The hope and expectation is that the OTEP process will **evolve** with the OpenTelemetry. The process is by no means fixed. Have suggestions? Concerns? Questions? **Please** raise an issue or raise the matter on our [community](https://github.com/open-telemetry/community) chat. ## Background on the OpenTelemetry OTEP process Our OTEP process borrows from the [Rust RFC](https://github.com/rust-lang/rfcs) and [Kubernetes Enhancement Proposal](https://github.com/kubernetes/enhancements) processes, the former also being [very influential](https://github.com/kubernetes/enhancements/blob/master/keps/sig-architecture/0000-kep-process/README.md#prior-art) on the latter; as well as the [OpenTracing OTEP](https://github.com/opentracing/specification/blob/master/rfc_process.md) process. Massive kudos and thanks to the respective authors and communities for providing excellent prior art 💖 --- ### Overview (`specification/README.md`) # OpenTelemetry Specification ## Contents - [Overview](overview.md) - [Glossary](glossary.md) - Principles and Guidelines - [Core Principles](specification-principles.md) - [Versioning and stability for OpenTelemetry clients](versioning-and-stability.md) - [Library Guidelines](library-guidelines.md) - [Package/Library Layout](library-layout.md) - [General error handling guidelines](error-handling.md) - [Self-Observability](self-observability.md) - [Performance](performance.md) - API Specification - [Context](context/README.md) - [Propagators](context/api-propagators.md) - [Environment Variable Carriers](context/env-carriers.md) - [Baggage](baggage/api.md) - [Tracing](trace/api.md) - [Metrics](metrics/api.md) - [Logs](logs/README.md) - [API](logs/api.md) - SDK Specification - [Tracing](trace/sdk.md) - [Metrics](metrics/sdk.md) - [Logs](logs/sdk.md) - [Resource](resource/sdk.md) - [Configuration](configuration/README.md) - Data Specification - [Semantic Conventions](overview.md#semantic-conventions) - [Protocol](protocol/README.md) - [Metrics](metrics/data-model.md) - [Logs](logs/data-model.md) - [Profiles](profiles/mappings.md) - Compatibility - [OpenCensus](compatibility/opencensus.md) - [OpenTracing](compatibility/opentracing.md) - [Prometheus and OpenMetrics](compatibility/prometheus_and_openmetrics.md) - [Trace Context in non-OTLP Log Formats](compatibility/logging_trace_context.md) ## Notation Conventions and Compliance The keywords "MUST", "MUST NOT", "REQUIRED", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in the [specification][] are to be interpreted as described in [BCP 14](https://www.rfc-editor.org/info/bcp14) [[RFC2119](https://datatracker.ietf.org/doc/html/rfc2119)] [[RFC8174](https://datatracker.ietf.org/doc/html/rfc8174)] when, and only when, they appear in all capitals, as shown here. An implementation of the [specification][] is not compliant if it fails to satisfy one or more of the "MUST", "MUST NOT", "REQUIRED", requirements defined in the [specification][]. Conversely, an implementation of the [specification][] is compliant if it satisfies all the "MUST", "MUST NOT", "REQUIRED", requirements defined in the [specification][]. ## Project Naming - The official project name is "OpenTelemetry" (with no space between "Open" and "Telemetry"). - The official acronym used by the OpenTelemetry project is "OTel". "OT" MAY be used only as a part of a longer acronym, such as OTCA (OpenTelemetry Certified Associate). - The official names for sub-projects, like language specific implementations, follow the pattern of "OpenTelemetry {the name of the programming language, runtime or component}", for example, "OpenTelemetry Python", "OpenTelemetry .NET" or "OpenTelemetry Collector". ## About the project See the [project repository][] for information about the following, and more: - [Change / contribution process](../README.md#change--contribution-process) - [Project timeline](../README.md#project-timeline) - [Versioning the specification](../README.md#versioning-the-specification) - [License](../README.md#license) [project repository]: https://github.com/open-telemetry/opentelemetry-specification [specification]: overview.md --- ### Overview (`specification/common/README.md`) # Common specification concepts **Status**: [Stable](../document-status.md)
Table of Contents - [AnyValue](#anyvalue) * [map](#mapstring-anyvalue) * [AnyValue representation for non-OTLP protocols](#anyvalue-representation-for-non-otlp-protocols) + [Strings](#strings) + [Booleans](#booleans) + [Integers](#integers) + [Floating Point Numbers](#floating-point-numbers) + [Byte Arrays](#byte-arrays) + [Empty Values](#empty-values) + [Arrays](#arrays) + [Maps](#maps) - [Attribute](#attribute) * [Attribute representation for non-OTLP](#attribute-representation-for-non-otlp) - [Attribute Collections](#attribute-collections) * [Attribute Collection representation for non-OTLP](#attribute-collection-representation-for-non-otlp) - [Attribute Limits](#attribute-limits) * [Configurable Parameters](#configurable-parameters) * [Exempt Entities](#exempt-entities)
## AnyValue `AnyValue` is either: - a primitive type: string, boolean, double precision floating point (IEEE 754-1985), or signed 64 bit integer, - a homogeneous array of primitive type values. A homogeneous array MUST NOT contain values of different types. - a byte array. - an array of `AnyValue`, - a [`map`](#mapstring-anyvalue), - an empty value if supported by the language, (e.g. `null`, `undefined` in JavaScript/TypeScript, `None` in Python, `nil` in Go/Ruby, not supported in Erlang, etc.) Arbitrary deep nesting of values for arrays and maps is allowed (essentially allows to represent an equivalent of a JSON object). APIs SHOULD be documented in a way to communicate to users that using array and map values may carry higher performance overhead compared to primitive values. AnyValues expressing an empty value, a numerical value of zero, an empty string, or an empty array are considered meaningful and MUST be stored and passed on to processors / exporters. While `null` is a valid attribute value, its use within homogeneous arrays SHOULD generally be avoided unless language constraints make this impossible. However, if it is impossible to make sure that no `null` values are accepted (e.g. in languages that do not have appropriate compile-time type checking), `null` values within arrays MUST be preserved as-is (i.e., passed on to processors / exporters as `null`). If exporters do not support exporting `null` values, they MAY replace those values by 0, `false`, or empty strings. This is required for map/dictionary structures represented as two arrays with indexes that are kept in sync (e.g., two attributes `header_keys` and `header_values`, both containing an array of strings to represent a mapping `header_keys[i] -> header_values[i]`). ### map `map` is a map of string keys to `AnyValue` values. The keys in the map are unique (duplicate keys are not allowed). Case sensitivity of keys MUST be preserved. Keys that differ in casing are treated as distinct keys. The representation of the map is language-dependent. The implementation MUST by default enforce that the exported maps contain only unique keys. The enforcement of uniqueness may be performed in a variety of ways as it best fits the limitations of the particular implementation (e.g. by removing duplicates). The implementation MAY have an option to allow exporting maps with duplicate keys (e.g. for better performance). If such option is provided, it MUST be documented that for many receivers, handling of maps with duplicate keys is unpredictable and it is the users' responsibility to ensure keys are not duplicate. Maps are equal when they contain the same key-value pairs, irrespective of the order in which those elements appear (unordered collection equality). ### AnyValue representation for non-OTLP protocols For protocols that do not natively support some of the AnyValue types, those values SHOULD be represented as strings following the encoding rules below. > [!NOTE] > This string representation is lossy. Type information is lost as all > values are converted to strings, and precision loss may occur for numeric values > (particularly for floating point numbers and large integers that exceed the > precision capabilities of the receiving system's string-to-number conversion). #### Strings Strings SHOULD be represented as-is without any additional encoding. They SHOULD NOT be encoded as JSON strings (with explicit surrounding quotes). Examples: `hello world`, (the empty string) #### Booleans Booleans SHOULD be represented as [JSON booleans](https://datatracker.ietf.org/doc/html/rfc8259#section-3). Examples: `true`, `false` #### Integers Integers SHOULD be represented as [JSON numbers](https://datatracker.ietf.org/doc/html/rfc8259#section-6). Examples: `42`, `-123` #### Floating Point Numbers Floating point numbers SHOULD be represented as [JSON numbers](https://datatracker.ietf.org/doc/html/rfc8259#section-6). The special floating point values NaN and Infinity SHOULD be represented as `NaN`, `Infinity`, and `-Infinity`. They SHOULD NOT be encoded as JSON strings (with explicit surrounding quotes). Examples: `3.14159`, `1.23e10`, `NaN`, `Infinity`, `-Infinity` #### Byte Arrays Byte arrays SHOULD be [Base64-encoded](https://datatracker.ietf.org/doc/html/rfc4648#section-4). They SHOULD NOT be encoded as JSON strings (with explicit surrounding quotes). Example: `aGVsbG8gd29ybGQ=` #### Empty Values Empty values SHOULD be represented as the empty string. They SHOULD NOT be encoded as a JSON string (with explicit surrounding quotes). #### Arrays Arrays, except for [byte arrays](#byte-arrays), SHOULD be represented as [JSON arrays](https://datatracker.ietf.org/doc/html/rfc8259#section-5). Array elements SHOULD be represented as JSON values using the following rules: - [Strings](#strings) as [JSON strings](https://datatracker.ietf.org/doc/html/rfc8259#section-7), - [Booleans](#booleans) as [JSON booleans](https://datatracker.ietf.org/doc/html/rfc8259#section-3) (`true` or `false`), - [Integers](#integers) and [floating point numbers](#floating-point-numbers) as [JSON numbers](https://datatracker.ietf.org/doc/html/rfc8259#section-6), except that the special floating point values NaN and Infinity SHOULD be represented as JSON strings `"NaN"`, `"Infinity"`, and `"-Infinity"`, - [Byte arrays](#byte-arrays) as [Base64-encoded](https://datatracker.ietf.org/doc/html/rfc4648#section-4) JSON strings, - [Empty values](#empty-values) as [JSON `null`](https://datatracker.ietf.org/doc/html/rfc8259#section-3), - [Arrays](#arrays) as [JSON arrays](https://datatracker.ietf.org/doc/html/rfc8259#section-5), - [Maps](#maps) as [JSON objects](https://datatracker.ietf.org/doc/html/rfc8259#section-4). Examples: `[]`, `[1, "-Infinity", "a", true, {"nested": "aGVsbG8gd29ybGQ="}]` #### Maps Maps SHOULD be represented as [JSON objects](https://datatracker.ietf.org/doc/html/rfc8259#section-4). Map keys SHOULD be represented as JSON string member names. Map values SHOULD be represented using the same rules as [array elements](#arrays). Examples: `{}`, `{"a": "-Infinity", "b": 2, "c": [3, null]}` ## Attribute An `Attribute` is a key-value pair, which MUST have the following properties: - The attribute key MUST be a non-`null` and non-empty string. - Case sensitivity of keys is preserved. Keys that differ in casing are treated as distinct keys. - The attribute value MUST be one of types defined in [AnyValue](#anyvalue). Attributes are equal when their keys and values are equal. See [Attribute Naming](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/naming.md#attributes) for naming guidelines. See [Requirement Level](https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/attribute-requirement-level.md) for requirement levels guidelines. See [this document](attribute-type-mapping.md) to find out how to map values obtained outside OpenTelemetry into OpenTelemetry attribute values. ### Attribute representation for non-OTLP For non-OTLP protocols that need to represent a single `Attribute` as a string, the RECOMMENDED form is a [JSON object](https://datatracker.ietf.org/doc/html/rfc8259#section-4) containing a single name/value pair (member). The attribute key SHOULD be represented as a JSON object member name. The attribute value SHOULD be represented as a JSON object member value and follow the encoding rules defined in [AnyValue representation for non-OTLP protocols](#anyvalue-representation-for-non-otlp-protocols), as it would be represented as an element in an [array](#arrays) and a value in a [map](#maps). Examples: `{"http.request.method": "GET"}`, `{"retries": 3}`, `{"payload": "aGVsbG8gd29ybGQ="}`, `{"session.id": null}`, `{"colors": ["red", "blue"]}`, `{"context": {"nested": true}}` > [!NOTE] > This string representation is lossy. Type information is lost as all > values are converted to strings, and precision loss may occur for numeric values > (particularly for floating point numbers and large integers that exceed the > precision capabilities of the receiving system's string-to-number conversion). ## Attribute Collections [Resources](../resource/sdk.md), [Instrumentation Scopes](instrumentation-scope.md), [Metric points](../metrics/data-model.md#metric-points), [Spans](../trace/api.md#set-attributes), Span [Events](../trace/api.md#add-events), Span [Links](../trace/api.md#link) and [Log Records](../logs/data-model.md), contain a collection of attributes. Attribute Collections are top-level collections of key-value pairs used in OpenTelemetry data models. Note that they are distinct from [`map`](#mapstring-anyvalue), which is a type of [`AnyValue`](#anyvalue) used to represent nested data structures e.g. in attribute values and [log record body](../logs/data-model.md#field-body). Implementation MUST by default enforce that the exported attribute collections contain only unique keys. The enforcement of uniqueness may be performed in a variety of ways as it best fits the limitations of the particular implementation (e.g. by removing duplicates). Normally for the telemetry generated using OpenTelemetry SDKs the attribute key-value pairs are set via an API that either accepts a single key-value pair or a collection of key-value pairs. Setting an attribute with the same key as an existing attribute SHOULD overwrite the existing attribute's value. See for example Span's [SetAttribute](../trace/api.md#set-attributes) API. A typical implementation of [SetAttribute](../trace/api.md#set-attributes) API will enforce the uniqueness by overwriting any existing attribute values pending to be exported, so that when the Span is eventually exported the exporters see only unique attributes. The OTLP format in particular requires that exported Resources, Spans, Metric data points and Log Records contain only unique attributes. Some other implementations may use a streaming approach where every [SetAttribute](../trace/api.md#set-attributes) API call immediately results in that individual attribute value being exported using a streaming wire protocol. In such cases the enforcement of uniqueness will likely be the responsibility of the recipient of this data. Implementations MAY have an option to allow exporting attribute collections with duplicate keys (e.g. for better performance). If such option is provided, it MUST be documented that for many receivers, handling of maps with duplicate keys is unpredictable and it is the users' responsibility to ensure keys are not duplicate. Collection of attributes are equal when they contain the same attributes, irrespective of the order in which those elements appear (unordered collection equality). ### Attribute Collection representation for non-OTLP For non-OTLP protocols that need to represent an Attribute Collection as a string, the RECOMMENDED form is a [JSON object](https://datatracker.ietf.org/doc/html/rfc8259#section-4). Each attribute key SHOULD be represented as a JSON object member name. Each attribute value SHOULD be represented as the corresponding JSON object member value and follow the encoding rules defined in [AnyValue representation for non-OTLP protocols](#anyvalue-representation-for-non-otlp-protocols), as it would be represented as an element in an [array](#arrays) and a value in a [map](#maps). This representation follows the same JSON object form as [maps](#maps), but applies to top-level Attribute Collections rather than nested [`map`](#mapstring-anyvalue) values. Examples: `{}`, `{"http.request.method": "GET", "retries": 3}`, `{"payload": "aGVsbG8gd29ybGQ=", "session.id": null}`, `{"colors": ["red", "blue"], "context": {"nested": true}}` ## Attribute Limits Execution of erroneous code can result in unintended attributes. If there are no limits placed on attribute collections, they can quickly exhaust available memory, resulting in crashes that are difficult to recover from safely. By default an SDK SHOULD apply truncation as per the list of [configurable parameters](#configurable-parameters) below. If an SDK provides a way to: - set an attribute value length limit such that for each attribute value: - if it is a string, if it exceeds that limit (counting any character in it as 1), SDKs MUST truncate that value, so that its length is at most equal to the limit, - if it is a byte array, if its length exceeds that limit (counting each byte as 1), SDKs MUST truncate that value, so that its length is at most equal to the limit, - if it is an array of strings, then apply the limit to each value within the array separately, - if it is an array of [AnyValue](#anyvalue), then apply the limit to each element of the array separately (and recursively), - if it is a [map](#mapstring-anyvalue), then apply the limit to each value within the map separately (and recursively), - otherwise a value MUST NOT be truncated; - set an attribute count limit such that: - if adding an attribute to an attribute collection would result in exceeding the limit (counting each attribute in the collection as 1), the SDK MUST discard that attribute, so that the total number of attributes in an attribute collection is at most equal to the limit; - the count limit applies only to top-level attributes, not to nested key-value pairs within [maps](#mapstring-anyvalue); - otherwise an attribute MUST NOT be discarded. - set an attribute value depth limit such that for each attribute value: - the SDK MUST start counting depth at 1 for the top-level attribute value, and increment depth when descending into arrays (both homogeneous and heterogeneous) or [maps](#mapstring-anyvalue); - arrays or map at a depth greater than the limit MUST be replaced with an empty value; - otherwise a value MUST NOT be changed due to the depth limit. There MAY be a log emitted to indicate to the user that an attribute was truncated, discarded, or replaced due to a limit. To prevent excessive logging, the log MUST NOT be emitted more than once per record on which an attribute is set. If the SDK implements the limits above, it MUST provide a way to change these limits programmatically. Names of the configuration options SHOULD be the same as in the list below. An SDK MAY implement model-specific limits, for example `SpanAttributeCountLimit` or `LogRecordAttributeCountLimit`. If both a general and a model-specific limit are implemented, then the SDK MUST first attempt to use the model-specific limit, if it isn't set, then the SDK MUST attempt to use the general limit. If neither are defined, then the SDK MUST try to use the model-specific limit default value, followed by the global limit default value. Note that the limits apply only to attribute collections. Attribute value length and depth limits apply recursively to attribute values, but they do not apply to values within other data structures such as [`LogRecord.Body`](../logs/data-model.md#field-body). ### Configurable Parameters * `AttributeCountLimit` (Default=128) - Maximum allowed attribute count per record; * `AttributeValueLengthLimit` (Default=Infinity) - Maximum allowed attribute value length (applies to string values and byte arrays); * `AttributeValueDepthLimit` (Default=64) - Maximum allowed attribute value depth (applies to arrays and maps); ### Exempt Entities Resource attributes SHOULD be exempt from the limits described above as resources are not susceptible to the scenarios (auto-instrumentation) that result in excessive attributes count or size. Resources are also sent only once per batch instead of per span so it is relatively cheaper to have more/larger attributes on them. Resources are also immutable by design and they are generally passed down to TracerProvider along with limits. This makes it awkward to implement attribute limits for Resources. Attributes, which belong to Metrics, are exempt from the limits described above at this time, as discussed in [Metrics Attribute Limits](../metrics/sdk.md#attribute-limits). --- ### Overview (`specification/configuration/README.md`) # Configuration OpenTelemetry SDK components are highly configurable. This specification outlines the mechanisms by which OpenTelemetry components can be configured. It does not attempt to specify the details of what can be configured. ## Configuration Interfaces ### Programmatic The SDK MUST provide a programmatic interface for all configuration. This interface SHOULD be written in the language of the SDK itself. All other configuration mechanisms SHOULD be built on top of this interface. An example of this programmatic interface is accepting a well-defined struct on an SDK builder class. From that, one could build a CLI that accepts a file (YAML, JSON, TOML, ...) and then transforms into that well-defined struct consumable by the programmatic interface. See [declarative configuration](#declarative-configuration). ### Environment variables Environment variable configuration defines a set of language agnostic environment variables for common configuration goals. See [OpenTelemetry Environment Variable Specification](./sdk-environment-variables.md). ### Declarative configuration Declarative configuration provides a mechanism for configuring OpenTelemetry which is more expressive and full-featured than the [environment variable](#environment-variables) based scheme, and language agnostic in a way not possible with [programmatic configuration](#programmatic). Notably, declarative configuration defines tooling allowing users to load OpenTelemetry components according to a file-based representation of a standardized configuration data model. Declarative configuration consists of the following main components: * [Data model](./data-model.md) defines data structures which allow users to specify an intended configuration of OpenTelemetry SDK components and instrumentation. The data model includes a file-based representation. * [Instrumentation configuration API](./api.md) allows instrumentation libraries to consume configuration by reading relevant configuration options during initialization. * [Configuration SDK](./sdk.md) defines SDK capabilities around file configuration, including an In-Memory configuration model, support for referencing custom plugin components in configuration files, and operations to parse configuration files and interpret the configuration data model. See also [supplementary guidelines](./supplementary-guidelines.md). ### Other Mechanisms Additional configuration mechanisms SHOULD be provided in whatever language/format/style is idiomatic for the language of the SDK. The SDK can include as many configuration mechanisms as appropriate. --- ### Overview (`specification/context/README.md`) # Context **Status**: [Stable](../document-status.md).
Table of Contents - [Overview](#overview) - [Create a key](#create-a-key) - [Get value](#get-value) - [Set value](#set-value) - [Optional Global operations](#optional-global-operations) * [Get current Context](#get-current-context) * [Attach Context](#attach-context) * [Detach Context](#detach-context)
## Overview A `Context` is a propagation mechanism which carries execution-scoped values across API boundaries and between logically associated [execution units](../glossary.md#execution-unit). Cross-cutting concerns access their data in-process using the same shared `Context` object. A `Context` MUST be immutable, and its write operations MUST result in the creation of a new `Context` containing the original values and the specified values updated. Languages are expected to use the single, widely used `Context` implementation if one exists for them. In the cases where an extremely clear, pre-existing option is not available, OpenTelemetry MUST provide its own `Context` implementation. Depending on the language, its usage may be either explicit or implicit. Users writing instrumentation in languages that use `Context` implicitly are discouraged from using the `Context` API directly. In those cases, users will manipulate `Context` through cross-cutting concerns APIs instead, in order to perform operations such as setting trace or baggage entries for a specified `Context`. A `Context` is expected to have the following operations, with their respective language differences: ## Create a key Keys are used to allow cross-cutting concerns to control access to their local state. They are unique such that other libraries which may use the same context cannot accidentally use the same key. It is recommended that concerns mediate data access via an API, rather than provide direct public access to their keys. The API MUST accept the following parameter: - The key name. The key name exists for debugging purposes and does not uniquely identify the key. Multiple calls to `CreateKey` with the same name SHOULD NOT return the same value unless language constraints dictate otherwise. Different languages may impose different restrictions on the expected types, so this parameter remains an implementation detail. The API MUST return an opaque object representing the newly created key. ## Get value Concerns can access their local state in the current execution state represented by a `Context`. The API MUST accept the following parameters: - The `Context`. - The key. The API MUST return the value in the `Context` for the specified key. ## Set value Concerns can record their local state in the current execution state represented by a `Context`. The API MUST accept the following parameters: - The `Context`. - The key. - The value to be set. The API MUST return a new `Context` containing the new value. ## Optional Global operations These operations are expected to only be implemented by languages using `Context` implicitly, and thus are optional. These operations SHOULD only be used to implement automatic scope switching and define higher level APIs by SDK components and OpenTelemetry instrumentation libraries. ### Get current Context The API MUST return the `Context` associated with the caller's current execution unit. ### Attach Context Associates a `Context` with the caller's current execution unit. The API MUST accept the following parameters: - The `Context`. The API MUST return a value that can be used as a `Token` to restore the previous `Context`. Note that every call to this operation should result in a corresponding call to [Detach Context](#detach-context). ### Detach Context Resets the `Context` associated with the caller's current execution unit to the value it had before attaching a specified `Context`. This operation is intended to help making sure the correct `Context` is associated with the caller's current execution unit. Users can rely on it to identify a wrong call order, i.e. trying to detach a `Context` that is not the current instance. In this case the operation can emit a signal to warn users of the wrong call order, such as logging an error or returning an error value. The API MUST accept the following parameters: - A `Token` that was returned by a previous call to attach a `Context`. The API MAY return a value used to check whether the operation was successful or not. --- ### Overview (`specification/entities/README.md`) # Entities
Table of Contents - [Overview](#overview) - [Specifications](#specifications)
## Overview Entity represents an object of interest associated with produced telemetry: traces, metrics, logs, profiles etc. Entities can be instantiated through two complementary approaches: 1. **Pull-based Model**: Entities discover themselves from the current environment by examining the runtime context, system properties, metadata services, and other available sources of entity information. 2. **Push-based Model**: Entities are explicitly provided from external sources, typically through environment variables or configuration. This approach allows entity information to be propagated across process boundaries. See [Entity Propagation](./entity-propagation.md) for more details. ## Specifications - [Data Model](./data-model.md) - [Entity Propagation](./entity-propagation.md) - Describes the push-based model for entity instantiation - [Entity Events](./entity-events.md) - Describes how to communicate entity information as structured log events --- ### Overview (`specification/logs/README.md`) # OpenTelemetry Logging
Table of Contents - [Introduction](#introduction) - [Limitations of non-OpenTelemetry Solutions](#limitations-of-non-opentelemetry-solutions) - [OpenTelemetry Solution](#opentelemetry-solution) - [Log Correlation](#log-correlation) - [Legacy and Modern Log Sources](#legacy-and-modern-log-sources) * [System Logs](#system-logs) * [Infrastructure Logs](#infrastructure-logs) * [Third-party Application Logs](#third-party-application-logs) * [Legacy First-Party Applications Logs](#legacy-first-party-applications-logs) + [Via File or Stdout Logs](#via-file-or-stdout-logs) + [Direct to Collector](#direct-to-collector) * [New First-Party Application Logs](#new-first-party-application-logs) - [OpenTelemetry Collector](#opentelemetry-collector) - [Auto-Instrumenting Existing Logging](#auto-instrumenting-existing-logging) - [Specifications](#specifications) - [References](#references)
## Introduction Of all telemetry signals logs have probably the biggest legacy. Most programming languages have built-in logging capabilities or well-known, widely used logging libraries. For metrics and traces OpenTelemetry takes the approach of a clean-sheet design, specifies a new API and provides full implementations of this API in multiple languages. Our approach with logs is somewhat different. For OpenTelemetry to be successful in logging space we need to support existing legacy of logs and logging libraries, while offering improvements and better integration with the rest of observability world where possible. This is in essence the philosophy behind OpenTelemetry's logs support. We embrace existing logging solutions and make sure OpenTelemetry works nicely with existing logging libraries, log collection and processing solutions. ## Limitations of non-OpenTelemetry Solutions Unfortunately existing logging solutions are currently weakly integrated with the rest of the observability signals. Logs typically have limited support in tracing and monitoring tools in the form of links that use available and often incomplete correlation information (such as the time and origin attributes). This correlation may be fragile because attributes are often added to logs, traces and metrics via different means (e.g. using different collection agents). There is no standardized way to include the information about the origin and source of logs (such as the application and the location/infrastructure where the application runs) that is uniform with traces and metrics and allows all telemetry data to be fully correlated in a precise and robust manner. Similarly, logs have no standardized way to propagate and record the request execution context. In distributed systems this often results in a disjoint set of logs collected from different components of the system. This is how a typical non-OpenTelemetry observability collection pipeline looks like today: There are often different libraries and different collection agents, using different protocols and data models, with telemetry data ending up in separate backends that don't know how to work well together. ## OpenTelemetry Solution Distributed tracing introduced the notion of trace context propagation. Fundamentally, though, nothing prevents the logs to adopt the same context propagation concepts. If the recorded logs contained trace context identifiers (such as trace and span IDs or user-defined baggage) it would result in much richer correlation between logs and traces, as well as correlation between logs emitted by different components of a distributed system. This would make logs significantly more valuable in distributed systems. This is one of the promising evolutionary directions for observability tools. Standardizing log correlation with traces and metrics, adding support for distributed context propagation for logs, unification of source attribution of logs, traces and metrics will increase the individual and combined value of observability information for legacy and modern systems. This is the vision of OpenTelemetry's collection of logs, traces and metrics: We emit logs, traces and metrics in a way that is compliant with OpenTelemetry data models, send the data through OpenTelemetry Collector, where it can be enriched and processed in a uniform manner. For example, Collector can add to all telemetry data coming from a Kubernetes Pod several attributes that describe the pod and it can be done automatically using [k8sattributesprocessor](https://pkg.go.dev/github.com/open-telemetry/opentelemetry-collector-contrib/processor/k8sattributesprocessor) without the need for the Application to do anything special. Most importantly such enrichment is completely uniform for all 3 signals. The Collector guarantees that logs, traces and metrics have precisely the same attribute names and values describing the Kubernetes Pod that they come from. This enables exact and unambiguous correlation of the signals by the Pod in the backend. For traces and metrics OpenTelemetry defines a new API that application developers must use to emit traces and metrics. For logs we did not take the same path. We realized that there is a much bigger and more diverse legacy in logging space. There are many existing logging libraries in different languages, each having their own API. Many programming languages have established standards for using particular logging libraries. For example in Java world there are several highly popular and widely used logging libraries, such as Log4j or Logback. There are also countless existing prebuilt applications or systems that emit logs in certain formats. Operators of such applications have no or limited control on how the logs are emitted. OpenTelemetry needs to support these logs. Given the above state of the logging space we took the following approach: - OpenTelemetry defines a [log data model](data-model.md). The purpose of the data model is to have a common understanding of what a LogRecord is, what data needs to be recorded, transferred, stored and interpreted by a logging system. - Newly designed logging systems are expected to emit logs according to OpenTelemetry's log data model. More on this [later](#new-first-party-application-logs). - Existing log formats can be [unambiguously mapped](data-model-appendix.md) to OpenTelemetry log data model. OpenTelemetry Collector can read such logs and translate them to OpenTelemetry log data model. - OpenTelemetry defines a [Logs API](./api.md) for [emitting LogRecords](./api.md#emit-a-logrecord). It was primarily designed for library authors to build [log appenders](../glossary.md#log-appender--bridge), which use the API to bridge between existing logging libraries and the OpenTelemetry log data model. The Logs API can also be used directly by applications, which is particularly important for: - **Instrumentation libraries** to avoid coupling to a particular logging library. - **Emitting structured events** following [semantic conventions](https://opentelemetry.io/docs/specs/semconv/), especially when the existing logging library cannot emit structured data or specify event names. - **Scenarios requiring direct control** over log emission without intermediary logging library. Note that existing logging libraries generally provide a much richer set of features than what is defined in OpenTelemetry. Yet, languages may provide an [ergonomic API](./api.md#ergonomic-api) for better developer experience when using it directly. - OpenTelemetry defines an [SDK](./sdk.md) implementation of the [API](./api.md), which enables configuration of [processing](./sdk.md#logrecordprocessor) and [exporting](./sdk.md#logrecordexporter) LogRecords. This approach allows OpenTelemetry to read existing system and application logs, provides a way for newly built application to emit rich, structured, OpenTelemetry-compliant logs, and ensures that all logs are eventually represented according to a uniform log data model on which the backends can operate. Later in this document we will discuss in more details [how various log sources are handled](#legacy-and-modern-log-sources) by OpenTelemetry, but first we need to describe in more details an important concept: the log correlation. ## Log Correlation Logs can be correlated with the rest of observability data in a few dimensions: - By the **time of execution**. Logs, traces and metrics can record the moment of time or the range of time the execution took place. This is the most basic form of correlation. - By the **execution context**, also known as the trace context. It is a standard practice to record the execution context (trace and span IDs as well as user-defined context) in the spans. OpenTelemetry extends this practice to logs where possible by including [TraceId](data-model.md#field-traceid) and [SpanId](data-model.md#field-spanid) in the LogRecords. This allows to directly correlate logs and traces that correspond to the same execution context. It also allows to correlate logs from different components of a distributed system that participated in the particular request execution. - By the **origin of the telemetry**, also known as the Resource context. OpenTelemetry traces and metrics contain information about the Resource they come from. We extend this practice to logs by including the [Resource](data-model.md#field-resource) in LogRecords. These 3 correlations can be the foundation of powerful navigational, filtering, querying and analytical capabilities. OpenTelemetry aims to record and collects logs in a manner that enables such correlations. ## Legacy and Modern Log Sources It is important to distinguish several sorts of legacy and modern log sources. Firstly, this directly affects how exactly we get access to these logs and how we collect them. Secondly, we have varying levels of control over how these logs are generated and whether we can amend the information that can be included in the logs. Below we list several categories of logs and describe what can be possibly done for each category to have better experience in the observability solutions. ### System Logs These are logs generated by the operating system and over which we have no control. We cannot change the format or affect what information is included. Examples of system format are Syslog and Windows Event Logs. System logs are written at the host level (which may be physical, virtual or containerized) and have a predefined format and content (note that applications may also be able to write records to standard system logs: this case is covered below in the [Third-Party Applications](#third-party-application-logs) section). System operations recorded in the logs can be a result of a request execution. However system logs either do not include any data about the trace context or if included it is highly idiosyncratic and thus difficult to identify, parse and use. This makes it nearly impossible to perform trace context correlation for system logs. However we can and should automatically enrich system logs with the resource context - the information about the host that is available during collection. This can include the hostname, IP address, container or pod name, etc. This information should be added to the Resource field of collected log data. OpenTelemetry Collector can read system logs (link TBD) and automatically enrich them with Resource information using the [resourcedetection](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor) processor. ### Infrastructure Logs These are logs generated by various infrastructure components, such as Kubernetes events. Like system logs, the infrastructure logs lack a trace context and can be enriched by the resource context - information about the node, pod, container, etc. OpenTelemetry Collector or other agents can be used to query logs from most common infrastructure controllers. ### Third-party Application Logs Applications typically write logs to standard output, to files or other specialized medium (e.g. Windows Event Logs for applications). These logs can be in many different formats, spanning a spectrum along these variations: - Free-form text formats with no easily automatable and reliable way to parse structured data from them. - Better specified and sometimes customizable formats that can be parsed to extract structured data (such as Apache logs or RFC5424 Syslog). - Formally structured formats (e.g. JSON files with well-defined schema or Windows Event Log). The collection system needs to be able to discover most commonly used applications and have parsers that can convert these logs into a structured format. Like system and infrastructure logs, application logs often lack request context but can be enriched by resource context, including the attributes that describe the host and infrastructure as well as application-level attributes (such as the application name, version, name of the database - if it is a DBMS, etc). OpenTelemetry recommends to collect application logs using Collector's [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). Alternatively, another log collection agent, such as FluentBit, can collect logs, [then send](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/fluentforwardreceiver) to OpenTelemetry Collector where the logs can be further processed and enriched. ### Legacy First-Party Applications Logs These are applications that are created in-house. People tasked with setting up log collection infrastructure sometimes are able to modify these applications to alter how logs are written and what information is included in the logs. For example, the application’s log formatters may be reconfigured to output JSON instead of plain text and by doing so help improve the reliability of log collection. More significant modifications to these applications can be done manually by their developers, such as addition of the trace context to every log statement, however this is likely going to be vanishingly rare due to the effort required. As opposed to manual efforts we have an interesting opportunity to "upgrade" application logs in a less laborious way by providing full or semi auto-instrumenting solutions that modify trace logging libraries used by the application to automatically output the trace context such as the trace ID or span ID with every log statement. The trace context can be automatically extracted from incoming requests if standard compliant request propagation is used, e.g. via [W3C TraceContext](https://www.w3.org/TR/trace-context/). In addition, the requests outgoing from the application may be injected with the same trace context data, thus resulting in context propagation through the application and creating an opportunity to have full trace context in logs collected from all applications that can be instrumented in this manner. Some logging libraries are designed to be extended in this manner relatively easily. There is no need to actually modify the libraries, instead we can implement "log appender" or "log bridge" components for such libraries and implement the additional LogRecord enrichment in these components. There are typically 2 ways to collect logs from these applications. #### Via File or Stdout Logs The first approach, assuming the logs are written to files or to standard output, requires ability to read file logs, tail them, work correctly when log rotation is used, optionally also parse the logs to convert them into more structured formats. Parsing requires support for different parser types, which can also be configured to parse custom formats as well as ability to add custom parsers. Examples of common formats that parsers need to support are: CSV, Common Log Format, Labeled Tab-separated Values (LTSV), key-value Pair format, JSON, etc. To support this approach OpenTelemetry recommends to collect logs using OpenTelemetry [Collector](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver). Alternatively, if the Collector does not have the necessary file reading and parsing capabilities, another log collection agent, such as FluentBit can collect the logs, [then send the logs](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/fluentforwardreceiver) to OpenTelemetry Collector. The benefit of using an intermediary medium is that how logs are produced and where they are written by the application requires no or minimal changes. The downside is that it requires the often non-trivial log file reading and parsing functionality. Parsing may also be not reliable if the output format is not well-defined. For details on recording and parsing trace context, see [Trace Context in Non-OTLP Log Formats](../compatibility/logging_trace_context.md). #### Direct to Collector The second approach is to modify the application so that the logs are output via a network protocol, e.g. via [OTLP](https://github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto). The most convenient way to achieve this is to provide addons or extensions to the commonly used logging libraries. The addons implement sending over such network protocols, which would then typically require small, localized changes to the application code to change the logging target. The application logs will be also enriched by the resource context, similarly to how it is done for third-party applications and so will potentially have full correlation information across all context dimensions. The downside of this approach is that the simplicity of having the logs in a local file is lost (e.g. ability to easily inspect the log file locally) and requires a full buy-in in OpenTelemetry's logging approach. This approach also only works if the destination that the logs need to be delivered is able to receive logs via the network protocol that OpenTelemetry can send in. The benefits of this approach is that it emits the logs in well-defined, formal, highly structured format, removes all complexity associated with file logs, such as parsers, log tailing and rotation. It also enables the possibility to send logs directly to the logging backend without using a log collection agent. To facilitate both approaches described above OpenTelemetry provides a [API](./api.md) and [SDK](./sdk.md), which can be used together with existing logging libraries to automatically inject the trace context in the emitted logs, and provide an easy way to send the logs via OTLP. Instead of modifying each logging statement, [log appenders](../glossary.md#log-appender--bridge) use the API to bridge logs from existing logging libraries to the OpenTelemetry data model, where the SDK controls how the logs are processed and exported. Application developers only need to configure the Appender and SDK at application startup. ### New First-Party Application Logs These are greenfield developments. OpenTelemetry provides recommendations and best practices about how to emit logs (along with traces and metrics) from these applications. Applications have several options for emitting logs: 1. **Using existing logging libraries with OpenTelemetry log appenders**: For applicable languages and frameworks, auto-instrumentation or simple configuration of a logging library to use an OpenTelemetry log appender will be the easiest way to emit context-enriched logs. As already described earlier, we provide extensions to some popular logging libraries to support manual instrumentation cases. The extensions support the inclusion of the trace context in the logs and allow sending logs via OTLP protocol to the backend or to the Collector, bypassing the need to have the logs represented as text files. 2. **Using the OpenTelemetry Logs API directly**: Applications can use the [Logs API](./api.md) directly to emit structured logs and events. This approach works well for emitting events following [semantic conventions](https://opentelemetry.io/docs/specs/semconv/) and enables easier reuse of attributes used across different signals. Note that a language can provide a more convenient [ergonomic API](./api.md#ergonomic-api). Regardless of the approach, emitted logs are automatically augmented by application-specific resource context (e.g. process ID, programming language, logging library name and version, etc). Full correlation across all context dimensions will be available for these logs. This is how a typical new application uses OpenTelemetry API, SDK and the existing log libraries: ## OpenTelemetry Collector To enable log collection according to this specification we use OpenTelemetry Collector. The following functionality exists to enable log collection: - Support for log data type and log pipelines based on the [log data model](data-model.md). This includes processors such as [attributesprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/attributesprocessor) that can operate on log data. - Ability to read logs from text files, tail the files, understand common log rotation schemes, watch directories for log file creation, ability to checkpoint file positions and resume reading from checkpoints. This ability is implemented by using Collector's [filelog receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver) or using an externally running agent (such as FluentBit). - Ability to parse logs in common text formats and to allow end users to customize parsing formats and add custom parsers as needed. Collector's [parsers](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/receiver/filelogreceiver/README.md#operators) or parsing in the external agent is used for this. - Ability to receive logs via common network protocols for logs, such as Syslog and interpret them according to semantic conventions defined in this specification. FluentBit or similar agent is used for this. Over time some of this functionality may be migrated directly to the Collector. - Ability to send logs via common network protocols for logs, such as Syslog, or vendor-specific log formats. Collector contains exporters that directly implement this ability. ## Auto-Instrumenting Existing Logging We can provide auto-instrumentation for most popular logging libraries. The auto-instrumented logging statements will do the following: - Read incoming trace context (this is part of broader instrumentation that auto-instrumenting libraries perform). - Configure logging libraries to use trace ID and span ID fields from request context as logging context and automatically include them in all logged statements. This is possible to do for certain languages (e.g. in Java) and we can reuse [existing open source libraries](https://docs.datadoghq.com/tracing/other_telemetry/connect_logs_and_traces/java/) that do this. A further optional modification would be to auto-instrument loggers to send logs directly to the backend via OTLP instead or in addition to writing to a file or standard output. ## Specifications * [Logs API](./api.md) * [Logs SDK](./sdk.md) * [Logs Data Model](./data-model.md) * [Trace Context in non-OTLP Log Formats](../compatibility/logging_trace_context.md) ## References - [OTEP0091 Logs: Vocabulary](../../oteps/logs/0091-logs-vocabulary.md) - [OTEP0092 OpenTelemetry Logs Vision](../../oteps/logs/0092-logs-vision.md) --- ### Overview (`specification/metrics/README.md`) # OpenTelemetry Metrics
Table of Contents - [Overview](#overview) * [Design Goals](#design-goals) * [Concepts](#concepts) + [API](#api) + [SDK](#sdk) + [Programming Model](#programming-model) - [Specifications](#specifications) - [References](#references)
## Overview ### Design Goals Given there are many well-established metrics solutions that exist today, it is important to understand the goals of OpenTelemetry’s metrics effort: * **Being able to connect metrics to other signals**. For example, metrics and traces can be correlated via [exemplars](data-model.md#exemplars), and metrics attributes can be enriched via [Baggage](../baggage/api.md) and [Context](../context/README.md). Additionally, [Resource](../resource/sdk.md) can be applied to [logs](../overview.md#log-signal)/[metrics](../overview.md#metric-signal)/[traces](../overview.md#tracing-signal) in a consistent way. * **Providing a path for [OpenCensus](https://opencensus.io/) customers to migrate to OpenTelemetry**. This was the original goal of OpenTelemetry - converging OpenCensus and OpenTracing. We will focus on providing the semantics and capability, instead of doing a 1-1 mapping of the APIs. * **Working with existing metrics instrumentation protocols and standards**. Here is the minimum set of goals: * Providing full support for [Prometheus](https://prometheus.io/) - users should be able to use OpenTelemetry clients and [Collector](../overview.md#collector) to collect and export metrics, with the ability to achieve the same functionality as the native Prometheus clients. * Providing the ability to collect [StatsD](https://github.com/statsd/statsd) metrics using the [OpenTelemetry Collector](../overview.md#collector). ### Concepts #### API The **OpenTelemetry Metrics API** ("the API" hereafter) serves two purposes: * Capturing raw measurements efficiently and simultaneously. * Decoupling the instrumentation from the [SDK](#sdk), allowing the SDK to be specified/included in the application. When no [SDK](#sdk) is explicitly included/enabled in the application, no telemetry data will be collected. Please refer to the overall [OpenTelemetry API](../overview.md#api) concept and [API and Minimal Implementation](../library-guidelines.md#api-and-minimal-implementation) for more information. #### SDK The **OpenTelemetry Metrics SDK** ("the SDK" hereafter) implements the API, providing functionality and extensibility such as configuration, aggregation, processors and exporters. OpenTelemetry requires a [separation of the API from the SDK](../library-guidelines.md#requirements), so that different SDKs can be configured at runtime. Please refer to the overall [OpenTelemetry SDK](../overview.md#sdk) concept for more information. #### Programming Model ```text +------------------+ | MeterProvider | +-----------------+ +--------------+ | Meter A | Measurements... | | Metrics... | | | Instrument X +-----------------> In-memory state +-------------> MetricReader | | Instrument Y | | | | | | Meter B | +-----------------+ +--------------+ | Instrument Z | | ... | +-----------------+ +--------------+ | ... | Measurements... | | Metrics... | | | ... +-----------------> In-memory state +-------------> MetricReader | | ... | | | | | | ... | +-----------------+ +--------------+ +------------------+ ``` ## Specifications * [Metrics API](./api.md) * [Metrics SDK](./sdk.md) * [Metrics Data Model and Protocol](./data-model.md) * [Metrics Requirement Levels](./metric-requirement-level.md) ## References * Scenarios for Metrics API/SDK Prototyping ([OTEP 146](../../oteps/metrics/0146-metrics-prototype-scenarios.md)) --- ### Overview (`specification/profiles/README.md`) # OpenTelemetry Profiles **Status**: [Alpha](../document-status.md)
Table of Contents - [Overview](#overview) - [Design goals](#design-goals) - [Data Format](#data-format) - [Known values](#known-values) - [Specifications](#specifications) - [References](#references)
## Overview A **profile** is a collection of stack traces with associated values representing resource consumption and code execution, collected from a running program. Profiling is the process of collecting such profiles, typically by sampling program state at regular intervals. For a general introduction to OpenTelemetry profiling and how it complements other observability signals, see [Profiles concepts](https://opentelemetry.io/docs/concepts/signals/profiles/). ## Design goals The profiles signal is designed with the following goals in mind: - **Low overhead**: Enable profiling agents to operate continuously in production environments without materially impacting application performance. - **Efficient representation**: Reduces volume of data stored and transmitted, by using dictionary tables to deduplicate repeated information across samples. - **Compatibility with existing formats**: The data format is a superset of established profiling formats such as [pprof](https://github.com/google/pprof) and in most cases supports lossless conversions to and from these formats. If that's not possible (e.g. custom extensions), the `original_payload_format` field can be used to transmit the original information for future lossless export or reinterpretation. - **Correlation with other signals**: Profiles MUST be linkable to logs, metrics and traces through shared resource context and, where applicable, direct trace/span references. ## Data Format The OpenTelemetry profiles data format is [here](./data-format.md). It builds on the [pprof protobuf format](https://github.com/google/pprof/tree/main/proto) and extends it with: - **Resource and scope context**: Each batch of profiles is associated with a [Resource](../resource/README.md) and an [InstrumentationScope](../common/instrumentation-scope.md), consistent with logs, metrics and traces. - **Generalized dictionary**: Deduplicates not only strings but also other messages that exhibit redundancy. - **Generalized attributes**: Most messages can carry attributes following the same conventions as other signals, augmented with Unit information (`KeyValueAndUnit`). - **Span context references**: Samples MAY include a `Link` (span ID and trace ID), enabling direct linking between a profile sample and the trace/span during which it was captured. For details on the required attributes for `Mapping` messages and the custom hashing scheme for build ID generation, see [Mappings](./mappings.md). For more information on compatibility with [pprof](https://github.com/google/pprof), see [pprof](./pprof.md). ## Known values [OpenTelemetry semantic conventions](https://opentelemetry.io/docs/specs/semconv/) are vital for profiles to correlate with other OpenTelemetry signals, enabling unified analysis of traces, metrics, logs, and profiles for a holistic system understanding. To enhance the compatibility of OpenTelemetry Profiles with existing profiling tools, known values are utilized. | Profile field | Known values | | ------------- | ------------ | | original_payload_format | [pprof](https://github.com/google/pprof/tree/main/proto), [jfr](https://en.wikipedia.org/wiki/JDK_Flight_Recorder) or [linux_perf](https://perfwiki.github.io/) | ## Specifications * [Profiles Data Format](./data-format.md) * [Profiles Mappings Attributes](./mappings.md) * [Profiles Pprof Compatibility](./pprof.md) ## References - [Profiles Concepts](https://opentelemetry.io/docs/concepts/signals/profiles/) - [Profiles Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/general/profiles/) - [OTEP0212 OpenTelemetry Profiles Vision](../../oteps/profiles/0212-profiling-vision.md) --- ### Overview (`specification/protocol/README.md`) # OpenTelemetry Protocol The OpenTelemetry protocol (OTLP) design goals, requirements, and [specification] have moved to [github.com/open-telemetry/opentelemetry-proto/docs](https://github.com/open-telemetry/opentelemetry-proto/tree/main/docs). You can also view the specification from the OpenTelemetry website, see [OTLP][specification]. For additional OTLP implementation requirements in the OpenTelemetry SDKs, see [SDK Exporter](exporter.md). [specification]: https://opentelemetry.io/docs/specs/otlp/ --- ### Overview (`specification/resource/README.md`) # Resource
Table of Contents - [Overview](#overview) * [Identity](#identity) * [Navigation](#navigation) * [Telescoping](#telescoping) - [Specifications](#specifications)
## Overview A Resource represents the observed entity for which telemetry is produced. Within OpenTelemetry, all signals are associated with a Resource, enabling contextual correlation of data from the same source. For example, if I see a high latency in a span, I need to check the metrics for the same entity during the time when the latency was observed. > [!NOTE] > The entity that *technically emits* telemetry and the entity *described by* > a Resource are not always the same. In auto-instrumentation scenarios, such > as eBPF-based agents that observe a running process without code changes, > the instrumentation agent produces the telemetry on behalf of the observed > workload. In these cases, the Resource describes the *observed* entity > (e.g., the service or process being instrumented), not the agent itself. > Attributes such as `telemetry.sdk.name` and `telemetry.distro.name` that > pertain to the instrumentation tooling should be applied according to their > own semantic conventions rather than used to identify the observed entity. Resource provides two important aspects for observability: - It MUST identify the observed entity for which telemetry is being produced. - It SHOULD allow users to determine where that entity resides within their infrastructure. ### Identity Resource provides a natural way to understand "what" produced an effect and evaluate other signals of that same source. This is done through attaching the same set of identifying attributes on all telemetry produced in an OpenTelemetry SDK. Resource identity provides a natural pivot point for observability signals, a key type of correlation in OpenTelemetry. ### Navigation Implicit in the design of Resource and attributes is ensuring users are able to navigate their infrastructure, tools, UIs, etc. to find the *same* entity that telemetry is reporting against. For example, in practice we could see Resource including more than one entity, like: - A process - A container - A Kubernetes pod name - A namespace - A deployment By including identifying attributes of each of these, we can help users navigate through their `kubectl` or Kubernetes UIs to find the specific process generating telemetry. This is as important as being able to uniquely identify one process from another. > [!TIP] > Observability signals SHOULD be actionable. Knowing a process is > struggling is not as useful as being able to scale up a deployment to take > load off the struggling process. If the only thing important to Resource was identity, we could simply use UUIDs. However, this would rely on some other, easily accessible, system to provide human-friendly understanding for these UUIDs. OpenTelemetry provides a model where a full UUID-only solution could be chosen, but defaults to a *blended* approach, where resource provides both Identity and Navigation. This leads to the next concept: Telescoping identity to the needs of a system. ### Telescoping Within OpenTelemetry, we want to give users the flexibility to decide what information needs to be sent *with* observability signals and what information can be later joined. We call this "telescoping identity" where users can decide how *small* or *large* the size of an OpenTelemetry resource will be on the wire (and correspondingly, how large data points are when stored, depending on storage solution). For example, in the extreme, OpenTelemery could synthesize a UUID for every system which produces telemetry. All identifying attributes for Resource and Entity could be sent via a side channel with known relationships to this UUID. While this would optimize the runtime generation and sending of telemetry, it comes at the cost of downstream storage systems needing to join data back together either at ingestion time or query time. For high performance use cases, e.g. alerting, these joins can be expensive. In practice, users control Resource identity via the configuration of Resource Detection within SDKs and the collector. Users wishing for minimal identity will limit their resource detection just to a `service.instance.id`, for example. Some users highly customize resource detection with many concepts being appended. ## Specifications - [Data Model](./data-model.md) - [Resource SDK](./sdk.md) --- ### Overview (`specification/schemas/README.md`) # Telemetry Schemas **Status**: [Stable](../document-status.md)
Table of Contents - [Motivation](#motivation) - [How Schemas Work](#how-schemas-work) - [What is Out of Scope](#what-is-out-of-scope) - [Use Cases](#use-cases) * [Full Schema-Aware](#full-schema-aware) * [Collector-Assisted Schema Transformation](#collector-assisted-schema-transformation) - [Schema URL](#schema-url) - [Schema Version Number](#schema-version-number) - [OTLP Support](#otlp-support) - [API Support](#api-support) - [OpenTelemetry Schema](#opentelemetry-schema)
## Motivation Telemetry sources such as instrumented applications and consumers of telemetry such as observability backends sometimes make implicit assumptions about the emitted telemetry. They assume that the telemetry will contain certain attributes or otherwise have a certain shape and composition of data (this is referred to as "telemetry schema" throughout this document). This makes it difficult or impossible to change the composition of the emitted telemetry data without breaking the consumers. For example changing the name of an attribute of a span created by an instrumentation library can break the backend if the backend expects to find that attribute by its name. Semantic conventions are an important part of this problem. These conventions define what names and values to use for span attributes, metric names and other fields. If semantic conventions are changed the existing implementations (telemetry source or consumers) need to be also changed correspondingly. Furthermore, to make things worse, the implementations of telemetry sources and implementations of telemetry consumers that work together and that depend on the changed semantic convention need to be changed simultaneously, otherwise such implementations will no longer work correctly together. Essentially there is a coupling between 3 parties: 1) OpenTelemetry semantic conventions, 2) telemetry sources and 3) telemetry consumers. The coupling complicates the independent evolution of these 3 parties. We recognize the following needs: - OpenTelemetry semantic conventions need to evolve over time. When conventions are first defined, mistakes are possible and we may want to fix the mistakes over time. We may also want to change conventions to re-group the attributes into different namespaces as our understanding of the attribute taxonomy improves. - Telemetry sources over time may want to change the schema of the telemetry they emit. This may be because for example the semantic conventions evolved and we want to make our telemetry match the newly introduced conventions. - In an observability system there may simultaneously exist telemetry sources that produce data that conforms to different telemetry schemas because different sources evolve at a different pace and are implemented and controlled by different entities. - Telemetry consumers have a need to understand what schema a particular piece of received telemetry confirms to. The consumers also need a way to be able to interpret the telemetry data that uses different telemetry schemas. Telemetry Schemas that were proposed and accepted in [OTEP0152](../../oteps/0152-telemetry-schemas.md) address these needs. ## How Schemas Work We believe that the 3 parties described above are able to evolve independently over time, while continuously retaining the ability to correctly work together. Telemetry Schemas are central to how we make this possible. Here is a summary of how the schemas work: - OpenTelemetry defines a [file format](file_format_v1.1.0.md) for defining telemetry schemas. - Telemetry schemas are versioned. Over time the schema may evolve and telemetry sources may emit data conforming to newer versions of the schema. - Telemetry schemas explicitly define transformations that are necessary to convert telemetry data between different versions of the schema, provided that such conversions are possible. When conversions are not possible it constitutes a breaking change between versions. - Telemetry schemas are identified by Schema URLs, that are unique for each schema version. - Telemetry sources (e.g. instrumentation libraries) will include a schema URL in the emitted telemetry. - Telemetry consumers will pay attention to the schema of the received telemetry. If necessary, telemetry consumers may transform the telemetry data from the received schema version to the target schema version as expected at the point of use (e.g. a dashboard may define which schema version it expects). - OpenTelemetry publishes a telemetry schema as part of the specification. The schema contains the list of transformations that semantic conventions undergo. The schema is to be available, to be referred and downloaded at a well known URL: `https://opentelemetry.io/schemas/` (where `` matches the specification version number). - OpenTelemetry instrumentation libraries include the OpenTelemetry Schema URL in all emitted telemetry. This is currently work-in-progress, [here is an example](https://github.com/open-telemetry/opentelemetry-go/blob/8ba6da8f3e5380629fcd72057fe0f827a9e23493/sdk/resource/os.go#L50) of how it is done in Go SDK's Resource detectors. - OTLP allows inclusion of a schema URL in the emitted telemetry. - Third-party libraries, instrumentation or applications are advised to define and publish their own telemetry schema if it is completely different from OpenTelemetry schema (or use OpenTelemetry schema) and include the schema URL in the emitted telemetry. ## What is Out of Scope - The concept of schema does not attempt to fully describe the shape of telemetry. The schema for example does not define all possible valid values for attributes or expected data types for metrics, etc. It is not a goal. Our goal is narrowly defined to solve the following problem only: to allow OpenTelemetry Semantic Conventions to evolve over time. For that reason this document is concerned with _changes_ to the schema as opposed to the _full state_ of the schema. We do not preclude this though: the schema file format is extensible and in the future may allow defining the full state of the schema. - We intentionally limit the types of transformations of schemas to the bare minimum that is necessary to handle the most common changes that we believe OpenTelemetry Semantic Conventions will require in the near future. More types of transformations may be proposed in the future. This proposal does not attempt to support a comprehensive set of possible transformation types that can handle all possible changes to schemas that we can imagine. That would be too complicated and very likely superfluous. Any new transformation types MUST be proposed and added in the future to the schema file format when there is an evidence that they are necessary for the evolution of OpenTelemetry. ## Use Cases This section shows a couple interesting use-cases for the telemetry schemas (other uses-cases are also possible, this is not an exhaustive list). ### Full Schema-Aware Here is an example on a schema-aware observability system: Let's have a closer look at what happens with the Telemetry Source and Backend pair as the telemetry data is emitted, delivered and stored: In this example the telemetry source produces spans that comply with version 1.2.0 of OpenTelemetry schema, where the "deployment.environment" attribute is used to record that the span is coming from production. The telemetry consumer desires to store the telemetry in version 1.1.0 of OpenTelemetry schema. The schema translator compares the schema_url in the received span with the desired schema and sees that a version conversion is needed. It then applies the change that is described in the schema file and renames the attribute from "deployment.environment" to "environment" before storing the span. And here is for example how the schemas can be used to query stored data: ### Collector-Assisted Schema Transformation Here is a somewhat different use case, where the backend is not aware of schemas and we rely on OpenTelemetry Collector to translate the telemetry to a schema that the backend expects to receive. The "Schema Translate Processor" is configured, the target schema_url is specified and all telemetry data that passes through the Collector is converted to that target schema: ## Schema URL Schema URL is an identifier of a Schema. The URL specifies a location of a [Schema File](file_format_v1.1.0.md) that can be retrieved (so it is a URL and not just a URI) using HTTP or HTTPS protocol. Fetching the specified URL may return an HTTP redirect status code. The fetcher MUST follow the HTTP standard and honour the redirect response and fetch the file from the redirected URL. The last part of the URL path is the version number of the schema. ``` http[s]://server[:port]/path/ ``` The part of the URL preceding the `` is called Schema Family identifier. All schemas in one Schema Family have identical Schema Family identifiers. To create a new version of the schema copy the schema file for the last version in the schema family and add the definition of the new version. The schema file that corresponds to the new version MUST be retrievable at a new URL. Important: schema files are immutable once they are published. Once the schema file is retrieved it is recommended to be cached permanently. Schema files may be also packaged at build time with the software that anticipates it may need the schema (e.g. the latest OpenTelemetry schema file can be packaged at build time with OpenTelemetry Collector's schema translation processor). ## Schema Version Number Version number follows the MAJOR.MINOR.PATCH format, similar to SemVer 2.0. Version numbers use the [ordering rules](https://semver.org/#spec-item-11) defined by SemVer 2.0 specification. See how ordering is used in the [Order of Transformations](file_format_v1.1.0.md#order-of-transformations). Other than the ordering rules the schema version numbers do not carry any other semantic meaning. OpenTelemetry schema version numbers match OpenTelemetry [Semantic Conventions](/specification/semantic-conventions.md) version numbers, see more details [here](#opentelemetry-schema). ## OTLP Support To allow carrying the Schema URL in emitted telemetry OTLP includes a schema_url field in the messages: - The schema_url field in the ResourceSpans, ResourceMetrics, ResourceLogs messages applies to the contained Resource, Span, SpanEvent, Metric, LogRecord messages. - The schema_url field in the InstrumentationLibrarySpans message applies to the contained Span and SpanEvent messages. - The schema_url field in the InstrumentationLibraryMetrics message applies to the contained Metric messages. - The schema_url field in the InstrumentationLibraryLogs message applies to the contained LogRecord messages. - If schema_url field is non-empty both in Resource\* message and in the contained InstrumentationLibrary\* message then the value in InstrumentationLibrary\* message takes the precedence. ## API Support The OpenTelemetry API allows getting a [Tracer](../trace/api.md#get-a-tracer)/[Meter](../metrics/api.md#get-a-meter) that is associated with a Schema URL. ## OpenTelemetry Schema OpenTelemetry publishes it own schema at [opentelemetry.io/](https://opentelemetry.io/)schemas/<_version_>. The version number of the schema is the same as the semantic-conventions version number which publishes the schema. Every time a new semantic-conventions version is released a corresponding new schema MUST be released simultaneously. If the semantic-conventions release did not introduce any change the "changes" section of the corresponding version in the schema file will be empty. --- --- METRICS --- - Files Extracted: 14 - Estimated Token Budget: ~22788 tokens - Canonical Reference: https://codewiki.google/github.com/open-telemetry/opentelemetry-specification