opentelemetry-specification Documentation

Specifications for OpenTelemetry

4,371 stars Makefile 14 files ยท ~22,887 tokens #opentelemetry
RAW Doc

Technical Documentation: open-telemetry/opentelemetry-specification

โ„น๏ธ Provenance: Multi-Tier Ingestion ยท CodeWiki Reference

Overview (`README.md`)

OpenTelemetry Specification




The OpenTelemetry specification describes the cross-language
requirements and expectations for all OpenTelemetry implementations.

Change / contribution process

For details, see CONTRIBUTING.md, in particular read
Proposing a change before submitting a PR.

Questions

New to CNCF Slack? Create an account first, then join
the
#otel-specification
channel for questions and discussion.

Questions that need additional attention can be brought to the regular
specifications meeting. EU and US timezone friendly meeting is held every
Tuesday at 8 AM Pacific time. Meeting notes are held in the Google
doc.
APAC timezone friendly meetings are held on request. See
OpenTelemetry calendar.

Escalations to technical committee may be made over the
email.
Technical committee holds regular meetings, notes are held
here.

Specification compliance matrix by language

See Compliance of Implementations with
Specification.

Project Timeline

The current project status as well as information on notable past releases is found at
the OpenTelemetry project page.

Information about current work and future development plans is found at the
specification development milestones.

Versioning the Specification

Changes to the specification are versioned according to Semantic Versioning 2.0 and described in CHANGELOG.md. Layout changes are not versioned. Specific implementations of the specification should specify which version they implement.

Changes to the change process itself are not currently versioned but may be independently versioned in the future.

License

By contributing to OpenTelemetry Specification repository, you agree that your contributions will be licensed under its Apache 2.0 License.

Maintainers

The maintainers of the specification are a superset of the Technical Committee.

For more information about the maintainer role, see the community repository.

Approvers

Specification sponsors represent the approver role for the specification, along with additional responsibilities and privileges.

For more information about the approver role, see the community repository.

Triagers

For more information about the triager role, see the community repository.

Emeritus

Overview (`development/README.md`)

Features in Development

This folder is used for features in
Development. Development status is for components
that are on track to become part of the specification, but that require a faster cadence
of merges and collaboration.

Features in Development must be:

  • Implementable as a plugin to OpenTelemetry components (API, SDK, collector, etc.).
  • Be in active development or testing.
  • Approved as a general direction via OTEP process.

To avoid any confusion, all files in this directory must have a note about its Development status.

Development status precedes the Alpha maturity level.
All changes in the development folder go through the regular review process. Changes are allowed to be merged faster as completeness of a solution is not a requirement. Approval means that proposed changes are OK for experimentation.

When the feature or parts of it are developed far enough to declare them as an alpha version of a main project and move out of the Development status, it must go through a new OTEP PR and it must be expected that design and APIs will be changed. In fact, the same people who approved the experiment may likely be the most critical reviewers. It demonstrates an interest and involvement, not critique.

Overview (`oteps/README.md`)

OpenTelemetry Enhancement Proposal (OTEP)

Evolving OpenTelemetry at the speed of Markdown

OpenTelemetry uses an "OTEP" (similar to a RFC) process for proposing changes to the OpenTelemetry Specification.

[!NOTE]
OTEPs are documents of intent. They become requirements only after their contents have been
integrated into the actual Specification.

Table of Contents

What changes require an OTEP

The OpenTelemetry OTEP process is intended for changes that are cross-cutting - that is, applicable across languages and implementations - and either introduce new behaviour, change desired behaviour, or otherwise modify requirements.

In practice, this means that OTEPs should be used for such changes as:

  • New tracer configuration options
  • Additions to span data
  • New metric types
  • Modifications to extensibility requirements

On the other hand, they do not need to be used for such changes as:

  • Bug fixes
  • Rephrasing, grammatical fixes, typos, etc.
  • Refactoring
  • Things that affect only a single language or implementation

Note: The above lists are intended only as examples and are not meant to be exhaustive. If you don't know whether a change requires an OTEP, please feel free to ask!

Extrapolating cross-cutting changes

Sometimes, a change that is only immediately relevant within a single language or implementation may be indicative of a problem upstream in the specification. We encourage you to add an OTEP if and when you notice such cases.

OTEP scope

While OTEPs are intended for "significant" changes, we recommend trying to keep each OTEP's scope as small as makes sense. A general rule of thumb is that if the core functionality proposed could still provide value without a particular piece, then that piece should be removed from the proposal and used instead as an example (and, ideally, given its own OTEP!).

For example, an OTEP proposing configurable sampling and various samplers should instead be split into one OTEP proposing configurable sampling as well as an OTEP per sampler.

Writing an OTEP

  • First, fork this repository.
  • Copy 0000-template.md to 0000-my-OTEP.md, where my-OTEP is a title relevant to your proposal, and 0000 is the OTEP ID.
    Leave the number as is for now. Once a Pull Request is made, update this ID to match the PR ID.
  • Fill in the template. Put care into the details: It is important to present convincing motivation, demonstrate an understanding of the design's impact, and honestly assess the drawbacks and potential alternatives.

Submitting the OTEP

  • An OTEP is proposed by posting it as a PR. Once the PR is created, update the OTEP filename to use the PR ID as the OTEP ID.
  • An OTEP is approved when four reviewers github-approve the PR. The OTEP is then merged.
  • If an OTEP is rejected or withdrawn, the PR is closed. Note that these OTEPs submissions are still recorded, as GitHub retains both the discussion and the proposal, even if the branch is later deleted.
  • If an OTEP discussion becomes long, and the OTEP then goes through a major revision, the next version of the OTEP can be posted as a new PR, which references the old PR. The old PR is then closed. This makes OTEP review easier to follow and participate in.

Integrating the OTEP into the Spec

  • Once an OTEP is approved, an issue is created in this repository to integrate the OTEP into the actual spec.
  • When reviewing the spec PR for the OTEP, focus on whether the spec is written clearly, and reflects the changes approved in the OTEP. Please abstain from relitigating the approved OTEP changes at this stage.
  • An OTEP is integrated when four reviewers github-approve the spec PR. The PR is then merged, and the spec is versioned.

Implementing the OTEP

  • Once an OTEP is integrated into the spec, an issue is created in the backlog of every relevant OpenTelemetry implementation.
  • PRs are made until the all the requested changes are implemented.
  • The status of the OpenTelemetry implementation is updated to reflect that it is implementing a new version of the spec.

Changes to the OTEP process

The hope and expectation is that the OTEP process will evolve with the OpenTelemetry. The process is by no means fixed.

Have suggestions? Concerns? Questions? Please raise an issue or raise the matter on our community chat.

Background on the OpenTelemetry OTEP process

Our OTEP process borrows from the Rust RFC and Kubernetes Enhancement Proposal processes, the former also being very influential on the latter; as well as the OpenTracing OTEP process. Massive kudos and thanks to the respective authors and communities for providing excellent prior art ๐Ÿ’–

Overview (`specification/README.md`)

OpenTelemetry Specification

Contents

Notation Conventions and Compliance

The keywords "MUST", "MUST NOT", "REQUIRED", "SHOULD",
"SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in the
specification are to be interpreted as described in BCP
14
[RFC2119]
[RFC8174] when, and only when, they
appear in all capitals, as shown here.

An implementation of the specification is not compliant if it fails to
satisfy one or more of the "MUST", "MUST NOT", "REQUIRED",
requirements defined in the specification. Conversely, an
implementation of the specification is compliant if it satisfies all the
"MUST", "MUST NOT", "REQUIRED", requirements defined in the specification.

Project Naming

  • The official project name is "OpenTelemetry" (with no space between "Open" and
    "Telemetry").
  • The official acronym used by the OpenTelemetry project is "OTel". "OT" MAY be
    used only as a part of a longer acronym, such as OTCA (OpenTelemetry Certified Associate).
  • The official names for sub-projects, like language specific implementations,
    follow the pattern of "OpenTelemetry {the name of the programming language,
    runtime or component}", for example, "OpenTelemetry Python", "OpenTelemetry
    .NET" or "OpenTelemetry Collector".

About the project

See the project repository for information about the following, and more:

Overview (`specification/common/README.md`)

Common specification concepts

Status: Stable

Table of Contents

AnyValue

AnyValue is either:

  • a primitive type: string, boolean, double precision floating point
    (IEEE 754-1985), or signed 64 bit integer,
  • a homogeneous array of primitive type values. A homogeneous array MUST NOT
    contain values of different types.
  • a byte array.
  • an array of AnyValue,
  • a map<string, AnyValue>,
  • an empty value if supported by the language,
    (e.g. null, undefined in JavaScript/TypeScript, None in Python, nil in Go/Ruby, not supported in Erlang, etc.)

Arbitrary deep nesting of values for arrays and maps is allowed (essentially
allows to represent an equivalent of a JSON object).

APIs SHOULD be documented in a way to communicate to users that using array and
map values may carry higher performance overhead compared to primitive values.

AnyValues expressing an empty value, a numerical value of zero, an empty string,
or an empty array are considered meaningful and MUST be stored and passed on to
processors / exporters.

While null is a valid attribute value, its use within homogeneous arrays
SHOULD generally be avoided unless language constraints make this impossible.
However, if it is impossible to make sure that no null values are accepted
(e.g. in languages that do not have appropriate compile-time type checking),
null values within arrays MUST be preserved as-is (i.e., passed on to
processors / exporters as null). If exporters do not support exporting null
values, they MAY replace those values by 0, false, or empty strings.
This is required for map/dictionary structures represented as two arrays with
indexes that are kept in sync (e.g., two attributes header_keys and header_values,
both containing an array of strings to represent a mapping
header_keys[i] -> header_values[i]).

map

map<string, AnyValue> is a map of string keys to AnyValue values.
The keys in the map are unique (duplicate keys are not allowed).

Case sensitivity of keys MUST be preserved.
Keys that differ in casing are treated as distinct keys.

The representation of the map is language-dependent.

The implementation MUST by default enforce that the exported maps contain only
unique keys. The enforcement of uniqueness may be performed
in a variety of ways as it best fits the limitations of the particular
implementation (e.g. by removing duplicates).

The implementation MAY have an option to allow exporting maps with duplicate keys
(e.g. for better performance).
If such option is provided, it MUST be documented that for many receivers,
handling of maps with duplicate keys is unpredictable and it is the users'
responsibility to ensure keys are not duplicate.

Maps are equal when they contain the same key-value pairs,
irrespective of the order in which those elements appear
(unordered collection equality).

AnyValue representation for non-OTLP protocols

For protocols that do not natively support some of the AnyValue types,
those values SHOULD be represented as strings following the encoding rules below.

[!NOTE]
This string representation is lossy. Type information is lost as all
values are converted to strings, and precision loss may occur for numeric values
(particularly for floating point numbers and large integers that exceed the
precision capabilities of the receiving system's string-to-number conversion).

Strings

Strings SHOULD be represented as-is without any additional encoding.
They SHOULD NOT be encoded as JSON strings (with explicit surrounding quotes).

Examples: hello world, (the empty string)

Booleans

Booleans SHOULD be represented as
JSON booleans.

Examples: true, false

Integers

Integers SHOULD be represented as
JSON numbers.

Examples: 42, -123

Floating Point Numbers

Floating point numbers SHOULD be represented as
JSON numbers.

The special floating point values NaN and Infinity SHOULD be represented as
NaN, Infinity, and -Infinity.
They SHOULD NOT be encoded as JSON strings (with explicit surrounding quotes).

Examples: 3.14159, 1.23e10, NaN, Infinity, -Infinity

Byte Arrays

Byte arrays SHOULD be Base64-encoded.
They SHOULD NOT be encoded as JSON strings (with explicit surrounding quotes).

Example: aGVsbG8gd29ybGQ=

Empty Values

Empty values SHOULD be represented as the empty string.
They SHOULD NOT be encoded as a JSON string (with explicit surrounding quotes).

Arrays

Arrays, except for byte arrays, SHOULD be represented as
JSON arrays.

Array elements SHOULD be represented as JSON values using the following rules:

Examples: [], [1, "-Infinity", "a", true, {"nested": "aGVsbG8gd29ybGQ="}]

Maps

Maps SHOULD be represented as JSON objects.

Map keys SHOULD be represented as JSON string member names.

Map values SHOULD be represented using the same rules as
array elements.

Examples: {}, {"a": "-Infinity", "b": 2, "c": [3, null]}

Attribute

An Attribute is a key-value pair, which MUST have the following properties:

  • The attribute key MUST be a non-null and non-empty string.
    • Case sensitivity of keys is preserved. Keys that differ in casing are treated as distinct keys.
  • The attribute value MUST be one of types defined in AnyValue.

Attributes are equal when their keys and values are equal.

See Attribute Naming for naming guidelines.

See Requirement Level for requirement levels guidelines.

See this document to find out how to map values obtained
outside OpenTelemetry into OpenTelemetry attribute values.

Attribute representation for non-OTLP

For non-OTLP protocols that need to
represent a single Attribute as a string, the RECOMMENDED form is a
JSON object
containing a single name/value pair (member).

The attribute key SHOULD be represented as a JSON object member
name.

The attribute value SHOULD be represented as a JSON object member value and
follow the encoding rules defined in
AnyValue representation for non-OTLP protocols,
as it would be represented as an element in an array and a value in
a map.

Examples: {"http.request.method": "GET"}, {"retries": 3},
{"payload": "aGVsbG8gd29ybGQ="}, {"session.id": null},
{"colors": ["red", "blue"]}, {"context": {"nested": true}}

[!NOTE]
This string representation is lossy. Type information is lost as all
values are converted to strings, and precision loss may occur for numeric values
(particularly for floating point numbers and large integers that exceed the
precision capabilities of the receiving system's string-to-number conversion).

Attribute Collections

Resources,
Instrumentation Scopes,
Metric points,
Spans, Span
Events, Span
Links and
Log Records,
contain a collection of attributes.

Attribute Collections are top-level collections of key-value pairs used in
OpenTelemetry data models.
Note that they are distinct from map<string, AnyValue>,
which is a type of AnyValue used to represent nested data
structures e.g. in attribute values and log record body.

Implementation MUST by default enforce that the exported attribute collections
contain only unique keys. The enforcement of uniqueness may be performed
in a variety of ways as it best fits the limitations of the particular
implementation (e.g. by removing duplicates).

Normally for the telemetry generated using OpenTelemetry SDKs the attribute
key-value pairs are set via an API that either accepts a single key-value pair
or a collection of key-value pairs. Setting an attribute with the same key as an
existing attribute SHOULD overwrite the existing attribute's value. See for
example Span's SetAttribute API.

A typical implementation of SetAttribute API
will enforce the uniqueness by overwriting any existing attribute values pending
to be exported, so that when the Span is eventually exported the exporters see
only unique attributes. The OTLP format in particular requires that exported
Resources, Spans, Metric data points and Log Records contain only unique
attributes.

Some other implementations may use a streaming approach where every
SetAttribute API call immediately results in
that individual attribute value being exported using a streaming wire protocol.
In such cases the enforcement of uniqueness will likely be the responsibility of
the recipient of this data.

Implementations MAY have an option to allow exporting attribute collections
with duplicate keys (e.g. for better performance).
If such option is provided, it MUST be documented that for many receivers,
handling of maps with duplicate keys is unpredictable and it is the users'
responsibility to ensure keys are not duplicate.

Collection of attributes are equal when they contain the same attributes,
irrespective of the order in which those elements appear
(unordered collection equality).

Attribute Collection representation for non-OTLP

For non-OTLP protocols that need to represent an Attribute Collection as a
string, the RECOMMENDED form is a
JSON object.

Each attribute key SHOULD be represented as a JSON object member name.

Each attribute value SHOULD be represented as the corresponding JSON object
member value and follow the encoding rules defined in
AnyValue representation for non-OTLP protocols,
as it would be represented as an element in an array and a value in
a map.

This representation follows the same JSON object form as maps, but
applies to top-level Attribute Collections rather than nested
map<string, AnyValue> values.

Examples: {}, {"http.request.method": "GET", "retries": 3},
{"payload": "aGVsbG8gd29ybGQ=", "session.id": null},
{"colors": ["red", "blue"], "context": {"nested": true}}

Attribute Limits

Execution of erroneous code can result in unintended attributes. If there are no
limits placed on attribute collections, they can quickly exhaust available memory,
resulting in crashes that are difficult to recover from safely.

By default an SDK SHOULD apply truncation as per the list of
configurable parameters below.

If an SDK provides a way to:

  • set an attribute value length limit such that for each
    attribute value:
    • if it is a string, if it exceeds that limit (counting any character in it as
      1), SDKs MUST truncate that value, so that its length is at most equal
      to the limit,
    • if it is a byte array,
      if its length exceeds that limit (counting each byte as 1),
      SDKs MUST truncate that value, so that its length is at most equal to the limit,
    • if it is an array of strings, then apply the limit to
      each value within the array separately,
    • if it is an array of AnyValue,
      then apply the limit to each element of the array separately (and recursively),
    • if it is a map,
      then apply the limit to each value within the map separately (and recursively),
    • otherwise a value MUST NOT be truncated;
  • set an attribute count limit such that:
    • if adding an attribute to an attribute collection would result
      in exceeding the limit (counting each attribute in the collection as 1),
      the SDK MUST discard that attribute, so that the total number of attributes in
      an attribute collection is at most equal to the limit;
    • the count limit applies only to top-level attributes, not to nested key-value
      pairs within maps;
    • otherwise an attribute MUST NOT be discarded.
  • set an attribute value depth limit such that for each attribute value:
    • the SDK MUST start counting depth at 1 for the top-level attribute value,
      and increment depth when descending into arrays (both homogeneous and
      heterogeneous) or maps;
    • arrays or map at a depth greater than the limit MUST be replaced with an
      empty value;
    • otherwise a value MUST NOT be changed due to the depth limit.

There MAY be a log emitted to indicate to the user that an attribute was
truncated, discarded, or replaced due to a limit. To prevent excessive logging,
the log MUST NOT be emitted more than once per record on which an attribute is
set.

If the SDK implements the limits above, it MUST provide a way to change these
limits programmatically. Names of the configuration options SHOULD be the same as
in the list below.

An SDK MAY implement model-specific limits, for example
SpanAttributeCountLimit or LogRecordAttributeCountLimit. If both a general
and a model-specific limit are implemented, then the SDK MUST first attempt to
use the model-specific limit, if it isn't set, then the SDK MUST attempt to use
the general limit. If neither are defined, then the SDK MUST try to use the
model-specific limit default value, followed by the global limit default value.

Note that the limits apply only to attribute collections. Attribute value length
and depth limits apply recursively to attribute values, but they do not apply to
values within other data structures such as
LogRecord.Body.

Configurable Parameters

  • AttributeCountLimit (Default=128) - Maximum allowed attribute count per record;
  • AttributeValueLengthLimit (Default=Infinity) - Maximum allowed attribute value length (applies to string values and byte arrays);
  • AttributeValueDepthLimit (Default=64) - Maximum allowed attribute value depth (applies to arrays and maps);

Exempt Entities

Resource attributes SHOULD be exempt from the limits described above as resources
are not susceptible to the scenarios (auto-instrumentation) that result in
excessive attributes count or size. Resources are also sent only once per batch
instead of per span so it is relatively cheaper to have more/larger attributes
on them. Resources are also immutable by design and they are generally passed
down to TracerProvider along with limits. This makes it awkward to implement
attribute limits for Resources.

Attributes, which belong to Metrics, are exempt from the limits described above
at this time, as discussed in
Metrics Attribute Limits.

Overview (`specification/configuration/README.md`)

Configuration

OpenTelemetry SDK components are highly configurable. This specification
outlines the mechanisms by which OpenTelemetry components can be configured. It
does not attempt to specify the details of what can be configured.

Configuration Interfaces

Programmatic

The SDK MUST provide a programmatic interface for all configuration.
This interface SHOULD be written in the language of the SDK itself.
All other configuration mechanisms SHOULD be built on top of this interface.

An example of this programmatic interface is accepting a well-defined
struct on an SDK builder class. From that, one could build a CLI that accepts a
file (YAML, JSON, TOML, ...) and then transforms into that well-defined struct
consumable by the programmatic interface. See
declarative configuration.

Environment variables

Environment variable configuration defines a set of language agnostic
environment variables for common configuration goals.

See OpenTelemetry Environment Variable Specification.

Declarative configuration

Declarative configuration provides a mechanism for configuring OpenTelemetry
which is more expressive and full-featured than
the environment variable based scheme, and language
agnostic in a way not possible with programmatic configuration.
Notably, declarative configuration defines tooling allowing users to load
OpenTelemetry components according to a file-based representation of a
standardized configuration data model.

Declarative configuration consists of the following main components:

  • Data model defines data structures which allow users to
    specify an intended configuration of OpenTelemetry SDK components and
    instrumentation. The data model includes a file-based representation.
  • Instrumentation configuration API allows
    instrumentation libraries to consume configuration by reading relevant
    configuration options during initialization.
  • Configuration SDK defines SDK capabilities around file
    configuration, including an In-Memory configuration model, support for
    referencing custom plugin components in configuration files, and
    operations to parse configuration files and interpret the configuration data
    model.

See also supplementary guidelines.

Other Mechanisms

Additional configuration mechanisms SHOULD be provided in whatever
language/format/style is idiomatic for the language of the SDK. The
SDK can include as many configuration mechanisms as appropriate.

Overview (`specification/context/README.md`)

Context

Status: Stable.

Table of Contents

Overview

A Context is a propagation mechanism which carries execution-scoped values
across API boundaries and between logically associated execution units.
Cross-cutting concerns access their data in-process using the same shared
Context object.

A Context MUST be immutable, and its write operations MUST
result in the creation of a new Context containing the original
values and the specified values updated.

Languages are expected to use the single, widely used Context implementation
if one exists for them. In the cases where an extremely clear, pre-existing
option is not available, OpenTelemetry MUST provide its own Context
implementation. Depending on the language, its usage may be either explicit
or implicit.

Users writing instrumentation in languages that use Context implicitly are
discouraged from using the Context API directly. In those cases, users will
manipulate Context through cross-cutting concerns APIs instead, in order to
perform operations such as setting trace or baggage entries for a specified
Context.

A Context is expected to have the following operations, with their
respective language differences:

Create a key

Keys are used to allow cross-cutting concerns to control access to their local state.
They are unique such that other libraries which may use the same context
cannot accidentally use the same key. It is recommended that concerns mediate
data access via an API, rather than provide direct public access to their keys.

The API MUST accept the following parameter:

  • The key name. The key name exists for debugging purposes and does not uniquely identify the key. Multiple calls to CreateKey with the same name SHOULD NOT return the same value unless language constraints dictate otherwise. Different languages may impose different restrictions on the expected types, so this parameter remains an implementation detail.

The API MUST return an opaque object representing the newly created key.

Get value

Concerns can access their local state in the current execution state
represented by a Context.

The API MUST accept the following parameters:

  • The Context.
  • The key.

The API MUST return the value in the Context for the specified key.

Set value

Concerns can record their local state in the current execution state
represented by a Context.

The API MUST accept the following parameters:

  • The Context.
  • The key.
  • The value to be set.

The API MUST return a new Context containing the new value.

Optional Global operations

These operations are expected to only be implemented by languages
using Context implicitly, and thus are optional. These operations
SHOULD only be used to implement automatic scope switching and define
higher level APIs by SDK components and OpenTelemetry instrumentation libraries.

Get current Context

The API MUST return the Context associated with the caller's current execution unit.

Attach Context

Associates a Context with the caller's current execution unit.

The API MUST accept the following parameters:

  • The Context.

The API MUST return a value that can be used as a Token to restore the previous
Context.

Note that every call to this operation should result in a corresponding call to
Detach Context.

Detach Context

Resets the Context associated with the caller's current execution unit
to the value it had before attaching a specified Context.

This operation is intended to help making sure the correct Context
is associated with the caller's current execution unit. Users can
rely on it to identify a wrong call order, i.e. trying to detach
a Context that is not the current instance. In this case the operation
can emit a signal to warn users of the wrong call order, such as logging
an error or returning an error value.

The API MUST accept the following parameters:

  • A Token that was returned by a previous call to attach a Context.

The API MAY return a value used to check whether the operation
was successful or not.

Overview (`specification/entities/README.md`)

Entities

Table of Contents

Overview

Entity represents an object of interest associated with produced telemetry:
traces, metrics, logs, profiles etc.

Entities can be instantiated through two complementary approaches:

  1. Pull-based Model: Entities discover themselves from the current environment by examining the runtime context, system properties, metadata services, and other available sources of entity information.

  2. Push-based Model: Entities are explicitly provided from external sources, typically through environment variables or configuration. This approach allows entity information to be propagated across process boundaries. See Entity Propagation for more details.

Specifications

Overview (`specification/logs/README.md`)

OpenTelemetry Logging

Table of Contents

Introduction

Of all telemetry signals logs have probably the biggest legacy. Most programming
languages have built-in logging capabilities or well-known, widely used logging
libraries.

For metrics and traces OpenTelemetry takes the approach of a clean-sheet design,
specifies a new API and provides full implementations of this API in multiple
languages.

Our approach with logs is somewhat different. For OpenTelemetry to be
successful in logging space we need to support existing legacy of logs and
logging libraries, while offering improvements and better integration with the
rest of observability world where possible.

This is in essence the philosophy behind OpenTelemetry's logs support. We
embrace existing logging solutions and make sure OpenTelemetry works nicely with
existing logging libraries, log collection and processing solutions.

Limitations of non-OpenTelemetry Solutions

Unfortunately existing logging solutions are currently weakly integrated with
the rest of the observability signals. Logs typically have limited support in
tracing and monitoring tools in the form of links that use available and often
incomplete correlation information (such as the time and origin attributes).
This correlation may be fragile because attributes are often added to logs,
traces and metrics via different means (e.g. using different collection agents).
There is no standardized way to include the information about the origin and
source of logs (such as the application and the location/infrastructure where
the application runs) that is uniform with traces and metrics and allows all
telemetry data to be fully correlated in a precise and robust manner.

Similarly, logs have no standardized way to propagate and record the request
execution context. In distributed systems this often results in a disjoint set
of logs collected from different components of the system.

This is how a typical non-OpenTelemetry observability collection pipeline looks
like today:

There are often different libraries and different collection agents, using
different protocols and data models, with telemetry data ending up in separate
backends that don't know how to work well together.

OpenTelemetry Solution

Distributed tracing introduced the notion of trace context propagation.

Fundamentally, though, nothing prevents the logs to adopt the same context
propagation concepts. If the recorded logs contained trace context identifiers
(such as trace and span IDs or user-defined baggage) it would result
in much richer correlation between logs and traces, as well as correlation
between logs emitted by different components of a distributed system. This would
make logs significantly more valuable in distributed systems.

This is one of the promising evolutionary directions for observability tools.
Standardizing log correlation with traces and metrics, adding support for
distributed context propagation for logs, unification of source attribution of
logs, traces and metrics will increase the individual and combined value of
observability information for legacy and modern systems. This is the vision of
OpenTelemetry's collection of logs, traces and metrics:

We emit logs, traces and metrics in a way that is compliant with OpenTelemetry
data models, send the data through OpenTelemetry Collector, where it can be
enriched and processed in a uniform manner. For example, Collector can add to
all telemetry data coming from a Kubernetes Pod several attributes that describe
the pod and it can be done automatically using
k8sattributesprocessor
without the need for the Application to do anything special. Most importantly
such enrichment is completely uniform for all 3 signals. The Collector
guarantees that logs, traces and metrics have precisely the same attribute names
and values describing the Kubernetes Pod that they come from. This enables exact
and unambiguous correlation of the signals by the Pod in the backend.

For traces and metrics OpenTelemetry defines a new API that application
developers must use to emit traces and metrics.

For logs we did not take the same path. We realized that there is a much bigger
and more diverse legacy in logging space. There are many existing logging
libraries in different languages, each having their own API. Many programming
languages have established standards for using particular logging libraries. For
example in Java world there are several highly popular and widely used logging
libraries, such as Log4j or Logback.

There are also countless existing prebuilt applications or systems that emit
logs in certain formats. Operators of such applications have no or limited
control on how the logs are emitted. OpenTelemetry needs to support these logs.

Given the above state of the logging space we took the following approach:

  • OpenTelemetry defines a log data model. The purpose of the
    data model is to have a common understanding of what a LogRecord is, what
    data needs to be recorded, transferred, stored and interpreted by a logging
    system.

  • Newly designed logging systems are expected to emit logs according to
    OpenTelemetry's log data model. More on this later.

  • Existing log formats can be
    unambiguously mapped to
    OpenTelemetry log data model. OpenTelemetry Collector can read such logs and
    translate them to OpenTelemetry log data model.

  • OpenTelemetry defines a Logs API
    for emitting LogRecords. It was primarily designed
    for library authors to build log appenders,
    which use the API to bridge between existing logging libraries and the
    OpenTelemetry log data model. The Logs API can also be used directly by
    applications, which is particularly important for:

    • Instrumentation libraries to avoid coupling to a particular logging library.
    • Emitting structured events following semantic conventions,
      especially when the existing logging library cannot emit structured data or specify event names.
    • Scenarios requiring direct control over log emission without
      intermediary logging library.

    Note that existing logging libraries generally provide a much richer set of
    features than what is defined in OpenTelemetry. Yet, languages may provide
    an ergonomic API for better developer experience
    when using it directly.

  • OpenTelemetry defines an SDK implementation of the API,
    which enables configuration of processing
    and exporting LogRecords.

This approach allows OpenTelemetry to read existing system and application logs,
provides a way for newly built application to emit rich, structured,
OpenTelemetry-compliant logs, and ensures that all logs are eventually
represented according to a uniform log data model on which the backends can
operate.

Later in this document we will discuss in more details
how various log sources are handled by
OpenTelemetry, but first we need to describe in more details an important
concept: the log correlation.

Log Correlation

Logs can be correlated with the rest of observability data in a few dimensions:

  • By the time of execution. Logs, traces and metrics can record the moment
    of time or the range of time the execution took place. This is the most basic
    form of correlation.

  • By the execution context, also known as the trace context. It is a
    standard practice to record the execution context (trace and span IDs as well
    as user-defined context) in the spans. OpenTelemetry extends this practice to
    logs where possible by including TraceId and
    SpanId in the LogRecords. This allows to
    directly correlate logs and traces that correspond to the same execution
    context. It also allows to correlate logs from different components of a
    distributed system that participated in the particular request execution.

  • By the origin of the telemetry, also known as the Resource context.
    OpenTelemetry traces and metrics contain information about the Resource they
    come from. We extend this practice to logs by including the
    Resource in LogRecords.

These 3 correlations can be the foundation of powerful navigational, filtering,
querying and analytical capabilities. OpenTelemetry aims to record and collects
logs in a manner that enables such correlations.

Legacy and Modern Log Sources

It is important to distinguish several sorts of legacy and modern log sources.
Firstly, this directly affects how exactly we get access to these logs and how
we collect them. Secondly, we have varying levels of control over how these logs
are generated and whether we can amend the information that can be included in
the logs.

Below we list several categories of logs and describe what can be possibly done
for each category to have better experience in the observability solutions.

System Logs

These are logs generated by the operating system and over which we have no
control. We cannot change the format or affect what information is included.
Examples of system format are Syslog and Windows Event Logs.

System logs are written at the host level (which may be physical, virtual or
containerized) and have a predefined format and content (note that applications
may also be able to write records to standard system logs: this case is covered
below in the Third-Party Applications section).

System operations recorded in the logs can be a result of a request execution.
However system logs either do not include any data about the trace context or
if included it is highly idiosyncratic and thus difficult to identify, parse and
use. This makes it nearly impossible to perform trace context correlation for
system logs. However we can and should automatically enrich system logs with the
resource context - the information about the host that is available during
collection. This can include the hostname, IP address, container or pod name,
etc. This information should be added to the Resource field of collected log
data.

OpenTelemetry Collector can read system logs (link TBD) and automatically enrich
them with Resource information using the
resourcedetection
processor.

Infrastructure Logs

These are logs generated by various infrastructure components, such as
Kubernetes events. Like system logs, the
infrastructure logs lack a trace context and can be enriched by the resource
context - information about the node, pod, container, etc.

OpenTelemetry Collector or other agents can be used to query logs from most
common infrastructure controllers.

Third-party Application Logs

Applications typically write logs to standard output, to files or other
specialized medium (e.g. Windows Event Logs for applications). These logs can be
in many different formats, spanning a spectrum along these variations:

  • Free-form text formats with no easily automatable and reliable way to parse
    structured data from them.

  • Better specified and sometimes customizable formats that can be parsed to
    extract structured data (such as Apache logs or RFC5424 Syslog).

  • Formally structured formats (e.g. JSON files with well-defined schema or
    Windows Event Log).

The collection system needs to be able to discover most commonly used
applications and have parsers that can convert these logs into a structured
format. Like system and infrastructure logs, application logs often lack request
context but can be enriched by resource context, including the attributes that
describe the host and infrastructure as well as application-level attributes
(such as the application name, version, name of the database - if it is a DBMS,
etc).

OpenTelemetry recommends to collect application logs using Collector's
filelog receiver.
Alternatively, another log collection agent, such as FluentBit, can collect
logs,
then send
to OpenTelemetry Collector where the logs can be further processed and enriched.

Legacy First-Party Applications Logs

These are applications that are created in-house. People tasked with setting up
log collection infrastructure sometimes are able to modify these applications to
alter how logs are written and what information is included in the logs. For
example, the applicationโ€™s log formatters may be reconfigured to output JSON
instead of plain text and by doing so help improve the reliability of log
collection.

More significant modifications to these applications can be done manually by
their developers, such as addition of the trace context to every log
statement, however this is likely going to be vanishingly rare due to the effort
required.

As opposed to manual efforts we have an interesting opportunity to "upgrade"
application logs in a less laborious way by providing full or semi
auto-instrumenting solutions that modify trace logging libraries used by the
application to automatically output the trace context such as the trace ID or
span ID with every log statement. The trace context can be automatically
extracted from incoming requests if standard compliant request propagation is
used, e.g. via W3C TraceContext. In
addition, the requests outgoing from the application may be injected with the
same trace context data, thus resulting in context propagation through the
application and creating an opportunity to have full trace context in logs
collected from all applications that can be instrumented in this manner.

Some logging libraries are designed to be extended in this manner relatively
easily. There is no need to actually modify the libraries, instead we can
implement "log appender" or "log bridge" components for such libraries and
implement the additional LogRecord enrichment in these components.

There are typically 2 ways to collect logs from these applications.

Via File or Stdout Logs

The first approach, assuming the logs are written to files or to standard
output, requires ability to read file logs, tail them, work correctly when log
rotation is used, optionally also parse the logs to convert them into more
structured formats. Parsing requires support for different parser types, which
can also be configured to parse custom formats as well as ability to add custom
parsers. Examples of common formats that parsers need to support are: CSV,
Common Log Format, Labeled Tab-separated Values (LTSV), key-value Pair format,
JSON, etc. To support this approach OpenTelemetry recommends to collect logs
using OpenTelemetry
Collector.

Alternatively, if the Collector does not have the necessary file reading and
parsing capabilities, another log collection agent, such as FluentBit can
collect the logs,
then send the logs
to OpenTelemetry Collector.

The benefit of using an intermediary medium is that how logs are produced and
where they are written by the application requires no or minimal changes. The
downside is that it requires the often non-trivial log file reading and parsing
functionality. Parsing may also be not reliable if the output format is not
well-defined. For details on recording and parsing trace context,
see Trace Context in Non-OTLP Log Formats.

Direct to Collector

The second approach is to modify the application so that the logs are output via
a network protocol, e.g. via
OTLP.
The most convenient way to achieve this is to provide addons or extensions to
the commonly used logging libraries. The addons implement sending over such
network protocols, which would then typically require small, localized changes
to the application code to change the logging target.

The application logs will be also enriched by the resource context, similarly to
how it is done for third-party applications and so will potentially have full
correlation information across all context dimensions.

The downside of this approach is that the simplicity of having the logs in a
local file is lost (e.g. ability to easily inspect the log file locally) and
requires a full buy-in in OpenTelemetry's logging approach. This approach also
only works if the destination that the logs need to be delivered is able to
receive logs via the network protocol that OpenTelemetry can send in.

The benefits of this approach is that it emits the logs in well-defined, formal,
highly structured format, removes all complexity associated with file logs, such
as parsers, log tailing and rotation. It also enables the possibility to send
logs directly to the logging backend without using a log collection agent.

To facilitate both approaches described above OpenTelemetry provides
a API and SDK, which can be used together with existing
logging libraries to automatically inject the trace context in the emitted logs,
and provide an easy way to send the logs via OTLP. Instead of
modifying each logging statement, log appenders
use the API to bridge logs from existing logging libraries to the OpenTelemetry
data model, where the SDK controls how the logs are processed and exported.
Application developers only need to configure the Appender and SDK at
application startup.

New First-Party Application Logs

These are greenfield developments. OpenTelemetry provides recommendations and
best practices about how to emit logs (along with traces and metrics) from these
applications.

Applications have several options for emitting logs:

  1. Using existing logging libraries with OpenTelemetry log appenders: For
    applicable languages and frameworks, auto-instrumentation or simple
    configuration of a logging library to use an OpenTelemetry log appender will
    be the easiest way to emit context-enriched logs. As already described earlier,
    we provide extensions to some popular logging libraries to support manual
    instrumentation cases. The extensions support the inclusion of the trace
    context in the logs and allow sending logs via OTLP protocol to the backend
    or to the Collector, bypassing the need to have the logs represented as text
    files.

  2. Using the OpenTelemetry Logs API directly: Applications can use the
    Logs API directly to emit structured logs and events. This approach
    works well for emitting events following semantic conventions
    and enables easier reuse of attributes used across different signals. Note
    that a language can provide a more convenient ergonomic API.

Regardless of the approach, emitted logs are automatically augmented by
application-specific resource context (e.g. process ID, programming language,
logging library name and version, etc). Full correlation across all context
dimensions will be available for these logs.

This is how a typical new application uses OpenTelemetry API, SDK and the
existing log libraries:

OpenTelemetry Collector

To enable log collection according to this specification we use OpenTelemetry
Collector.

The following functionality exists to enable log collection:

  • Support for log data type and log pipelines based on the
    log data model. This includes processors such as
    attributesprocessor
    that can operate on log data.

  • Ability to read logs from text files, tail the files, understand common log
    rotation schemes, watch directories for log file creation, ability to
    checkpoint file positions and resume reading from checkpoints. This ability is
    implemented by using Collector's
    filelog receiver
    or using an externally running agent (such as FluentBit).

  • Ability to parse logs in common text formats and to allow end users to
    customize parsing formats and add custom parsers as needed. Collector's
    parsers
    or parsing in the external agent is used for this.

  • Ability to receive logs via common network protocols for logs, such as Syslog
    and interpret them according to semantic conventions defined in this
    specification. FluentBit or similar agent is used for this. Over time some of
    this functionality may be migrated directly to the Collector.

  • Ability to send logs via common network protocols for logs, such as Syslog, or
    vendor-specific log formats. Collector contains exporters that directly
    implement this ability.

Auto-Instrumenting Existing Logging

We can provide auto-instrumentation for most popular logging libraries. The
auto-instrumented logging statements will do the following:

  • Read incoming trace context (this is part of broader instrumentation that
    auto-instrumenting libraries perform).

  • Configure logging libraries to use trace ID and span ID fields from request
    context as logging context and automatically include them in all logged
    statements.

This is possible to do for certain languages (e.g. in Java) and we can reuse
existing open source libraries
that do this.

A further optional modification would be to auto-instrument loggers to send logs
directly to the backend via OTLP instead or in addition to writing to a file or
standard output.

Specifications

References

Overview (`specification/metrics/README.md`)

OpenTelemetry Metrics

Table of Contents

Overview

Design Goals

Given there are many well-established metrics solutions that exist today, it is
important to understand the goals of OpenTelemetryโ€™s metrics effort:

  • Being able to connect metrics to other signals. For example, metrics and
    traces can be correlated via exemplars, and metrics attributes can be enriched
    via Baggage and Context.
    Additionally, Resource can be applied to
    logs/metrics/traces
    in a consistent way.

  • Providing a path for OpenCensus customers to
    migrate to OpenTelemetry    . This was the original goal of OpenTelemetry -
    converging OpenCensus and OpenTracing. We will focus on providing the
    semantics and capability, instead of doing a 1-1 mapping of the APIs.

  • Working with existing metrics instrumentation protocols and standards.
    Here is the minimum set of goals:

    • Providing full support for Prometheus - users
      should be able to use OpenTelemetry clients and
      Collector to collect and export metrics, with
      the ability to achieve the same functionality as the native Prometheus
      clients.
    • Providing the ability to collect StatsD
      metrics using the OpenTelemetry Collector.

Concepts

API

The OpenTelemetry Metrics API ("the API" hereafter) serves two purposes:

  • Capturing raw measurements efficiently and simultaneously.
  • Decoupling the instrumentation from the SDK, allowing the SDK to be
    specified/included in the application.

When no SDK is explicitly included/enabled in the application, no
telemetry data will be collected. Please refer to the overall OpenTelemetry
API concept and API and Minimal
Implementation for
more information.

SDK

The OpenTelemetry Metrics SDK ("the SDK" hereafter) implements the API,
providing functionality and extensibility such as configuration, aggregation,
processors and exporters.

OpenTelemetry requires a separation of the API from the
SDK, so that different SDKs can be
configured at runtime. Please refer to the overall OpenTelemetry
SDK concept for more information.

Programming Model

text
+------------------+
| MeterProvider    |                 +-----------------+             +--------------+
|   Meter A        | Measurements... |                 | Metrics...  |              |
|     Instrument X +-----------------> In-memory state +-------------> MetricReader |
|     Instrument Y |                 |                 |             |              |
|   Meter B        |                 +-----------------+             +--------------+
|     Instrument Z |
|     ...          |                 +-----------------+             +--------------+
|     ...          | Measurements... |                 | Metrics...  |              |
|     ...          +-----------------> In-memory state +-------------> MetricReader |
|     ...          |                 |                 |             |              |
|     ...          |                 +-----------------+             +--------------+
+------------------+

Specifications

References

  • Scenarios for Metrics API/SDK Prototyping (OTEP
    146    )

Overview (`specification/profiles/README.md`)

OpenTelemetry Profiles

Status: Alpha

Table of Contents

Overview

A profile is a collection of stack traces with associated values
representing resource consumption and code execution, collected from a
running program. Profiling is the process of collecting such profiles,
typically by sampling program state at regular intervals.

For a general introduction to OpenTelemetry profiling and how it complements other
observability signals, see Profiles concepts.

Design goals

The profiles signal is designed with the following goals in mind:

  • Low overhead: Enable profiling agents to operate continuously in production
    environments without materially impacting application performance.
  • Efficient representation: Reduces volume of data stored and transmitted,
    by using dictionary tables to deduplicate repeated information across samples.
  • Compatibility with existing formats: The data format is a superset of
    established profiling formats such as pprof
    and in most cases supports lossless conversions to and from these formats.
    If that's not possible (e.g. custom extensions), the original_payload_format
    field can be used to transmit the original information for future lossless
    export or reinterpretation.
  • Correlation with other signals: Profiles MUST be linkable to logs, metrics
    and traces through shared resource context and, where applicable,
    direct trace/span references.

Data Format

The OpenTelemetry profiles data format is here.
It builds on the pprof protobuf format
and extends it with:

  • Resource and scope context: Each batch of profiles is associated with a
    Resource and an InstrumentationScope,
    consistent with logs, metrics and traces.
  • Generalized dictionary: Deduplicates not only strings but also other messages
    that exhibit redundancy.
  • Generalized attributes: Most messages can carry attributes following the
    same conventions as other signals, augmented with Unit information (KeyValueAndUnit).
  • Span context references: Samples MAY include a Link (span ID and trace ID),
    enabling direct linking between a profile sample and the trace/span during
    which it was captured.

For details on the required attributes for Mapping messages and the custom
hashing scheme for build ID generation, see Mappings.

For more information on compatibility with pprof,
see pprof.

Known values

OpenTelemetry semantic conventions
are vital for profiles to correlate with other OpenTelemetry signals, enabling
unified analysis of traces, metrics, logs, and profiles for a holistic
system understanding.

To enhance the compatibility of OpenTelemetry Profiles with existing profiling
tools, known values are utilized.

Profile field Known values
original_payload_format pprof, jfr or linux_perf

Specifications

References

Overview (`specification/protocol/README.md`)

OpenTelemetry Protocol

The OpenTelemetry protocol (OTLP) design goals, requirements, and
specification have moved to
github.com/open-telemetry/opentelemetry-proto/docs.

You can also view the specification from the OpenTelemetry website, see OTLP.

For additional OTLP implementation requirements in the OpenTelemetry SDKs, see
SDK Exporter.

Overview (`specification/resource/README.md`)

Resource

Table of Contents

Overview

A Resource represents the observed entity for which telemetry is produced.
Within OpenTelemetry, all signals are associated with a Resource, enabling
contextual correlation of data from the same source. For example, if I see
a high latency in a span, I need to check the metrics for the same entity
during the time when the latency was observed.

[!NOTE]
The entity that technically emits telemetry and the entity described by
a Resource are not always the same. In auto-instrumentation scenarios, such
as eBPF-based agents that observe a running process without code changes,
the instrumentation agent produces the telemetry on behalf of the observed
workload. In these cases, the Resource describes the observed entity
(e.g., the service or process being instrumented), not the agent itself.
Attributes such as telemetry.sdk.name and telemetry.distro.name that
pertain to the instrumentation tooling should be applied according to their
own semantic conventions rather than used to identify the observed entity.

Resource provides two important aspects for observability:

  • It MUST identify the observed entity for which telemetry is being produced.
  • It SHOULD allow users to determine where that entity resides within their infrastructure.

Identity

Resource provides a natural way to understand "what" produced an effect and
evaluate other signals of that same source. This is done through attaching the
same set of identifying attributes on all telemetry produced in an
OpenTelemetry SDK.

Resource identity provides a natural pivot point for observability signals, a
key type of correlation in OpenTelemetry.

Implicit in the design of Resource and attributes is ensuring users are able to
navigate their infrastructure, tools, UIs, etc. to find the same entity that
telemetry is reporting against. For example, in practice we could see Resource
including more than one entity, like:

  • A process
  • A container
  • A Kubernetes pod name
  • A namespace
  • A deployment

By including identifying attributes of each of these, we can help users navigate
through their kubectl or Kubernetes UIs to find the specific process
generating telemetry. This is as important as being able to uniquely identify
one process from another.

[!TIP]
Observability signals SHOULD be actionable. Knowing a process is
struggling is not as useful as being able to scale up a deployment to take
load off the struggling process.

If the only thing important to Resource was identity, we could simply use UUIDs.
However, this would rely on some other, easily accessible, system to provide
human-friendly understanding for these UUIDs. OpenTelemetry provides a model
where a full UUID-only solution could be chosen, but defaults to a blended
approach, where resource provides both Identity and Navigation.

This leads to the next concept: Telescoping identity to the needs of a system.

Telescoping

Within OpenTelemetry, we want to give users the flexibility to decide what
information needs to be sent with observability signals and what information
can be later joined. We call this "telescoping identity" where users can decide
how small or large the size of an OpenTelemetry resource will be on the wire
(and correspondingly, how large data points are when stored, depending on
storage solution).

For example, in the extreme, OpenTelemery could synthesize a UUID for every
system which produces telemetry. All identifying attributes for Resource and
Entity could be sent via a side channel with known relationships to this UUID.
While this would optimize the runtime generation and sending of telemetry, it
comes at the cost of downstream storage systems needing to join data back
together either at ingestion time or query time. For high performance use cases,
e.g. alerting, these joins can be expensive.

In practice, users control Resource identity via the configuration of Resource
Detection within SDKs and the collector. Users wishing for minimal identity will
limit their resource detection just to a service.instance.id, for example.
Some users highly customize resource detection with many concepts being appended.

Specifications

Overview (`specification/schemas/README.md`)

Telemetry Schemas

Status: Stable

Table of Contents

Motivation

Telemetry sources such as instrumented applications and consumers of telemetry
such as observability backends sometimes make implicit assumptions about the
emitted telemetry. They assume that the telemetry will contain certain
attributes or otherwise have a certain shape and composition of data (this is
referred to as "telemetry schema" throughout this document).

This makes it difficult or impossible to change the composition of the emitted
telemetry data without breaking the consumers. For example changing the name of
an attribute of a span created by an instrumentation library can break the
backend if the backend expects to find that attribute by its name.

Semantic conventions are an important part of this problem. These conventions
define what names and values to use for span attributes, metric names and other
fields. If semantic conventions are changed the existing implementations
(telemetry source or consumers) need to be also changed correspondingly.
Furthermore, to make things worse, the implementations of telemetry sources and
implementations of telemetry consumers that work together and that depend on the
changed semantic convention need to be changed simultaneously, otherwise such
implementations will no longer work correctly together.

Essentially there is a coupling between 3 parties: 1) OpenTelemetry semantic
conventions, 2) telemetry sources and 3) telemetry consumers. The coupling
complicates the independent evolution of these 3 parties.

We recognize the following needs:

  • OpenTelemetry semantic conventions need to evolve over time. When conventions
    are first defined, mistakes are possible and we may want to fix the mistakes
    over time. We may also want to change conventions to re-group the attributes
    into different namespaces as our understanding of the attribute taxonomy
    improves.

  • Telemetry sources over time may want to change the schema of the telemetry
    they emit. This may be because for example the semantic conventions evolved
    and we want to make our telemetry match the newly introduced conventions.

  • In an observability system there may simultaneously exist telemetry sources
    that produce data that conforms to different telemetry schemas because
    different sources evolve at a different pace and are implemented and
    controlled by different entities.

  • Telemetry consumers have a need to understand what schema a particular piece
    of received telemetry confirms to. The consumers also need a way to be able to
    interpret the telemetry data that uses different telemetry schemas.

Telemetry Schemas that were proposed and accepted in
OTEP0152
address these needs.

How Schemas Work

We believe that the 3 parties described above are able to evolve
independently over time, while continuously retaining the ability to correctly
work together.

Telemetry Schemas are central to how we make this possible. Here is a summary of
how the schemas work:

  • OpenTelemetry defines a file format for defining
    telemetry schemas.

  • Telemetry schemas are versioned. Over time the schema may evolve and telemetry
    sources may emit data conforming to newer versions of the schema.

  • Telemetry schemas explicitly define transformations that are necessary to
    convert telemetry data between different versions of the schema, provided that
    such conversions are possible. When conversions are not possible it
    constitutes a breaking change between versions.

  • Telemetry schemas are identified by Schema URLs, that are unique for each
    schema version.

  • Telemetry sources (e.g. instrumentation libraries) will include a schema URL
    in the emitted telemetry.

  • Telemetry consumers will pay attention to the schema of the received
    telemetry. If necessary, telemetry consumers may transform the telemetry data
    from the received schema version to the target schema version as expected at
    the point of use (e.g. a dashboard may define which schema version it
    expects).

  • OpenTelemetry publishes a telemetry schema as part of the specification. The
    schema contains the list of transformations that semantic conventions undergo.
    The schema is to be available, to be referred and downloaded at a well known URL:
    https://opentelemetry.io/schemas/<version> (where <version> matches the
    specification version number).

  • OpenTelemetry instrumentation libraries include the OpenTelemetry Schema URL
    in all emitted telemetry. This is currently work-in-progress,
    here is an example
    of how it is done in Go SDK's Resource detectors.

  • OTLP allows inclusion of a schema URL in the emitted telemetry.

  • Third-party libraries, instrumentation or applications are advised to define
    and publish their own telemetry schema if it is completely different from
    OpenTelemetry schema (or use OpenTelemetry schema) and include the schema URL
    in the emitted telemetry.

What is Out of Scope

  • The concept of schema does not attempt to fully describe the shape of
    telemetry. The schema for example does not define all possible valid values
    for attributes or expected data types for metrics, etc. It is not a goal. Our
    goal is narrowly defined to solve the following problem only: to allow
    OpenTelemetry Semantic Conventions to evolve over time. For that reason this
    document is concerned with changes to the schema as opposed to the full
    state     of the schema. We do not preclude this though: the schema file format
    is extensible and in the future may allow defining the full state of the
    schema.

  • We intentionally limit the types of transformations of schemas to the bare
    minimum that is necessary to handle the most common changes that we believe
    OpenTelemetry Semantic Conventions will require in the near future. More types
    of transformations may be proposed in the future. This proposal does not
    attempt to support a comprehensive set of possible transformation types that
    can handle all possible changes to schemas that we can imagine. That would be
    too complicated and very likely superfluous. Any new transformation types
    MUST be proposed and added in the future to the schema file format when
    there is an evidence that they are necessary for the evolution of
    OpenTelemetry.

Use Cases

This section shows a couple interesting use-cases for the telemetry schemas
(other uses-cases are also possible, this is not an exhaustive list).

Full Schema-Aware

Here is an example on a schema-aware observability system:

Let's have a closer look at what happens with the Telemetry Source and Backend
pair as the telemetry data is emitted, delivered and stored:

In this example the telemetry source produces spans that comply with version
1.2.0 of OpenTelemetry schema, where the "deployment.environment" attribute is used
to record that the span is coming from production.

The telemetry consumer desires to store the telemetry in version 1.1.0 of
OpenTelemetry schema. The schema translator compares the schema_url in the
received span with the desired schema and sees that a version conversion is
needed. It then applies the change that is described in the schema file and
renames the attribute from "deployment.environment" to "environment" before
storing the span.

And here is for example how the schemas can be used to query stored data:

Collector-Assisted Schema Transformation

Here is a somewhat different use case, where the backend is not aware of schemas
and we rely on OpenTelemetry Collector to translate the telemetry to a schema
that the backend expects to receive. The "Schema Translate Processor" is
configured, the target schema_url is specified and all telemetry data that
passes through the Collector is converted to that target schema:

Schema URL

Schema URL is an identifier of a Schema. The URL specifies a location of a
Schema File that can be retrieved (so it is a URL and
not just a URI) using HTTP or HTTPS protocol.

Fetching the specified URL may return an HTTP redirect status code. The fetcher
MUST follow the HTTP standard and honour the redirect response and fetch the
file from the redirected URL.

The last part of the URL path is the version number of the schema.

text
http[s]://server[:port]/path/<version>

The part of the URL preceding the <version> is called Schema Family
identifier. All schemas in one Schema Family have identical Schema Family
identifiers.

To create a new version of the schema copy the schema file for the last version
in the schema family and add the definition of the new version. The schema file
that corresponds to the new version MUST be retrievable at a new URL.

Important: schema files are immutable once they are published. Once the schema
file is retrieved it is recommended to be cached permanently. Schema files may
be also packaged at build time with the software that anticipates it may need
the schema (e.g. the latest OpenTelemetry schema file can be packaged at build
time with OpenTelemetry Collector's schema translation processor).

Schema Version Number

Version number follows the MAJOR.MINOR.PATCH format, similar to SemVer 2.0.

Version numbers use the ordering rules
defined by SemVer 2.0 specification. See how ordering is used in the
Order of Transformations. Other than the ordering
rules the schema version numbers do not carry any other semantic meaning.

OpenTelemetry schema version numbers match OpenTelemetry
Semantic Conventions version
numbers, see more details here.

OTLP Support

To allow carrying the Schema URL in emitted telemetry OTLP includes a schema_url
field in the messages:

  • The schema_url field in the ResourceSpans, ResourceMetrics, ResourceLogs
    messages applies to the contained Resource, Span, SpanEvent, Metric, LogRecord
    messages.

  • The schema_url field in the InstrumentationLibrarySpans message applies to the
    contained Span and SpanEvent messages.

  • The schema_url field in the InstrumentationLibraryMetrics message applies to
    the contained Metric messages.

  • The schema_url field in the InstrumentationLibraryLogs message applies to the
    contained LogRecord messages.

  • If schema_url field is non-empty both in Resource* message and in the
    contained InstrumentationLibrary* message then the value in
    InstrumentationLibrary* message takes the precedence.

API Support

The OpenTelemetry API allows getting a
Tracer/Meter that
is associated with a Schema URL.

OpenTelemetry Schema

OpenTelemetry publishes it own schema at
opentelemetry.io/schemas/<_version_>.
The version number of the schema
is the same as the semantic-conventions version number which publishes the schema.
Every time a new semantic-conventions version is released a corresponding new schema
MUST be released simultaneously. If the semantic-conventions release did not introduce
any change the "changes" section of the corresponding version in the schema file
will be empty.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories โ†’

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#opentelemetry Browse opentelemetry catalog โ†’ โšก MCP Servers Model Context Protocol โ†’ ๐Ÿง  Agent Skills Prompt & rule files โ†’ ๐Ÿ”Œ OpenAPI Specs REST & Swagger schemas โ†’