micrometer Documentation

Modules/ROOT/Pages/Concepts/Counters

[[counters]]
= Counters

Counters report a single metric: a count. The Counter interface lets you increment by a fixed amount, which must be positive.

TIP: Never count something you can time with a Timer or summarize with a DistributionSummary! Both Timer and DistributionSummary always publish a count of events in addition to other measurements.

When building graphs and alerts off of counters, you should generally be most interested in measuring the rate at which some event occurs over a given time interval. Consider a simple queue. You could use counters to measure various things, such as the rate at which items are being inserted and removed.

It is tempting, at first, to conceive of visualizing absolute counts rather than a rate, but the absolute count is usually both a function of the rapidity with which something is used and the longevity of the application instance under instrumentation. Building dashboards and alerts of the rate of a counter per some interval of time disregards the longevity of the app, letting you see aberrant behavior long after the application has started.

NOTE: Be sure to read through the timer section before jumping into using counters, as timers record a count of timed events as part of the suite of metrics that go into timing. For those pieces of code you intend to time, you do NOT need to add a separate counter.

The following code simulates a real counter whose rate exhibits some perturbation over a short time window:

[source,java]

Normal rand = ...; // a random generator

MeterRegistry registry = ...
Counter counter = registry.counter("counter"); <1>

Flux.interval(Duration.ofMillis(10))
.doOnEach(d -> {
if (rand.nextDouble() + 0.1 > 0) { <2>
counter.increment(); <3>
}
})
.blockLast();

<1> Most counters can be created off of the registry itself with a name and, optionally, a set of tags.
<2> A slightly positively biased random walk.
<3> This is how you interact with a counter. You could also call counter.increment(n) to increment by more than one in a single operation.

A fluent builder for counters on the Counter interface itself provides access to less frequently used options, such as
base units and description. You can register the counter as the last step of its construction by calling register:

[source, java]

Counter counter = Counter
.builder("counter")
.baseUnit("beans") // optional
.description("a description of what this counter does") // optional
.tags("region", "test") // optional
.register(registry);

== The @Counted Annotation

The micrometer-core module contains a @Counted annotation that frameworks can use to add counting support to either specific types of methods such as those serving web request endpoints or, more generally, to all methods.

Also, an incubating AspectJ aspect is included in micrometer-core. You can use it in your application either through compile/load time AspectJ weaving or through framework facilities that interpret AspectJ aspects and proxy targeted methods in some other way, such as Spring AOP. Here is a sample Spring AOP configuration:

[source,java]

@Configuration
public class CountedConfiguration {
@Bean
public CountedAspect countedAspect(MeterRegistry registry) {
return new CountedAspect(registry);
}

}

Applying CountedAspect makes @Counted usable on any arbitrary method in an AspectJ proxied instance, as the following example shows:

[source,java]

@Service
public class ExampleService {

@Counted
public void sync() {
// @Counted will record the number of executions of this method
...
}

@Async
@Counted
public CompletableFuture<?> async() {
// @Counted will record the number of executions of this method
return CompletableFuture.supplyAsync(...);
}

}

=== @MeterTag on Method Parameters

To support using the @MeterTag annotation on method parameters, you need to configure the @CountedAspect to add the CountedMeterTagAnnotationHandler.

[source,java,subs=+attributes]

include::{include-java}/metrics/CountedAspectTest.java[tags=resolvers,indent=0]

include::{include-java}/metrics/CountedAspectTest.java[tags=meter_tag_annotation_handler,indent=0]

Let's assume that we have the following interface.

[source,java,subs=+attributes]

include::{include-java}/metrics/CountedAspectTest.java[tags=interface,indent=0]

When its implementations would be called with different arguments (remember that the implementation needs to be annotated with @Counted annotation too), the following counters would be created:

[source,java,subs=+attributes]

// Example for returning <toString()> on the parameter
include::{include-java}/metrics/CountedAspectTest.java[tags=example_value_to_string,indent=0]

// Example for calling the provided on the parameter
include::{include-java}/metrics/CountedAspectTest.java[tags=example_value_resolver,indent=0]

// Example for calling the provided
include::{include-java}/metrics/CountedAspectTest.java[tags=example_value_spel,indent=0]

// Example for using multiple @MeterTag annotations on the same parameter // @MeterTags({ @MeterTag(...), @MeterTag(...) }) can be also used include::{include-java}/metrics/CountedAspectTest.java[tags=example_multi_annotations,indent=0]

NOTE: CountedAspect doesn't support meta-annotations with @Counted.

== Function-tracking Counters

Micrometer also provides a more infrequently used counter pattern that tracks a monotonically increasing function (a function that stays the same or increases over time but never decreases). Some monitoring systems, such as Prometheus, push cumulative values for counters to the backend, but others publish the rate at which a counter is incrementing over the push interval. By employing this pattern, you let the Micrometer implementation for your monitoring system choose whether to rate-normalize the counter, and your counter remains portable across different types of monitoring systems.

[source, java]

Cache cache = ...; // suppose we have a Guava cache with stats recording on registry.more().counter("evictions", tags, cache, c -> c.stats().evictionCount()); <1>

<1> evictionCount() is a monotonically increasing function that increments with every cache eviction from the beginning of its life.

The function-tracking counter, in concert with the monitoring system's rate normalizing functionality (whether this is an artifact of the query language or the way data is pushed to the system), adds a layer of richness on top of the cumulative value of the function itself. You can reason about the rate at which the value is increasing, whether that rate is within an acceptable bound, is increasing or decreasing over time, and so on.

WARNING: Micrometer cannot guarantee the monotonicity of the function for you. By using this signature, you assert its monotonicity based on what you know about its definition.

A fluent builder for function counters on the FunctionCounter interface itself provides access to less frequently used options, such as base units and description. You can register the counter as the last step of its construction by calling register(MeterRegistry):

[source, java]

MyCounterState state = ...;

FunctionCounter counter = FunctionCounter
.builder("counter", state, state -> state.count())
.baseUnit("beans") // optional
.description("a description of what this counter does") // optional
.tags("region", "test") // optional
.register(registry);

If the function tracks a time value, you can specify the time unit of the value returned by the function when calling FunctionCounter.builder(...). This creates a FunctionCounter that automatically scales its time-based count to the registry's base time unit. This is analogous to how TimeGauge works, and provides a way to register monotonically increasing time values (e.g. RuntimeMXBean::getUptime) in a way that automatically scales to the preferred time unit of the metrics backend.

[source, java]

MyCounterState state = ...;

FunctionCounter.builder("counter.time", state, state -> state.totalTime(), TimeUnit.NANOSECONDS)
.description("a description of what this counter does") // optional
.tags("region", "test") // optional
.register(registry);

WARNING: Attempting to construct a function-tracking counter with a primitive number or one of its java.lang object forms is always incorrect. These numbers are immutable. Thus, the function-tracking counter cannot ever be changed. Attempting to "re-register" the function-tracking counter with a different number does not work, as the registry maintains only one meter for each unique combination of name and tags. "Re-registering" a function-tracking counter can happen indirectly for example as the result of a MeterFilter modifying the name and/or the tags of two different function-tracking counters so that they will be the same after the filter is applied.

Attempting to "re-register" a function-tracking counter will result in a warning like this: [source]

WARNING: This FunctionCounter has been already registered (MeterId{name='my.fc', tags=[]}), the registration will be ignored. Note that subsequent logs will be logged at debug level.

Modules/ROOT/Pages/Concepts/Distribution Summaries

[[distribution-summaries]]
= Distribution Summaries

A distribution summary tracks the distribution of events. It is structurally similar to a timer but records values that do not represent a unit of time. For example, you could use a distribution summary to measure the payload sizes of requests hitting a server.

The following example creates a distribution summary:

[source, java]

DistributionSummary summary = registry.summary("response.size");

The interface contains a fluent builder for distribution summaries:

[source, java]

DistributionSummary summary = DistributionSummary
.builder("response.size")
.description("a description of what this summary does") // optional
.baseUnit("bytes") // optional <1>
.tags("region", "test") // optional
.scale(100) // optional <2>
.register(registry);

<1> Add base units for maximum portability. Base units are part of the naming convention for some monitoring systems. Leaving it off and violating the naming convention has no adverse effect if you forget.
<2> Optionally, you can provide a scaling factor by which each recorded sample is multiplied as it is recorded.

NOTE: The maximum (which is named max) for basic DistributionSummary implementations, such as CumulativeDistributionSummary and StepDistributionSummary, is a time window maximum (TimeWindowMax).
It means that its value is the maximum value during a time window.
If no new values are recorded for the time window length, the maximum is reset to 0 as a new time window starts.
The time window size until values are fully expired is the expiry multiplied by the bufferLength in DistributionStatisticConfig.
expiry defaults to the step size of the meter registry unless it's explicitly set to a different value, and bufferLength defaults to 3.
A time window max is used to capture the maximum latency in a subsequent interval after heavy resource pressure triggers the latency and prevents metrics from being published.
Percentiles are also time window percentiles (TimeWindowPercentileHistogram).

== Scaling and Histograms

Micrometer's preselected percentile histogram buckets are all integers from 1 to Long.MAX_VALUE. Currently, minimumExpectedValue and maximumExpectedValue serve to control the cardinality of the bucket set. If we try to detect that your min/max yields a small range and scale the preselected bucket domain to your summary's range, we do not have another lever to control bucket cardinality.

Instead, if your summary's domain is more constrained, scale your summary's range by a fixed factor. The use case we have heard so far is for summaries of ratios whose domain is [0,1]. Given that scenario, we can use the following code to create values from 0 to 100:

[source,java]

DistributionSummary.builder("my.ratio").scale(100).register(registry)

This way, the ratio winds up in the range [0,100] and we can set maximumExpectedValue to 100. You can pair this with custom SLO boundaries if you care about particular ratios:

[source,java]

DistributionSummary.builder("my.ratio") .scale(100) .serviceLevelObjectives(70, 80, 90) .register(registry)

== Memory Footprint Estimation

The total memory footprint of a distribution summary can vary dramatically, depending on which options you choose. The following table of memory consumption is based on the use of various features. These figures assume no tags and a ring buffer length of 3. Adding tags adds somewhat to the total, as does increasing the buffer length. Total storage can also vary somewhat depending on the registry implementation.

R = Ring buffer length. We assume the default of 3 in all examples. R is set with DistributionSummary.Builder#distributionStatisticBufferLength.
B = Total histogram buckets. It can be SLO boundaries or percentile histogram buckets. By default, summaries have NO minimum and maximum expected value, so we ship all 276 predetermined histogram buckets. You should always clamp distribution summaries with a minimumExpectedValue and maximumExpectedValue when you intend to ship percentile histograms.
M = Time-decaying max. 104 bytes.
Fb = Fixed boundary histogram. 8b * B * R.
Pp = Percentile precision. By default, it is 1. It is generally in the range of [0, 3]. Pp is set with DistributionSummary.Builder#percentilePrecision.
Hdr(Pp) = High dynamic range histogram.
- When Pp = 0: 1.9kb * R + 0.8kb
- When Pp = 1: 3.8kb * R + 1.1kb
- When Pp = 2: 18.2kb * R + 4.7kb
- When Pp = 3: 66kb * R + 33kb

|No |No |M| ~0.1kb
|No |Yes |M + Fb|For percentile histogram clamped to 66 buckets, ~6kb
|Yes |Yes |M + Hdr(Pp)|For the addition of a 0.95 percentile with defaults otherwise, ~12.6kb
|=========================================================

NOTE: For Prometheus, R is always equal to 1, regardless of how you attempt to configure it through DistributionSummary.Builder. This special case exists for Prometheus because it expects cumulative histogram data that never rolls over.

Modules/ROOT/Pages/Concepts/Gauges

[[gauges]]
= Gauges

A gauge is a handle to get the current value. Typical examples for gauges would be the size of a collection or map or the number of threads in a running state.

TIP: Gauges are useful for monitoring things with natural upper bounds. We do not recommend using a gauge to monitor things like request count, as they can grow without bound for the duration of an application instance's life.

TIP: Never gauge something you can count with a Counter!

Micrometer takes the stance that gauges should be sampled and not be set, so there is no information about what might have occurred between samples. Any intermediate values set on a gauge are lost by the time the gauge value is reported to a metrics backend, so there is little value in setting those intermediate values in the first place.

Think of a Gauge as a "heisen-gauge": a meter that changes only when it is observed. Every other meter type accumulates intermediate counts toward the point where the data is sent to the metrics backend.

The MeterRegistry interface contains methods for building gauges to observe numeric values, functions, collections, and maps:

[source, java]

List list = registry.gauge("listGauge", Collections.emptyList(), new ArrayList<>(), List::size); <1> List list2 = registry.gaugeCollectionSize("listSize2", Tags.empty(), new ArrayList<>()); <2> Map map = registry.gaugeMapSize("mapGauge", Tags.empty(), new HashMap<>());

<1> A slightly more common form of gauge is one that monitors some non-numeric object. The last argument establishes the function that is used to determine the value of the gauge when the gauge is observed.
<2> A more convenient form of (1) for when you want to monitor collection size.

All of the different forms of creating a gauge maintain only a weak reference to the object being observed, so as not to prevent garbage collection of the object.

== Manually Incrementing or Decrementing a Gauge

A gauge can be made to track any java.lang.Number subtype that is settable, such as AtomicInteger and AtomicLong found in java.util.concurrent.atomic and similar types, such as Guava's AtomicDouble:

[source,java]

// maintain a reference to myGauge
AtomicInteger myGauge = registry.gauge("numberGauge", new AtomicInteger(0));

// ... elsewhere you can update the value it holds using the object reference myGauge.set(27); myGauge.set(11);

Note that, in this form, unlike other meter types, you do not get a reference to the Gauge when creating one. Rather, you get a reference to the thing being observed. This is because of the "heisen-gauge" principal: The gauge is self-sufficient once created, so you should never need to interact with it. This lets us give you back only the instrumented object, which allows for quick one-liners that both create the object to be observed and set up metrics around it.

This pattern should be less common than the DoubleFunction form. Remember that frequent setting of the observed Number results in a lot of intermediate values that never get published. Only the instantaneous value of the gauge at publish time is ever sent to the monitoring system.

WARNING: Attempting to construct a gauge with a primitive number or one of its java.lang object forms is always incorrect. These numbers are immutable. Thus, the gauge cannot ever be changed. Attempting to "re-register" the gauge with a different number does not work, as the registry maintains only one meter for each unique combination of name and tags. "Re-registering" a gauge can happen indirectly for example as the result of a MeterFilter modifying the name and/or the tags of two different gauges so that they will be the same after the filter is applied.

Attempting to "re-register" a gauge will result in a warning like this: [source]

WARNING: This Gauge has been already registered (MeterId{name='my.gauge', tags=[]}), the registration will be ignored. Note that subsequent logs will be logged at debug level.

== Gauge Fluent Builder

The interface contains a fluent builder for gauges:

[source, java]

Gauge gauge = Gauge
.builder("gauge", myObj, myObj::gaugeValue)
.description("a description of what this gauge does") // optional
.tags("region", "test") // optional
.register(registry);

Generally the returned Gauge instance is not useful except in testing, as the gauge is already set up to track a value automatically upon registration.

== Why is My Gauge Reporting NaN or Disappearing?

It is your responsibility to hold a strong reference to the state object that you are measuring with a Gauge. Micrometer is careful to not create strong references to objects that would otherwise be garbage collected. Once the object being gauged is de-referenced and is garbage collected, Micrometer starts reporting a NaN or nothing for a gauge, depending on the registry implementation.

If you see your gauge reporting for a few minutes and then disappearing or reporting NaN, it almost certainly suggests that the underlying object being gauged has been garbage collected.

== TimeGauge

TimeGauge is a specialized gauge that tracks a time value, to be scaled to the base unit of time expected by each registry implementation.

TimeGauge can be registered with TimeUnit as follows:

[source, java]

AtomicInteger msTimeGauge = new AtomicInteger(4000); AtomicInteger usTimeGauge = new AtomicInteger(4000); TimeGauge.builder("my.gauge", msTimeGauge, TimeUnit.MILLISECONDS, AtomicInteger::get).register(registry); TimeGauge.builder("my.other.gauge", usTimeGauge, TimeUnit.MICROSECONDS, AtomicInteger::get).register(registry);

And for example, if the registry is Prometheus, they will be converted in seconds as follows:

text

# HELP my_gauge_seconds
# TYPE my_gauge_seconds gauge
my_gauge_seconds 4.0
# HELP my_other_gauge_seconds
# TYPE my_other_gauge_seconds gauge
my_other_gauge_seconds 0.004

== Multi-gauge

Micrometer supports one last special type of Gauge, called a MultiGauge, to help manage gauging a growing or shrinking list of criteria.
This feature lets you select a set of well-bounded but slightly changing set of criteria from something like an SQL query and report each Row (see the example below) as a Gauge. The following example creates a MultiGauge:

[source, java]

// SELECT count(*) from job group by status WHERE job = 'dirty'
MultiGauge statuses = MultiGauge.builder("statuses")
.tag("job", "dirty")
.description("The number of widgets in various statuses")
.baseUnit("widgets")
.register(registry);

...

// run this whenever you re-run your query
statuses.register(
resultSet.stream()
.map(result -> Row.of(Tags.of("status", result.getAsString("status")), result.getAsInt("count")))
.collect(toList()),
true // whether to overwrite the previous value or only record it once

);

The overwrite argument controls how rows with the same set of tags are handled on subsequent registrations.
Use overwrite=true when each registration represents a new snapshot and previously registered rows should be replaced with the latest values.
Use overwrite=false when existing rows should remain unchanged on later registrations with the same tags.
Rows that are no longer present in the new registration are removed.

Modules/ROOT/Pages/Concepts/High Cardinality Tags Detector

[[high-cardinality-tags-detector]]
= High Cardinality Tags Detector

High cardinality tags can cause memory and performance issues in your application and your metrics backend. When tag values are unbounded (for example userID, requestID, traceID), each unique combination creates a new Meter, potentially leading to millions of time series and excessive memory consumption.

HighCardinalityTagsDetector helps you to identify Meters that may have high cardinality tags by monitoring the number of Meters that share the same name. When the count exceeds a configurable threshold, it indicates that the Meter likely has high cardinality tags.

NOTE: HighCardinalityTagsDetector detects potential high cardinality tags by counting Meters with the same name. It does not detect other issues, such as appending random values to Meter names. Its sole purpose is detecting the potential presence of high cardinality tags.

== Understand High Cardinality

High cardinality occurs when a tag has an unbounded or large number of possible values that does not fit in memory. Common examples include:

UserID, Email, RequestID, SessionID, TraceID
Timestamps
Full URLs (/users/123)
Any user input that is not validated/normalized

In contrast, low cardinality tags have a bounded, typically "small" set of values:

HTTP methods (method=GET)
HTTP status codes (status=200)
Application names (application=payments-app)
Environment names (env=prod)
Templated URLs (/users/\{id})

TIP: For more information on tag naming best practices, see xref:./naming.adoc#_tag_naming[Tag Naming].

== How It Works

The detector works by:

Counting how many meters exist with the same name in the registry
Comparing this count against a threshold
Notifying you (via logging or custom consumer) when the threshold is exceeded

== Usage

You can configure the detector through the MeterRegistry. In this case, the detector is automatically started and managed by the registry:

=== Configure with the defaults

[source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_default,indent=0]

=== Configure with a Factory

You can provide your own Function<MeterRegistry, HighCardinalityTagsDetector> that receives a registry and returns a detector:

[source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_factory,indent=0]

=== Configure with a Builder

HighCardinalityTagsDetector.Builder can be used to conveniently set parameters:

[source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_builder,indent=0]

=== One-Time Check

You can perform a one-time check to see if any meters exceed the threshold:

[source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=one_time_check,indent=0]

This approach can be useful for:

Ad-hoc investigation of potential issues
Tests to verify your instrumentation

=== Scheduled Monitoring

For continuous monitoring in production, you don't need to create a scheduled job that periodically calls the detector. The detector has a built in scheduler, but you don't need to manage that one either (by calling start() and close()). Instead you can configure the detector through the MeterRegistry so that the detector is automatically started and managed by the registry for you. See the various configuration options with the registry above.

=== Custom Consumer

Instead of the default logging behavior, you can provide a Consumer<HighCardinalityMeterInfo> to handle high cardinality notifications:

[source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=custom_consumer_config,indent=0]

And here's an example implementation: [source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=custom_consumer,indent=0]

This can be useful for:

Custom logging or reporting
Publishing alerts
Collecting metrics about high cardinality detections
Triggering any kind of custom actions

== Preventing High Cardinality

When the detector identifies high cardinality tags, consider the following possible solutions:

=== Remove Problematic Tags

If a tag provides little value but high cardinality, you should remove it. If you control the instrumentation, you should update it to not add the high cardinality tag. If you don't control the instrumentation, you can remove the tag using a MeterFilter:

[source,java]

registry.config().meterFilter(MeterFilter.ignoreTags("userId"));

See xref:./meter-filters.adoc[Meter Filters] for more information and configuration options.

=== Normalize Tag Values

Instead of using "raw" values, normalize them to reduce cardinality:

Use templated URLs instead of actual URLs: /users/\{id} instead of /users/123
Group values into ranges: age.group=20-30 instead of age=25
Use categories: outcome=CLIENT_ERROR (or status=4xx) instead of status=404
Limit unknown/unexpected values to known values: status=OTHER

NOTE: age and status are not necessarily high cardinality data since they are usually fall into a finite set of values (except if input is not validated and age=12345 and status=54321 are possible to send to the app). The normalization techniques to reduce cardinality are still valid though.

=== Use High Cardinality data with the Observation API

If you are using Micrometer's Observation API, you can mark certain metadata as high cardinality. These key-values should not be used for recording metrics. Typically they are only used in outputs that can handle high cardinality (for example: distributed tracing systems, logs):

[source,java]

observation.highCardinalityKeyValue("userId", userId);

See xref:../observation/introduction.adoc[Observation API] for more information.

=== Use MeterFilters to Set Bounds

If none of the above works, you can use maximumAllowableTags and maximumAllowableMetrics as a last resort to protect your application and your monitoring system. They are for situations in which you cannot fix the cardinality problem, for example with an HTTP client instrumentation that does not offer a way to tag the URI with a template (low cardinality) instead of the expanded URI (high cardinality). You can use MeterFilter to prevent high cardinality before it becomes a problem:

[source,java]

registry.config().meterFilter(MeterFilter.maximumAllowableTags("http.requests", "uri", 100, MeterFilter.deny()));

TIP: In the code above, instead of MeterFilter.deny(), you can implement your own logic that not only denies the registration of the Meter but also logs the event out. These logs can be overwhelming, it's a good idea to limit them.

See xref:./meter-filters.adoc[Meter Filters] for more information.

== Best Practices

Enable in Production: Run the detector in production to catch issues that may not appear in testing.
Monitor Regularly: Use the scheduled monitoring mode rather than one-time checks.
Act on Warnings: When high cardinality is detected, investigate and fix the root cause rather than just increasing the threshold.
Set Appropriate Thresholds: The default threshold might work for most applications, but adjust it based on your monitoring system's capabilities and your application's scale as needed.
Test Your Instrumentation: Use the detector in integration tests to verify that your custom instrumentation doesn't introduce high cardinality.

== Limitations

The detector only identifies high cardinality by counting Meters with the same name. It cannot detect:
** Random values appended to Meter names
** Memory leaks from other sources
** High cardinality that hasn't yet exceeded the threshold
The detector examines the first meter that exceeds the threshold. If multiple meters have high cardinality, you'll need to fix them iteratively.
The threshold is a heuristic based on memory and may need adjustment for your specific use case.

== See Also

xref:./meter-filters.adoc[Meter Filters]: Using filters to control meter registration and tag cardinality
xref:../observation/introduction.adoc[Observation API]: Using high- and low cardinality tags with Observations

Modules/ROOT/Pages/Concepts/Histogram Quantiles

[[histograms]]
= Histograms and Percentiles

Timers and distribution summaries support collecting data to observe their percentile distributions. There are two main approaches to viewing percentiles:

Percentile histograms: Micrometer accumulates values to an underlying histogram and ships a predetermined set of buckets to the monitoring system. The monitoring system's query language is responsible for calculating percentiles off of this histogram. Currently, only Prometheus, Atlas, and Wavefront support histogram-based percentile approximations, through histogram_quantile, :percentile, and hs(), respectively. If you target Prometheus, Atlas, or Wavefront, prefer this approach, since you can aggregate the histograms across dimensions (by summing the values of the buckets across a set of dimensions) and derive an aggregable percentile from the histogram.
Client-side percentiles: Micrometer computes a percentile approximation for each meter ID (set of name and tags) and ships the percentile value to the monitoring system. This is not as flexible as a percentile histogram because it is not possible to aggregate percentile approximations across tags. Nevertheless, it provides some level of insight into percentile distributions for monitoring systems that do not support server-side percentile calculation based on a histogram.

The following example builds a timer with a histogram:

[source,java]

Timer.builder("my.timer") .publishPercentiles(0.5, 0.95) // median and 95th percentile <1> .publishPercentileHistogram() // <2> .serviceLevelObjectives(Duration.ofMillis(100)) // <3> .minimumExpectedValue(Duration.ofMillis(1)) // <4> .maximumExpectedValue(Duration.ofSeconds(10))

<1> publishPercentiles: Used to publish percentile values computed in your application. These values are non-aggregable across dimensions.
<2> publishPercentileHistogram: Used to publish a histogram suitable for computing aggregable (across dimensions) percentile approximations in Prometheus (by using histogram_quantile), Atlas (by using :percentile), and Wavefront (by using hs()). For Prometheus and Atlas, the buckets in the resulting histogram are preset by Micrometer based on a generator that has been determined empirically by Netflix to yield a reasonable error bound on most real world timers and distribution summaries. By default, the generator yields 276 buckets, but Micrometer includes only those that are within the range set by minimumExpectedValue and maximumExpectedValue, inclusive. Micrometer clamps timers by default to a range of 1 millisecond to 1 minute, yielding 73 histogram buckets per timer dimension. publishPercentileHistogram has no effect on systems that do not support aggregable percentile approximations. No histogram is shipped for these systems.
<3> serviceLevelObjectives: Used to publish a cumulative histogram with buckets defined by your SLOs. When used in concert with publishPercentileHistogram on a monitoring system that supports aggregable percentiles, this setting adds additional buckets to the published histogram. When used on a system that does not support aggregable percentiles, this setting causes a histogram to be published with only these buckets.
<4> minimumExpectedValue/maximumExpectedValue: Controls the number of buckets shipped by publishPercentileHistogram and controls the accuracy and memory footprint of the underlying HdrHistogram structure.

NOTE: For those monitoring systems, where percentiles can be approximated using the histogram (see publishPercentileHistogram above), it is usually unnecessary to also publish client-side percentiles (publishPercentiles) since in those scenarios client-side percentiles are redundant and also non-aggregable across dimensions (unlike histograms). Prometheus Java Client (1.x) does not support having both under the same metric name.

Since shipping percentiles to the monitoring system generates additional time series, it is generally preferable to not configure them in core libraries that are included as dependencies in applications. Instead, applications can turn on this behavior for some set of timers and distribution summaries by using a meter filter.

For example, suppose we have a handful of timers in a common library. We have prefixed these timer names with myservice:

[source,java]

registry.timer("myservice.http.requests").record(..); registry.timer("myservice.db.requests").record(..);

We can turn on client-side percentiles for both timers by using a meter filter:

[source,java]

registry.config().meterFilter(
new MeterFilter() {
@Override
public DistributionStatisticConfig configure(Meter.Id id, DistributionStatisticConfig config) {
if(id.getName().startsWith("myservice")) {
return DistributionStatisticConfig.builder()
.percentiles(0.95)
.build()
.merge(config);
}
return config;
}
});

Modules/ROOT/Pages/Concepts/Implementations

[[monitoring-systems]]
= Supported Monitoring Systems

Micrometer contains a core module with an instrumentation https://en.wikipedia.org/wiki/Service_provider_interface[SPI], a set of modules containing implementations for various monitoring systems (each is called a registry), and a test kit. You need to understand three important characteristics of monitoring systems:

Dimensionality. Whether the system supports metric names to be enriched with tag key/value pairs. If a system is not dimensional, it is hierarchical, which means it supports only a flat metric name. When publishing metrics to hierarchical systems, Micrometer flattens the set of tag key/value pairs and adds them to the name.

[cols=2*,options="header"]
|===
|Dimensional
|Hierarchical

|AppOptics, Atlas, Azure Monitor, Cloudwatch, Datadog, Datadog StatsD, Dynatrace, Elastic, Humio, Influx, KairosDB, New Relic, Prometheus, SignalFx, Sysdig StatsD, Telegraf StatsD, Wavefront
|Graphite, Ganglia, JMX, Etsy StatsD
|===

xref:./rate-aggregation.adoc[Rate Aggregation]. In this context, we mean aggregation of a set of samples over a prescribed time interval. Some monitoring systems expect some types of discrete samples (such as counts) to be converted to a rate by the application prior to being published. Other systems expect cumulative values to always be sent. Still others have no opinion on it either way.

[cols=2*,options="header"]
|===
|Client-side
|Server-side

|AppOptics, Atlas, Azure Monitor, Datadog, Dynatrace, Elastic, Graphite, Ganglia, Humio, Influx, JMX, Kairos, New Relic, all StatsD flavors, SignalFx
|Prometheus, Wavefront footnote:[As of 1.2.0, Micrometer sends cumulative values to Wavefront.]
|===

Publishing. Some systems expect to poll applications for metrics at their leisure, while others expect metrics to be pushed to them on a regular interval.

[cols=2*,options="header"]
|===
|Client pushes
|Server polls

|AppOptics, Atlas, Azure Monitor, Datadog, Dynatrace, Elastic, Graphite, Ganglia, Humio, Influx, JMX, Kairos, New Relic, SignalFx, Wavefront
|Prometheus, all StatsD flavors
|===

There are other, more minor, variations in expectations from one monitoring system to another, such as their conception of base units of measurement (particularly time) and the canonical naming convention for metrics. Micrometer customizes your metrics to meet these demands on a per-registry basis.

Modules/ROOT/Pages/Concepts/Long Task Timers

[[long-task-timers]]
= Long Task Timers

The long task timer is a special type of timer that lets you measure time while an event being measured is still running. A normal Timer only records the duration after the task is complete.

Long task timers publish at least the following statistics:

Active task count
Total duration of active tasks
The maximum duration of active tasks

Unlike a regular Timer, a long task timer does not publish statistics about completed tasks.

Consider a background process to refresh metadata from a data store. For example, https://github.com/Netflix/edda[Edda] caches AWS resources, such as instances, volumes, auto-scaling groups, and others. Normally all data can be refreshed in a few minutes. If the AWS services have problems, it can take much longer. A long task timer can be used to track the active time for refreshing the metadata.

For example, in a Spring application, it is common for such long running processes to be implemented with @Scheduled. Micrometer provides a special @Timed annotation for instrumenting these processes with a long task timer:

[source, java]

@Timed(value = "aws.scrape", longTask = true)
@Scheduled(fixedDelay = 360000)
void scrapeResources() {
// find instances, volumes, auto-scaling groups, etc...

}

It is up to the application framework to make something happen with @Timed. If your framework of choice does not support it, you can still use the long task timer:

[source, java]

LongTaskTimer scrapeTimer = registry.more().longTaskTimer("scrape");
void scrapeResources() {
scrapeTimer.record(() => {
// find instances, volumes, auto-scaling groups, etc...
});

}

If we wanted to alert when this process exceeds a threshold, with a long task timer, we receive that alert at the first reporting interval after we have exceeded the threshold. With a regular timer, we would not receive the alert until the first reporting interval after the process completed, over an hour later!

The interface contains a fluent builder for long task timers:

[source, java]

LongTaskTimer longTaskTimer = LongTaskTimer
.builder("long.task.timer")
.description("a description of what this timer does") // optional
.tags("region", "test") // optional
.register(registry);

The exact series exposed by a long task timer depend on the registry. For example, the Prometheus registry exposes a different set of series when publishPercentileHistogram() is enabled; see the xref:../implementations/prometheus.adoc#_long_task_timers[Long task timers section of the Prometheus implementation docs] for details.

Modules/ROOT/Pages/Concepts/Meter Convention

[[meter-convention]]
= Meter Convention

While MeterFilter can be used to customize the name and tags (collectively, the convention) of Meters in any instrumentation, it can be more convenient and robust to customize the convention for a common instrumentation more directly.
This is the purpose of the MeterConvention.

For examples of its usage in instrumentation provided by Micrometer as well as examples of different implementations of a convention for an instrumentation, see the xref:../reference/jvm.adoc#meter-conventions[JVM metrics].

Modules/ROOT/Pages/Concepts/Meter Filters

[[meter-filters]]
= Meter Filters

You can configure each registry with meter filters, which give you greater control over how and when meters are registered and what kinds of statistics they emit. Meter filters serve three basic functions:

Deny (or Accept) meters being registered.
Transform meter IDs (for example, changing the name, adding or removing tags, and changing the description or base units).
Configure distribution statistics for some meter types.

Implementations of MeterFilter are added to the registry programmatically:

==== [source, java]

registry.config()
.meterFilter(MeterFilter.ignoreTags("too.much.information"))
.meterFilter(MeterFilter.denyNameStartsWith("jvm"));

====

Meter filters are applied in order, and the results of transforming or configuring a meter are chained.

== Deny or Accept Meters

The verbose form of an accept or deny filter is:

==== [source, java]

new MeterFilter() {
@Override
public MeterFilterReply accept(Meter.Id id) {
if(id.getName().contains("test")) {
return MeterFilterReply.DENY;
}
return MeterFilterReply.NEUTRAL;
}

}

====

MeterFilterReply has three possible states:

DENY: Do not let this meter be registered. When you try to register a meter against a registry and the filter returns DENY, the registry returns a NOOP version of that meter (for example, NoopCounter or NoopTimer). Your code can continue to interact with the NOOP meter, but anything recorded to it is discarded immediately with minimal overhead.
NEUTRAL: If no other meter filter has returned DENY, registration of meters proceeds normally.
ACCEPT: If a filter returns ACCEPT, the meter is immediately registered without interrogating the accept methods of any further filters.

=== Convenience Methods

MeterFilter provides several convenience static builders for deny and accept type filters:

accept(): Accept every meter, overriding the decisions of any filters that follow.
accept(Predicate<Meter.Id>): Accept any meter matching the predicate.
acceptNameStartsWith(String): Accept every meter with a matching prefix.
deny(): Deny every meter, overriding the decisions of any filters that follow.
denyNameStartsWith(String): Deny every meter with a matching prefix. All MeterBinder implementations provided by Micrometer have names with common prefixes to allow for easy grouping visualization in UIs but also to make them easy to disable or enable as a group with a prefix. For example, you can deny all JVM metrics with MeterFilter.denyNameStartsWith("jvm").
deny(Predicate<Meter.Id>): Deny any meter that matches the predicate.
maximumAllowableMetrics(int): Deny any meter after the registry has reached a certain number of meters.
maximumAllowableTags(String meterNamePrefix, String tagKey, int maximumTagValues, MeterFilter onMaxReached): Places an upper bound on the number of tags produced by the matching series.

Whitelisting only a certain group of metrics is a particularly common case for monitoring systems that are expensive. This can be achieved with a static call:

denyUnless(Predicate<Meter.Id>): Deny all meters that do not match the predicate.

=== Chaining Deny Accept Meters

Meter filters are applied in the order in which they are configured on the registry, so it is possible to stack deny and accept filters to achieve more complex rules:

==== [source, java]

registry.config()
.meterFilter(MeterFilter.acceptNameStartsWith("http"))
.meterFilter(MeterFilter.deny());

====

This achieves another form of whitelisting by stacking two filters together. Only http metrics are allowed to exist in this registry.

== Transforming Metrics

The following example shows a transform filter:

==== [source, java]

new MeterFilter() {
@Override
public Meter.Id map(Meter.Id id) {
if(id.getName().startsWith("test")) {
return id.withName("extra." + id.getName()).withTag("extra.tag", "value");
}
return id;
}

}

====

This filter adds a name prefix and an additional tag conditionally to meters starting with a name of test.

[IMPORTANT] .`MeterFilter` implementations of `map` should be "static"

The `id` parameter is the only dynamic input that changes over the lifecycle of the `MeterFilter` on which `MeterFilter` implementations should depend. Depending on a value that will be fixed at runtime such as an instance ID or hostname is fine because it does not change after application start-up. Use cases where dynamic behavior is desired, such as defining tags based on the context of a request etc., should be implemented in the instrumentation itself rather than in a `MeterFilter`. For example, see `MongoMetricsCommandListener` and the `MongoCommandTagsProvider` it takes in a constructor argument as well as the default implementation `DefaultMongoCommandTagsProvider`. See also xref:../observation/components.adoc#micrometer-observation-predicates-filters[ObservationFilter] which allows dynamic implementations.

=== Convenience Methods

MeterFilter provides convenience builders for many common transformation cases:

commonTags(Iterable<Tag>): Adds a set of tags to all metrics. Adding common tags for application name, host, region, and others is a highly recommended practice.
ignoreTags(String...): Drops matching tag keys from every meter. This is particularly useful when a tag provably comes to have
too high cardinality and starts stressing your monitoring system or costing too much but you cannot change all the instrumentation points quickly.
replaceTagValues(String tagKey, Function<String, String> replacement, String... exceptions): Replace tag values according to the provided mapping for all matching tag keys. You can use this to reduce the total cardinality of a tag by mapping some portion of tag values to something else.
renameTag(String meterNamePrefix, String fromTagKey, String toTagKey): Rename a tag key for every metric that begins with a given prefix.

== Configuring Distribution Statistics

Timer and DistributionSummary contain a set of optional distribution statistics (in addition to the basics of count, total, and max) that you can configure through filters. These distribution statistics include pre-computed percentiles, SLOs, and histograms.

==== [source, java]

new MeterFilter() {
@Override
public DistributionStatisticConfig configure(Meter.Id id, DistributionStatisticConfig config) {
if (id.getName().startsWith("prefix")) {
return DistributionStatisticConfig.builder()
.percentiles(0.9, 0.95)
.build()
.merge(config);
}
return config;
}

};

====

Generally, you should create a new DistributionStatisticConfig with only the pieces you wish to configure and then merge it with the input configuration. This lets you drop down on registry-provided defaults for distribution statistics and to chain multiple filters together, each configuring some part of the distribution statistics (for example, you might want a 100ms SLO for all HTTP requests but only percentile histograms on a few critical endpoints).

MeterFilter provides convenience builders for:

maxExpected(Duration/long): Governs the upper bound of percentile histogram buckets shipped from a timer or summary.
minExpected(Duration/long): Governs the lower bound of percentile histogram buckets shipped from a timer or summary.

Spring Boot offers property-based filters for configuring SLOs, percentiles, and percentile histograms by name prefix.

== Apply MeterFilters conditionally

It is a common use-case to apply MeterFilters only for a selection of Meters (for example, for ones that have a certain name or start with a given prefix).
In order to do this, Micrometer provides a convenience method forMeters(Predicate<Meter.Id> predicate, MeterFilter delegate) that enables the provided (delegate) filter for the Meters selected by the predicate.

See the following example:

==== [source, java]

registry.config()
.meterFilter(MeterFilter.forMeters(startsWith("prefix"), MeterFilter.ignoreTags("extra")));

Predicate<Meter.Id> startsWith(String prefix) {
return id -> id.getName().startsWith(prefix);

}

====

Modules/ROOT/Pages/Concepts/Meter Provider

[[meter-provider]]
= Meter Provider

It's a common use-case to attach tags dynamically to a Meter. Let's say we execute a job and we want to use a Timer to instrument it:

[source, java]

Timer.Sample sample = Timer.start(registry);

Result result = job.execute();

Timer timer = Timer.builder("job.execution")
.tag("job.name", "job")
.tag("status", result.status())
.register(registry);

sample.stop(timer);

This lets us dynamically determine the status tag from the end state of the operation we are timing. There are two drawbacks of doing this:

Every time the above is executed, a new Timer.Builder instance is created. This increases the amount of data that the GC needs to collect.
The code above is somewhat boilerplate, it does not let you define the common properties of a Timer and attach what is dynamically changing but everything is always present.

NOTE: In some cases you can use registry.timer("job.execution", "job.name", "my-job", "status", result.status()) instead of using Timer.Builder which can save you some extra objects but this is not always possible.

You can resolve both of these issues by using a MeterProvider. It's a convenience interface to create new meters from tags using a common "template".

NOTE: Not every Meter can do this, MeterProvider can be used with Counter, Timer, LongTaskTimer, and DistributionSummary.

Here's what you can do instead of the above:

[source, java]

private MeterProvider timerProvider = Timer.builder("job.execution")
.tag("job.name", "my-job")
.withRegistry(registry); <1>

// ...

Timer.Sample sample = Timer.start(registry);

Result result = job.execute();

sample.stop(timerProvider.withTags("status", result.status())); <2>

<1> Definition of the MeterProvider for Timer with all the "static" fields necessary. Please note the withRegistry method call.
<2> Definition of the dynamic tags. Note that only those tags are defined here that are dynamic and everything else is defined where the MeterProvider is created. The withTags method returns a Timer that is created using the tags defined in withTags plus everything else that is defined by the MeterProvider.

This and the previous example produce the same output, the only difference is the amount of boilerplate in your code and the amount of builder objects created in the heap.

Modules/ROOT/Pages/Concepts/Meters

[[meters]]
= Meters

A Meter is the interface for collecting a set of measurements (which we individually call metrics) about your application.

Micrometer supports a set of Meter primitives, including Timer, Counter, Gauge, DistributionSummary, LongTaskTimer, FunctionCounter, FunctionTimer, and TimeGauge. Different meter types result in a different number of time series metrics. For example, while there is a single metric that represents a Gauge, a Timer measures both the count of timed events and the total time of all timed events.

TIP: Recording a measurement for a Meter is expected to be a relatively cheap operation and should not throw any exception.
If the xref:./registry.adoc[registry] supports publishing metrics to a monitoring system, this is done in a separate thread and should not affect recording metrics.

A meter is uniquely identified by its name and dimensions. We use the terms, "dimensions" and "tags," interchangeably, and the Micrometer interface is Tag simply because it is shorter. As a general rule, it should be possible to use the name as a pivot. Dimensions let a particular named metric be sliced to drill down and reason about the data. This means that, if only the name is selected, you can drill down by using other dimensions and reason about the value being shown.

Modules/ROOT/Pages/Concepts/Naming

[[naming-meters]]
= Naming Meters

Micrometer employs a naming convention that separates lowercase words with a . (dot) character. Different monitoring systems have different recommendations regarding naming conventions, and some naming conventions may be incompatible between one system and another. Each Micrometer implementation for a monitoring system comes with a naming convention that transforms lowercase dot notation names to the monitoring system's recommended naming convention. Additionally, this naming convention implementation removes special characters that are disallowed by the monitoring system from the metric names and tags. You can override the default naming convention for a registry by implementing NamingConvention and setting it on the registry:

==== [source,java]

registry.config().namingConvention(myCustomNamingConvention);

====

With naming conventions in place, the following timer registered in Micrometer looks good natively in a wide variety of monitoring systems:

==== [source,java]

registry.timer("http.server.requests");

====

. Prometheus - http_server_requests_duration_seconds
. Atlas - httpServerRequests
. Graphite - http.server.requests
. InfluxDB - http_server_requests

By adhering to Micrometer's lowercase dot notation convention, you guarantee the maximum degree of portability for your metric names across monitoring systems.

== Tag Naming

TIP: We recommend that you follow the same lowercase dot notation described for meter names when naming tags. Using this consistent naming convention for tags allows for better translation into the respective monitoring system's idiomatic naming schemes.

Suppose we are trying to measure the number of http requests and the number of database calls.

Recommended Approach

==== [source,java]

registry.counter("database.calls", "db", "users") registry.counter("http.requests", "uri", "/api/users")

====

This variant provides enough context so that, if only the name is selected, the value can be reasoned about and is at least potentially meaningful. For example if we select database.calls, we can see the total number of calls to all databases. Then we can group by or select by db to drill down further or perform comparative analysis on the contribution of calls to each database.

Bad Approach

==== [source,java]

registry.counter("calls",
"class", "database",
"db", "users");

registry.counter("calls",
"class", "http",
"uri", "/api/users");

====

In this approach, if we select calls, we get a value that is an aggregate of the number of calls to the database and to our API endpoint. This time series is not useful without further dimensional drill-down.

== Common Tags

You can define common tags at the registry level and add them to every metric reported to the monitoring system. This is generally used for dimensional drill-down on the operating environment, such as host, instance, region, stack, and others. The following lines set the same tag in two equivalent ways:

==== [source,java]

registry.config().commonTags("stack", "prod", "region", "us-east-1"); registry.config().commonTags(Arrays.asList(Tag.of("stack", "prod"), Tag.of("region", "us-east-1"))); // equivalently

====

Further calls to commonTags append additional common tags.

[IMPORTANT]

Common tags generally have to be added to the registry before any (possibly autoconfigured) meter binders. Depending on your environment, there are different ways to achieve this.

If you use Spring Boot, you have two options:

Add your common tags with https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.customizing.common-tags[configuration properties]
If you need more flexibility (for example, you have to add common tags to a registry defined in a shared library), register a `MeterRegistryCustomizer` callback interface as a bean to add your common tags. See the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.getting-started[Spring Boot Reference Documentation] for more information.

== Tag Values

Tag values must be non-null.

WARNING: Beware of the potential for tag values coming from user-supplied sources to blow up the cardinality of a metric. You should always carefully normalize and add bounds to user-supplied input. Sometimes, the cause is sneaky. Consider the URI tag for recording HTTP requests on service endpoints. If we do not constrain 404's to a value like NOT_FOUND, the dimensionality of the metric would grow with each resource that cannot be found.

Modules/ROOT/Pages/Concepts/Rate Aggregation

[[rate-aggregation]]
= Rate Aggregation

Micrometer is aware of whether a particular monitoring system expects rate aggregation to happen client-side before metrics are published or ad-hoc as part of the query on the server. It accumulates metrics according to which style the monitoring system expects.

Not all measurements are reported or best viewed as a rate. For example, gauge values and the active task count long task timers are not rates.

== Server-side

Monitoring systems that perform server-side rate math expect absolute values to be reported at each publishing interval. For example, the absolute count of all increments to a counter since the beginning of the application is sent on each publishing interval.

Suppose we have a slightly positively biased random walk that chooses to increment a counter once every 10 milliseconds. If we view the raw counter value in a system like Prometheus, we see a step-wise monotonically increasing function (the width of the step is the interval at which Prometheus is polling or scraping for data).

image::concepts/prometheus-counter-norate.png[Absolute counter value]

Representing a counter without rate aggregation over some time window is rarely useful, as the representation is a function of both the rapidity with which the counter is incremented and the longevity of the service. In the preceding example, the counter drops back to zero on service restart. The rate-aggregated graph would return back to a value of around 55 as soon as the new instance (say on a production deployment) was in service.

image::concepts/prometheus-counter.png[Rate-aggregated counter]

If you have achieved zero-downtime deployments (for example, through red-black deployments), you should be able to comfortably set minimum alert thresholds on the rate-aggregated graph without service restarts causing dips in the counter value.

IMPORTANT: For most production purposes, whether it be alerting, automated canary analysis, or other use cases, base your automation off of rate-aggregated data.

== Client-side

Two other classes of monitoring system either:

Expect rate-aggregated data. Given the key insight that, for most production purposes, we should be basing our decisions off of rates rather than absolute values. Such systems benefit from having to do less math to satisfy queries.
Have relatively little or no math operations that would let us rate-aggregate data through our queries. For these systems, publishing pre-aggregated data is the only way to build meaningful representations.

Micrometer efficiently maintains rate data by means of a step value that accumulates data for the current publishing interval. When the step value is polled (when publishing, for example), if the step value detects that the current interval has elapsed, it moves current data to "previous" state. This previous state is what is reported until the next time current data overwrites it. The following image shows the interaction of current and previous state, along with polling:

image::concepts/rate-normalizing.png[Behavior of a step value,width=1200]

The value returned by the poll function is always rate per second * interval. If the step value shown in the preceding image represents the values of a counter, we could say that the counter saw "0.3 increments per second" in the first interval, which is reportable to the backend at any time during the second interval.

Micrometer timers track at least a count and the total time as separate measurements. Suppose we configure publishing at 10-second intervals and we saw 20 requests that each took 100ms. Then, for the first interval:

. count = 10 seconds * (20 requests / 10 seconds) = 20 requests
. totalTime = 10 seconds * (20 * 100 ms / 10 seconds) = 2 seconds

The count statistic is meaningful by itself: It is a measure of throughput. totalTime represents the total latency of all requests in the interval. Additionally, consider:

totalTime / count = 2 seconds / 20 requests = 0.1 seconds / request = 100 ms / request

This is a useful measure of average latency. When the same idea is applied to the totalAmount and count emanating from distribution summaries, the measure is called a distribution average. Average latency is just the distribution average for a distribution summary measured in time (a timer). Some monitoring systems (such as Atlas) provide facilities for computing the distribution average from these statistics, and Micrometer includes totalTime and count as separate statistics. Others, (such as Datadog) do not have this kind of operation built-in, and Micrometer calculates the distribution average client-side and ships that.

Shipping the rate for the publishing interval is sufficient to reason about the rate over any time window greater than or equal to the publishing interval. In our example, if a service continues to receive 20 requests that each take 100ms for every 10 second interval in a given minute, we could say:

. Micrometer reported "20 requests" for count on every 10-second interval. The monitoring system sums these six 10-second intervals and arrives at the conclusion that there are 120 requests / minute. Note that it is the monitoring system doing this summation, not Micrometer.
. Micrometer reported "2 seconds" of totalTime on every 10 second interval. The monitoring system can sum all total time statistics over the minute to yield "12 seconds" of total time in the minute interval. Then, the average latency is as we expect: 12 seconds / 120 requests = 100 ms / request.

Modules/ROOT/Pages/Concepts/Registry

[[registry]]
= Registry

Meters in Micrometer are created from and held in a MeterRegistry. Each supported monitoring system has an implementation of MeterRegistry. How a registry is created varies for each implementation.

Micrometer includes a SimpleMeterRegistry that holds the latest value of each meter in memory and does not export the data anywhere. If you do not yet have a preferred monitoring system, you can get started playing with metrics by using the simple registry:

==== [source,java]

MeterRegistry registry = new SimpleMeterRegistry();

====

NOTE: A SimpleMeterRegistry is autowired for you in Spring-based applications.

== Composite Registries

Micrometer provides a CompositeMeterRegistry to which you can add multiple registries, letting you publish metrics to more than one monitoring system simultaneously:

==== [source,java]

CompositeMeterRegistry composite = new CompositeMeterRegistry();

Counter compositeCounter = composite.counter("counter");
compositeCounter.increment(); <1>

SimpleMeterRegistry simple = new SimpleMeterRegistry();
composite.add(simple); <2>

compositeCounter.increment(); <3>

<1> Increments are NOOP'd until there is a registry in the composite. The counter's count still yields 0 at this point. <2> A counter named `counter` is registered to the simple registry. <3> The simple registry counter is incremented, along with counters for any other registries in the composite.

== Global Registry

Micrometer provides a static global registry called Metrics.globalRegistry and a set of static builders for generating meters based on this registry (note that globalRegistry is a composite registry):

==== [source,java]

class MyComponent {
Counter featureCounter = Metrics.counter("feature", "region", "test"); <1>

text

void feature() {
    featureCounter.increment();
}

void feature2(String type) {
    Metrics.counter("feature.2", "type", type).increment(); <2>
}

}

class MyApplication {
void start() {
// wire your monitoring system to global static state
Metrics.addRegistry(new SimpleMeterRegistry()); <3>
}

}

<1> Wherever possible (and especially where instrumentation performance is critical), store `Meter` instances in fields to avoid a lookup on their name or tags on each use. <2> When tags need to be determined from local context, you have no choice but to construct or lookup the Meter inside your method body. The lookup cost is just a single hash lookup, so it is acceptable for most use cases. <3> It is OK to add registries _after_ meters have been created with code like `Metrics.counter(...)`. These meters are added to each registry, as it is bound to the global composite.

Modules/ROOT/Pages/Concepts/Timers

[[timers]]
= Timers

Timers are intended for measuring short-duration latencies and the frequency of such events. All implementations of Timer report at least the total time and the count of events as separate time series but other time series can also be reported depending on what is supported by the backend (max, percentiles, histograms). While you can use timers for other use cases, note that negative values are not supported, and recording many longer durations could cause overflow of the total time at Long.MAX_VALUE nanoseconds (292.3 years).

As an example, consider a graph showing request latency to a typical web server. The server can be expected to respond to many requests quickly, so the timer gets updated many times per second.

The appropriate base unit for timers varies by metrics backend, and for good reason. Micrometer is decidedly un-opinionated about this. However, because of the potential for confusion, Micrometer requires a TimeUnit when interacting with Timer implementations. Micrometer is aware of the preferences of each implementation and publishes your timing in the appropriate base unit, based on the implementation. The following listing shows part of the Timer interface:

[source,java]

public interface Timer extends Meter {
...
void record(long amount, TimeUnit unit);
void record(Duration duration);
double totalTime(TimeUnit unit);

}

The interface contains a fluent builder for timers:

[source,java]

Timer timer = Timer
.builder("my.timer")
.description("a description of what this timer does") // optional
.tags("region", "test") // optional
.register(registry);

NOTE: The maximum statistical value for basic Timer implementations, such as CumulativeTimer and StepTimer, is a time window maximum (TimeWindowMax).
It means that its value is the maximum value during a time window.
If no new values are recorded for the time window length, the max is reset to 0 as a new time window starts.
The time window size until values are fully expired is the expiry multiplied by the bufferLength in DistributionStatisticConfig.
expiry defaults to the step size of the meter registry unless it's explicitly set to a different value, and bufferLength defaults to 3.
A time window maximum is used to capture maximum latency in a subsequent interval after heavy resource pressure triggers the latency and prevents metrics from being published.
Percentiles are also time window percentiles (TimeWindowPercentileHistogram). Histogram buckets usually behave like counters, so depending on the backend, they can be reported as cumulative values (for example in the case of Prometheus) or as a rate at which a counter increments over the push interval.

== Recording Blocks of Code

The Timer interface exposes several convenience overloads for recording timings inline, including the following:

[source,java]

timer.record(() -> dontCareAboutReturnValue());
timer.recordCallable(() -> returnValue());

Runnable r = timer.wrap(() -> dontCareAboutReturnValue()); <1> Callable c = timer.wrap(() -> returnValue());

<1> Wrap Runnable or Callable and return the instrumented version of it for use later.

NOTE: A Timer is really a specialized distribution summary that is aware of how to scale durations to the base unit of time of each monitoring system and has an automatically
determined base unit. In every case where you want to measure time, you should use a Timer rather than a DistributionSummary.

== Storing Start State in Timer.Sample

You may also store the start state in a sample instance that can be stopped later. The sample records a start time based on the registry's clock. After starting a sample, execute the code to be timed and finish the operation by calling stop(Timer) on the sample:

[source, java]

Timer.Sample sample = Timer.start(registry);

// do stuff
Response response = ...

sample.stop(registry.timer("my.timer", "response", response.status()));

Note how we do not decide the timer to which to accumulate the sample until it is time to stop the sample. This lets us dynamically determine certain tags from the end state of the operation we are timing.

== The @Timed Annotation

The micrometer-core module contains a @Timed annotation that frameworks can use to add timing support to either specific types of methods such as those serving web request endpoints or, more generally, to all methods.

[source,java]

@Configuration
public class TimedConfiguration {
@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
return new TimedAspect(registry);
}

}

Applying TimedAspect makes @Timed usable on any arbitrary method in an AspectJ proxied instance, as the following example shows:

[source,java]

@Service
public class ExampleService {

@Timed
public void sync() {
// @Timed will record the execution time of this method,
// from the start and until it exits normally or exceptionally.
...
}

@Async
@Timed
public CompletableFuture<?> async() {
// @Timed will record the execution time of this method,
// from the start and until the returned CompletableFuture
// completes normally or exceptionally.
return CompletableFuture.supplyAsync(...);
}

}

NOTE: TimedAspect doesn't support meta-annotations with @Timed.

=== @MeterTag on Method Parameters

To support using the @MeterTag annotation on method parameters, you need to configure the @TimedAspect to add the MeterTagAnnotationHandler.

[source,java,subs=+attributes]

include::{include-java}/metrics/TimedAspectTest.java[tags=resolvers,indent=0]

include::{include-java}/metrics/TimedAspectTest.java[tags=meter_tag_annotation_handler,indent=0]

Let's assume that we have the following interface.

[source,java,subs=+attributes]

include::{include-java}/metrics/TimedAspectTest.java[tags=interface,indent=0]

When its implementations would be called with different arguments (remember that the implementation needs to be annotated with @Timed annotation too), the following timers would be created:

[source,java,subs=+attributes]

// Example for returning <toString()> on the parameter
include::{include-java}/metrics/TimedAspectTest.java[tags=example_value_to_string,indent=0]

// Example for calling the provided on the parameter
include::{include-java}/metrics/TimedAspectTest.java[tags=example_value_resolver,indent=0]

// Example for calling the provided
include::{include-java}/metrics/TimedAspectTest.java[tags=example_value_spel,indent=0]

// Example for using multiple @MeterTag annotations on the same parameter // @MeterTags({ @MeterTag(...), @MeterTag(...) }) can be also used include::{include-java}/metrics/TimedAspectTest.java[tags=example_multi_annotations,indent=0]

== Function-tracking Timers

Micrometer also provides a more infrequently used timer pattern that tracks two monotonically increasing functions (a function that stays the same or increases over time but never decreases): a count function and a total time function. Some monitoring systems, such as Prometheus, push cumulative values for counters (which apply to both the count and total time functions in this case) to the backend, but others publish the rate at which a counter increments over the push interval. By employing this pattern, you let the Micrometer implementation for your monitoring system choose whether to rate normalize the timer, and your timer remains portable across different types of monitoring systems.

[source, java]

IMap cache = ...; // suppose we have a Hazelcast cache
registry.more().timer("cache.gets.latency", Tags.of("name", cache.getName()), cache,
c -> c.getLocalMapStats().getGetOperationCount(), <1>
c -> c.getLocalMapStats().getTotalGetLatency(),
TimeUnit.NANOSECONDS <2>

);

<1> getGetOperationCount() is a monotonically increasing function incrementing with every cache get from the beginning of its life.
<2> This represents the unit of time represented by getTotalGetLatency(). Each registry implementation specifies what its expected base unit of time is, and the total time reported will be scaled to this value.

The function-tracking timer, in concert with the monitoring system's rate normalizing functionality (whether this is an artifact of the query language or the way data is pushed to the system), adds a layer of richness to the cumulative value of the functions themselves. You can reason about the rate of throughput and latency, whether that rate is within an acceptable bound, is increasing or decreasing over time, and so on.

WARNING: Micrometer cannot guarantee the monotonicity of the count and total time functions for you. By using this signature, you are asserting their monotonicity based on what you know about their definitions.

There is also a fluent builder for function timers on the FunctionTimer interface itself, providing access to less frequently used options, such as base units and description. You can register the timer as the last step of its construction by calling register(MeterRegistry):

[source, java]

IMap cache = ...

FunctionTimer.builder("cache.gets.latency", cache,
c -> c.getLocalMapStats().getGetOperationCount(),
c -> c.getLocalMapStats().getTotalGetLatency(),
TimeUnit.NANOSECONDS)
.tags("name", cache.getName())
.description("Cache gets")
.register(registry);

WARNING: Attempting to construct a function-tracking timer with a primitive number or one of its java.lang object forms is always incorrect. These numbers are immutable. Thus, the function-tracking timer cannot ever be changed. Attempting to "re-register" the function-tracking timer with a different number does not work, as the registry maintains only one meter for each unique combination of name and tags. "Re-registering" a function-tracking timer can happen indirectly for example as the result of a MeterFilter modifying the name and/or the tags of two different function-tracking timers so that they will be the same after the filter is applied.

Attempting to "re-register" a function-tracking timer will result in a warning like this: [source]

WARNING: This FunctionTimer has been already registered (MeterId{name='my.ft', tags=[]}), the registration will be ignored. Note that subsequent logs will be logged at debug level.

== Pause Detection

Micrometer uses the LatencyUtils package to compensate for https://highscalability.com/blog/2015/10/5/your-load-generator-is-probably-lying-to-you-take-the-red-pi.html[coordinated omission] -- extra latency arising from system and VM pauses that skew your latency statistics downward. Distribution statistics, such as percentiles and SLO counts, are influenced by a pause detector implementation that adds additional latency here and there to compensate for pauses.

Micrometer supports two pause detector implementations: a clock-drift based detector and a no-op detector. The no-op detector is configured by default, but the clock-drift detector can be configured as shown in the next example.

The clock-drift based detector has a configurable sleep interval and pause threshold. CPU consumption is inversely proportional to sleepInterval, as is pause detection accuracy. 100ms for both values is a reasonable default to offer decent detection of long pause events while consuming a negligible amount of CPU time.

NOTE: You need to add LatencyUtils(org.latencyutils:LatencyUtils) to your runtime classpath to use the ClockDriftPauseDetector.
Without it on the runtime classpath, a warning will be logged and no pause detection will happen.

You can customize the pause detector as follows:

[source,java]

registry.config().pauseDetector(new ClockDriftPauseDetector(sleepInterval, pauseThreshold)); registry.config().pauseDetector(new NoPauseDetector());

In the future, we may provide further detector implementations. Some pauses may be able to be inferred from GC logging in some circumstances, for example, without requiring a constant CPU load, however minimal. Also, a future JDK may provide direct access to pause events.

== Memory Footprint Estimation

Timers are the most memory-consuming meter, and their total footprint can vary dramatically, depending on which options you choose. The following table of memory consumption is based on the use of various features. These figures assume no tags and a ring buffer length of 3. Adding tags adds somewhat to the total, as does increasing the buffer length. Total storage can also vary somewhat, depending on the registry implementation.

R = Ring buffer length. We assume the default of 3 in all examples. R is set with Timer.Builder#distributionStatisticBufferLength.
B = Total histogram buckets. Can be SLO boundaries or percentile histogram buckets. By default, timers are clamped to a minimum expected value of 1ms and a maximum expected value of 30 seconds, yielding 66 buckets for percentile histograms, when applicable.
I = Interval estimator for pause compensation. 1.7 kb.
M = Time-decaying max. 104 bytes.
Fb = Fixed boundary histogram. 8b * B * R.
Pp = Percentile precision. By default, it is 1. Generally in the range [0, 3]. Pp is set with Timer.Builder#percentilePrecision.
Hdr(Pp) = High dynamic range histogram.
- When Pp = 0: 1.9kb * R + 0.8kb
- When Pp = 1: 3.8kb * R + 1.1kb
- When Pp = 2: 18.2kb * R + 4.7kb
- When Pp = 3: 66kb * R + 33kb

|Yes |No |No |I + M| ~1.8kb
|Yes |No |Yes |I + M + Fb|For default percentile histogram, ~7.7kb
|Yes |Yes |Yes |I + M + Hdr(Pp)|For the addition of a 0.95 percentile with defaults otherwise, ~14.3kb
|No |No |No |M| ~0.1kb
|No |No |Yes |M + Fb|For default percentile histogram, ~6kb
|No |Yes |Yes |M + Hdr(Pp)|For the addition of a 0.95 percentile with defaults otherwise, ~12.6kb
|=========================================================

NOTE: For Prometheus, specifically, R is always equal to 1, regardless of how you attempt to configure it through Timer.Builder. This special case exists because Prometheus expects cumulative histogram data that never rolls over.

Modules/ROOT/Pages/Guides/Console Reporter

= Passing through to Dropwizard's Console Reporter

This guide shows how to plug in less commonly used Dropwizard Reporter implementations -- in this case, the ConsoleReporter.

[source,java]

text

@Bean
public MetricRegistry dropwizardRegistry() {
	return new MetricRegistry();
}

@Bean
public ConsoleReporter consoleReporter(MetricRegistry dropwizardRegistry) {
	ConsoleReporter reporter = ConsoleReporter.forRegistry(dropwizardRegistry)
			.convertRatesTo(TimeUnit.SECONDS)
			.convertDurationsTo(TimeUnit.MILLISECONDS)
			.build();
	reporter.start(1, TimeUnit.SECONDS);
	return reporter;
}

@Bean
public MeterRegistry consoleLoggingRegistry(MetricRegistry dropwizardRegistry) {
	DropwizardConfig consoleConfig = new DropwizardConfig() {

		@Override
		public String prefix() {
			return "console";
		}

		@Override
		public String get(String key) {
			return null;
		}

	};

	return new DropwizardMeterRegistry(consoleConfig, dropwizardRegistry, HierarchicalNameMapper.DEFAULT, Clock.SYSTEM) {
		@Override
		protected Double nullGaugeValue() {
			return null;
		}
	};
}

Note that, in a Spring environment, this registry is added to other implementations in a composite and is used for all metrics, both custom and
auto-configured.

[source,java]

class MyComponent {
private final MeterRegistry registry;

text

public MyComponent(MeterRegistry registry) {
    this.registry = registry;
}

public void doSomeWork(String lowCardinalityInput) {
    registry.timer("my.latency", "input", lowCardinalityInput).record(() -> {
        // do work
    });
}

}

The following listing shows typical output for this custom timer:

[source,txt]

3/2/18 10:14:27 AM =============================================================

-- Timers ----------------------------------------------------------------------
myLatency.lowCardinalityInput.INPUT
count = 1
mean rate = 1.02 calls/second
1-minute rate = 0.00 calls/second
5-minute rate = 0.00 calls/second
15-minute rate = 0.00 calls/second
min = 100.00 milliseconds
max = 100.00 milliseconds
mean = 100.00 milliseconds
stddev = 0.00 milliseconds
median = 100.00 milliseconds
75% <= 100.00 milliseconds
95% <= 100.00 milliseconds
98% <= 100.00 milliseconds
99% <= 100.00 milliseconds
99.9% <= 100.00 milliseconds

Modules/ROOT/Pages/Guides/Custom Meter Registry

= Custom Meter Registry

Micrometer includes support for popular meter registries, so you should check those first.
For an existing meter registry, if you think that your requirements are generally useful, consider creating an issue or PR against the Micrometer issue tracker.
For a non-existent meter registry, if it is widely-used, consider creating an issue or PR for it.

If you need to create your own custom meter registry, you can create it by extending one of the base classes for meter registries: MeterRegistry, PushMeterRegistry, or StepMeterRegistry.

The most common way is to extend StepMeterRegistry.
You can create your own custom StepMeterRegistry.

First, define your own meter registry configuration by extending StepRegistryConfig, as follows:

[source,java]

public interface CustomRegistryConfig extends StepRegistryConfig {

text

CustomRegistryConfig DEFAULT = k -> null;

@Override
default String prefix() {
	return "custom";
}

}

Second, define your own meter registry by extending StepMeterRegistry, as follows:

[source,java]

public class CustomMeterRegistry extends StepMeterRegistry {

text

public CustomMeterRegistry(CustomRegistryConfig config, Clock clock) {
	super(config, clock);

	start(new NamedThreadFactory("custom-metrics-publisher"));
}

@Override
protected void publish() {
	getMeters().stream().forEach(meter -> System.out.println("Publishing " + meter.getId()));
}

@Override
protected TimeUnit getBaseTimeUnit() {
	return TimeUnit.MILLISECONDS;
}

}

Finally, create the meter registry configuration and the meter registry.
If you use Spring Boot, you can do so as follows:

[source,java]

@Configuration
public class MetricsConfig {

text

@Bean
public CustomRegistryConfig customRegistryConfig() {
	return CustomRegistryConfig.DEFAULT;
}

@Bean
public CustomMeterRegistry customMeterRegistry(CustomRegistryConfig customRegistryConfig, Clock clock) {
	return new CustomMeterRegistry(customRegistryConfig, clock);
}

}

Modules/ROOT/Pages/Guides/Http Sender Resilience4j Retry

= HttpSender with Resilience4j Retry

HttpSender is an interface for HTTP clients that are used in meter registries for HTTP-based metrics publication. There
are two implementations:

HttpUrlConnectionSender: HttpURLConnection-based HttpSender
OkHttpSender: OkHttp-based HttpSender

Micrometer does not include support for retry, but you can decorate it with Resilience4j retry, as follows:

[source,java]

text

@Bean
public DatadogMeterRegistry datadogMeterRegistry(DatadogConfig datadogConfig, Clock clock) {
	return DatadogMeterRegistry.builder(datadogConfig)
			.clock(clock)
			.httpClient(new RetryHttpClient())
			.build();
}

private static class RetryHttpClient extends HttpUrlConnectionSender {

	private final RetryConfig retryConfig = RetryConfig.custom()
			.maxAttempts(2)
			.waitDuration(Duration.ofSeconds(5))
			.build();

	private final Retry retry = Retry.of("datadog-metrics", this.retryConfig);

	@Override
	public Response send(Request request) {
		CheckedFunction0<Response> retryableSupplier = Retry.decorateCheckedSupplier(
				this.retry,
				() -> super.send(request));
		return Try.of(retryableSupplier).get();
	}

}

Modules/ROOT/Pages/Implementations/ Install

[id=installing-micrometer-registry-{system}]
== Installing micrometer-registry-{system}

It is recommended to use the BOM provided by Micrometer (or your framework if any), you can see how to configure it xref:../installing.adoc[here]. The examples below assume you are using a BOM.

=== Gradle

After the BOM is xref:../installing.adoc[configured], add the following dependency:

[source,groovy,subs=+attributes]

implementation 'io.micrometer:micrometer-registry-{system}'

NOTE: The version is not needed for this dependency since it is defined by the BOM.

=== Maven

After the BOM is xref:../installing.adoc[configured], add the following dependency:

[source,xml,subs=+attributes]

io.micrometer micrometer-registry-{system} ----

NOTE: The version is not needed for this dependency since it is defined by the BOM.

Modules/ROOT/Pages/Implementations/AppOptics

= Micrometer AppOptics
:sectnums:
:system: appoptics

AppOptics is a dimensional time-series SaaS with built-in dashboarding.

include::_install.adoc[]

== Configuring

The following example configures an AppOptics instance:

[source,java]

AppOpticsConfig appopticsConfig = new AppOpticsConfig() {
@Override
public String apiToken() {
return MY_TOKEN;
}

text

@Override
public @Nullable String get(String k) {
    return null;
}

}; MeterRegistry registry = new AppOpticsMeterRegistry(appopticsConfig, Clock.SYSTEM);

AppOpticsConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.appoptics directly to the AppOpticsConfig:

[source,yml]

management.metrics.export.appoptics:
api-token: YOURKEY

text

# You will probably want disable AppOptics publishing in a local development profile.
enabled: true

# The interval at which metrics are sent to AppOptics. The default is 1 minute.
step: 1m

== Graphing

This section serves as a quick start to rendering useful representations in AppOptics for metrics that originate in Micrometer.

=== Timers

The AppOptics implementation of Timer produces three fields in AppOptics:

count: Rate of number of calls/second.
sum: Rate of total time/second.
max: A sliding window showing the maximum amount recorded.

.Dimensionally aggregable average in AppOptics.
image::implementations/appoptics-timer-average.png[AppOptics timer average]

AppOptics performs the sum/count division dimensionally to generate aggregable averages on your behalf.

.Timer over a simulated service.
image::implementations/appoptics-timer.png[AppOptics-rendered timer]

Modules/ROOT/Pages/Implementations/Atlas

= Micrometer Atlas
:sectnums:
:system: atlas

Atlas is an in-memory dimensional time series database with built-in graphing, a custom stack-based query language, and advanced math operations. Atlas originated at Netflix, where it remains the operational metrics solution.

include::_install.adoc[]

== Configuring

[source,java]

AtlasConfig atlasConfig = new AtlasConfig() {
@Override
public Duration step() {
return Duration.ofSeconds(10);
}

text

@Override
public String get(String k) {
    return null; // accept the rest of the defaults
}

}; MeterRegistry registry = new AtlasMeterRegistry(atlasConfig, Clock.SYSTEM);

Micrometer uses Netflix's https://github.com/netflix/spectator[Spectator] as the underlying instrumentation library when recording metrics destined for Atlas. AtlasConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties prefixed with management.metrics.export.atlas directly to the AtlasConfig:

[source,yml]

management.metrics.export.atlas:
# The location of your Atlas server
uri: http://localhost:7101/api/v1/publish

text

# You will probably want to conditionally disable Atlas publishing in local development.
enabled: true

# The interval at which metrics are sent to Atlas. The default is 1 minute.
step: 1m

== Graphing

This section serves as a quick start to rendering useful representations in Atlas for metrics originating in Micrometer. See the https://github.com/netflix/atlas/wiki[Atlas wiki] for a far more complete reference of what is possible in Atlas.

=== Counters

Atlas serves up graphs in the form of PNG images (and other https://github.com/Netflix/atlas/wiki/Output-Formats[output formats] as well).

We use the following query to visualize the counter from Atlas. Note that the value is rate-normalized over the step interval rather than monotonically increasing. Atlas always expects xref:../concepts/rate-aggregation.adoc#_client_side[rate-aggregated] data for counters from Micrometer.

.Counter over a positive-biased random walk.
image::implementations/atlas-counter.png[Atlas-rendered counter]

[source,http]

GET /api/v1/graph?
q=
name,counter,:eq,
2,:lw
&tz=US/Central
&s=e-15m <1>
&w=400 <2>
&l=0 <3>

Host: localhost:7101

<1> The range of time we want to visualize along the x-axis. e represents the end time or "now". This graph's axis is from 15 minutes ago until now. Atlas automatically chooses the finest grained step interval available from the data that would render at least 1px wide on the resultant image.
<2> The overall width of the PNG image returned should be 400px.
<3> Set the y-axis lower limit to 0 so that random perturbation in the walk does not look so dramatic.

=== Timers

While reading directly from a Timer returns a double, the underlying value is
stored in https://github.com/netflix/spectator[Spectator] as a nanosecond-precise long. What precision is lost by
converting to a double in the Timer interface does not affect a system like
Atlas, because it has been configured to read measurements from the underlying
Spectator Timer that Micrometer is hiding from you.

The Spectator Atlas Timer produces four time series, each with a different statistic tag:

count: Rate of calls per second.
totalTime: Rate of total time per second.
totalOfSquares: Rate of total time squared per second (useful for standard deviation).
max: The maximum amount recorded.

Therefore, you can achieve a throughput (requests/second) line with the following query:

http

name,timer,:eq,statistic,count,:eq,:and

Notice that statistic is just a dimension that can be drilled down and selected like any other.

Furthermore, totalTime/count represents average latency and can be selected with a short-hand :dist-avg query, which selects the totalTime and count time series and performs the division for us:

http

name,timer,:eq,:dist-avg

In the preceding example, you can see these two lines plotted on a single dual-axis graph.

.Timer over a simulated service.
image::implementations/atlas-timer.png[Atlas-rendered timer]

=== Long Task Timers

Suppose we had a task that took two minutes to complete when it was expected to complete in less than 70 seconds. A key benefit of long task timers is the ability to receive an alert at the first reporting interval after we have exceeded the threshold. With a regular timer, we would not receive an alert until the first reporting interval after the process completed. If we had a ten-second publishing interval, the regular timer alert would arrive almost a minute after the long task timer alert.

.Simulated back-to-back long tasks.
image::implementations/atlas-long-task-timer.png[Atlas-rendered long task timer]

[source, http]

GET /api/v1/graph?
q=
name,longTaskTimer,:eq,statistic,duration,:eq,:and, <1>
:dup,
70,:gt,:vspan,f00,:color,40,:alpha,alerted,:legend, <2>
70,f00,:color,alert+threshold,:legend <3>
&tz=US/Central
&s=e-15m
&w=400
&l=0
&title=Peaks+of+Long+Tasks
&ylabel=time

Host: localhost:7101

<1> A representation of long tasks that are happening back-to-back.
<2> A vertical span that appears whenever the long task exceeds our threshold of 70 seconds. So that it does not overwhelm the graph, we also decrease the opacity of the vspan.
<3> Plot the threshold of 70 seconds as a separate line.

Modules/ROOT/Pages/Implementations/Azure Monitor

= Micrometer Azure Monitor
:sectnums:
:system: azure-monitor

Azure Monitor is a dimensional time-series SaaS with built-in dashboarding.

include::_install.adoc[]

== Configuring

The following example configures a Micrometer Azure Monitor:

[source,java]

AzureMonitorConfig azureMonitorConfig = new AzureMonitorConfig() {
@Override
public String instrumentationKey() {
return MY_KEY;
}

text

@Override
public String get(String key) {
    return null;
}

}; MeterRegistry registry = new AzureMonitorMeterRegistry(azureMonitorConfig, Clock.SYSTEM);

AzureMonitorConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.azure-monitor directly to the AzureMonitorConfig:

[source,yml]

management.metrics.export.azure-monitor:
instrumentation-key: YOURKEY

text

# You will probably want disable Azure Monitor publishing in a local development profile.
enabled: true

# The interval at which metrics are sent to Azure Monitor. The default is 1 minute.
step: 1m

Modules/ROOT/Pages/Implementations/Cloudwatch

= Micrometer CloudWatch
Tommy Ludwig [email protected]
:sectnums:
:system: cloudwatch2

https://aws.amazon.com/cloudwatch/[Amazon CloudWatch] is a dimensional time-series SaaS on Amazon's cloud.

include::_install.adoc[]

== Configuring

The following example configures Micrometer CloudWatch:

[source,java]

CloudWatchConfig cloudWatchConfig = new CloudWatchConfig() {
@Override
public String get(String s) {
return null;
}

text

@Override
public String namespace() {
    return "mynamespace";
}

}; MeterRegistry meterRegistry = new CloudWatchMeterRegistry(cloudWatchConfig, Clock.SYSTEM, CloudWatchAsyncClient.create());

You can provide your own CloudWatchAsyncClient to the constructor of the registry.

CloudWatchConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, https://docs.awspring.io/spring-cloud-aws/docs/3.1.0/reference/html/index.html#cloudwatch-metrics[Micrometer support in Spring Cloud AWS] binds properties prefixed with management.metrics.export.cloudwatch directly to the CloudWatchConfig:

[source,yml]

management.metrics.export.cloudwatch:
namespace: YOURNAMESPACE

text

# You will probably want to disable publishing in a local development profile.
enabled: true

# The interval at which metrics are sent to CloudWatch. The default is 1 minute.
step: 1m

Modules/ROOT/Pages/Implementations/Datadog

= Micrometer Datadog
:sectnums:
:system: datadog

Datadog is a dimensional time-series SaaS with built-in dashboarding and alerting.

== Installation and Configuration

Micrometer supports shipping metrics to Datadog directly by using its HTTP API or by using DogStatsD through the xref:./statsD.adoc[StatsD registry].
If you can choose between the two, the API approach is far more efficient.

NOTE: If you encounter a rate limit problem with the Datadog API approach, try the DogStatsD approach or one of https://docs.datadoghq.com/metrics/guide/micrometer/[alternatives that are described in the Datadog documentation].

It is recommended to use the BOM provided by Micrometer (or your framework if any), you can see how to configure it xref:../installing.adoc[here]. The examples below assume you are using a BOM.

=== Direct to Datadog API Approach

==== Gradle

After the BOM is xref:../installing.adoc[configured], add the following dependency:

[source,groovy]

implementation 'io.micrometer:micrometer-registry-datadog'

NOTE: The version is not needed for this dependency since it is defined by the BOM.

==== Maven

After the BOM is xref:../installing.adoc[configured], add the following dependency:

[source,xml]

io.micrometer micrometer-registry-datadog ----

NOTE: The version is not needed for this dependency since it is defined by the BOM.

Metrics are rate-aggregated and pushed to datadoghq on a periodic interval. Rate aggregation performed by the registry yields datasets that are similar to those produced by dogstatsd.

[source, java]

DatadogConfig config = new DatadogConfig() {
@Override
public Duration step() {
return Duration.ofSeconds(10);
}

text

@Override
public String get(String k) {
    return null; // accept the rest of the defaults
}

}; MeterRegistry registry = new DatadogMeterRegistry(config, Clock.SYSTEM);

DatadogConfig is an interface with a set of default methods.
If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration through properties.
For example, Spring Boot's Micrometer support binds properties directly to the DatadogConfig.
See the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.export.datadog[Datadog] section in the Spring Boot reference documentation.

DatadogConfig.hostTag() specifies a tag key that is mapped to https://docs.datadoghq.com/api/v1/metrics/#submit-metrics[the host field] when shipping metrics to Datadog.
For example, if DatadogConfig.hostTag() returns host, the tag having host as its key is used.
You can set the tag by using common tags, as follows:

[source,java]

registry.config().commonTags("host", "my-host");

uri is an important property to configure.
The default value is https://api.datadoghq.com.
Depending on the Datadog site (region), the api endpoint will be different.
To find your the correct uri for your account, do the following:

Read about the https://docs.datadoghq.com/getting_started/site/[Datadog site].
Go to the https://docs.datadoghq.com/api/latest/metrics/[Metrics API reference] and select your own option from the "DATADOG SITE" dropdown.
Check any API request's endpoint.

For example, for the US5 site, the correct API endpoint is https://api.us5.datadoghq.com. For the US3 site, it is https://api.us3.datadoghq.com.

=== Through DogStatsD Approach

==== Gradle

After the BOM is xref:../installing.adoc[configured], add the following dependency:

[source,groovy]

implementation 'io.micrometer:micrometer-registry-statsd'

NOTE: The version is not needed for this dependency since it is defined by the BOM.

==== Maven

After the BOM is xref:../installing.adoc[configured]d, add the following dependency:

[source,xml]

io.micrometer micrometer-registry-statsd ----

NOTE: The version is not needed for this dependency since it is defined by the BOM.

Metrics are immediately shipped to DogStatsD using Datadog's flavor of the StatsD line protocol. java-dogstatsd-client is not needed on the classpath for this to work, as Micrometer uses its own implementation.

[source,java]

StatsdConfig config = new StatsdConfig() {
@Override
public String get(String k) {
return null;
}

text

@Override
public StatsdFlavor flavor() {
    return StatsdFlavor.DATADOG;
}

};

MeterRegistry registry = new StatsdMeterRegistry(config, Clock.SYSTEM);

If the DD_ENTITY_ID environment variable is properly set, Micrometer supports DogStatsD's https://docs.datadoghq.com/developers/dogstatsd/?tab=kubernetes#origin-detection-over-udp[origin detection over UDP] feature on Kubernetes.

Micrometer, by default, publishes Timer meters to DogStatsD as the StatsD "timing" metric type, ms.
These meters are sent to Datadog as https://docs.datadoghq.com/metrics/types/?tab=histogram#metric-types[histogram] type metrics.
Also, by default, Micrometer publishes DistributionSummary meters as histogram type metrics.

When percentileHistogram is enabled for the meter, Micrometer sends Timer and DistributionSummary meters as Datadog https://docs.datadoghq.com/metrics/distributions[Distributions] to DogStatsD.
You can make a DistributionSummary with percentileHistogram enabled, as follows:

[source,java]

DistributionSummary responseSizeSummary = DistributionSummary.builder("http.server.response.size")
.baseUnit("bytes")
.publishPercentileHistogram()
.register(registry);

Modules/ROOT/Pages/Implementations/Dynatrace

= Micrometer Dynatrace
:sectnums:
:system: dynatrace

https://www.dynatrace.com/[*Dynatrace*] is a Software Intelligence Platform featuring application performance monitoring (APM), artificial intelligence for operations (AIOps), IT infrastructure monitoring, digital experience management (DEM), and digital business analytics capabilities.
It can ingest multi-purpose dimensional time-series data and has built-in dashboarding.
Both SaaS and self-hosted (Managed) deployments are offered.

include::_install.adoc[]

== Configuring

For setting up new integrations with Dynatrace, we recommend using the latest version of the https://docs.dynatrace.com/docs/shortlink/api-metrics-v2[Dynatrace Metrics API] (currently version 2).
If you use Micrometer with Spring Boot, see the https://docs.dynatrace.com/docs/shortlink/micrometer-metrics-ingest[Dynatrace documentation] and the https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.metrics.export.dynatrace[Spring Boot documentation].
Dynatrace provides different ways of setting up integrations:

=== Using Dynatrace auto-configuration (preferred) [[bookmark-auto-configuration]]

Dynatrace auto-configuration is available for hosts that are monitored by OneAgent or by the Dynatrace Operator for Kubernetes.

If Dynatrace OneAgent is installed on the host running Micrometer, metrics can be exported directly by using OneAgent without having to specify an endpoint URI or API token.
If running in Kubernetes with the Dynatrace operator installed, the registry picks up your endpoint URI and API token from the operator instead.
In this case, there is no need to configure anything, so you can use the following code in your project to export Micrometer metrics to Dynatrace:

[source,java]

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
@Override
public @Nullable String get(String k) {
// This method of the interface is used by the other configuration methods and needs to be
// implemented here. Returning null accepts the defaults for the other configuration items.
return null;
}
};

MeterRegistry registry = new DynatraceMeterRegistry(dynatraceConfig, Clock.SYSTEM);

If you use Micrometer 1.10.0 or above, you can also use the DEFAULT config to achieve the same with less code:

[source,java]

MeterRegistry registry = new DynatraceMeterRegistry(DynatraceConfig.DEFAULT, Clock.SYSTEM);

You can also change other properties by creating an instance of the DynatraceConfig and overwriting the respective methods.
For example, you can specify the exporter version, which defaults to v2 unless a deviceId is set:

[source,java]

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
@Override
public DynatraceApiVersion apiVersion() {
return DynatraceApiVersion.V2;
}

text

@Override
public @Nullable String get(String k) {
    return null; // accept the rest of the defaults
}

}; MeterRegistry registry = new DynatraceMeterRegistry(dynatraceConfig, Clock.SYSTEM);

DynatraceConfig is an interface with a set of default methods.
Spring Boot's Micrometer support binds properties prefixed with management.dynatrace.metrics.export directly to the DynatraceConfig.

NOTE: Property names for binding attributes from Spring Boot have changed in Spring Boot version 3.0.0. If you use a Spring Boot version before 3.0.0, use management.metrics.export.dynatrace instead of management.dynatrace.metrics.export.

Using Spring Boot Micrometer support allows configuring the Dynatrace exporter by using <>.
When using Micrometer with Spring Boot, you need not instantiate the DynatraceMeterRegistry manually, as Spring Boot does it automatically for you.
All configuration options that can be set by overwriting methods can also be set through Spring Boot properties, and adding a separate MeterRegistry can lead to metrics not being exported, as auto-configuration might break.

To use the Dynatrace metrics exporter for Micrometer in your Spring Boot project, include the runtimeOnly 'io.micrometer:micrometer-registry-dynatrace' dependency.
In this default configuration, metrics are exported to the local OneAgent or Kubernetes operator-provided endpoint.

=== Using a Custom Endpoint

If auto-configuration is not available on the host, both the Dynatrace Metrics API v2 endpoint and an API token have to be specified.
The https://docs.dynatrace.com/docs/shortlink/api-authentication[Dynatrace API token documentation] contains more information on how to create an API token.
To ingest metrics, the 'Ingest metrics' (metrics.ingest) permission is required on the token.
We recommend limiting scope to only this permission.

[source,java]

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
@Override
public DynatraceApiVersion apiVersion() {
// not strictly required, but makes the code more clear/explicit
return DynatraceApiVersion.V2;
}

text

@Override
public String uri() {
    // The endpoint of the Dynatrace Metrics API v2 including path, e.g.:
    // "https://{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest"
    String endpoint = System.getenv("ENVVAR_METRICS_INGEST_URL");
    return endpoint != null ? endpoint : DynatraceConfig.super.uri();
}

@Override
public String apiToken() {
    // should be read from a secure source
    String token = System.getenv("ENVVAR_METRICS_INGEST_TOKEN");
    return token != null ? token : "";
}

@Override
public @Nullable String get(String k) {
    return null; // accept the rest of the defaults
}

}; MeterRegistry registry = new DynatraceMeterRegistry(dynatraceConfig, Clock.SYSTEM);

These properties can also be set through Spring Boot, by using property or yaml files.
You can also reference environment variables by using the Spring property placeholders (such as management.dynatrace.metrics.export.uri: $\{DT_METRICS_INGEST_URL}).

NOTE: v2 is used as the default API version unless a deviceId is set, as described <>.

[source,yml]

For Spring Boot 3.0.0 and above:

management.dynatrace.metrics.export:

For Spring Boot versions below 3.0.0, use the line below instead of the line above:

management.metrics.export.dynatrace:

text

# for SaaS: https://\{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest
# for managed deployments: https://{your-domain}/e/\{your-environment-id}/api/v2/metrics/ingest
uri: YOUR_METRICS_INGEST_URL

# should be read from a secure source
api-token: YOUR_METRICS_INGEST_TOKEN

=== Meter metadata

Starting with Micrometer 1.12.0, the Dynatrace registry v2 exports meter metadata to Dynatrace.
Currently supported types of metadata are unit (called "base unit" in Micrometer) and description.
No changes are required to start exporting Metadata to Dynatrace - upgrading to version 1.12.0 or above is enough.
Find more information about metrics metadata in the https://docs.dynatrace.com/docs/shortlink/metric-ingestion-protocol#metadata[Dynatrace documentation].

The export of metrics metadata can be disabled by setting the exportMeterMetadata property on the DynatraceConfig (see <> below) to false.

== API Versions

=== API v2 [[bookmark-apiv2]]

When the API version is configured to v2, the registry sends data by using the https://docs.dynatrace.com/docs/shortlink/api-metrics-v2[Metrics API v2].
To maintain backwards compatibility, when a deviceId is set (which is required for v1 and not used in v2), v1 is used as the default.
Otherwise, the version defaults to v2 and does not have to be set explicitly.
With no endpoint URI and token set, metrics are exported to the local OneAgent endpoint or, if running in Kubernetes with the Dynatrace operator installed, to the endpoint provided by the operator.
If you do not need auto-configuration, you can specify the endpoint and token explicitly, to export metrics to that specific endpoint.
Explicitly specifying these will overwrite auto-configuration.

Minimal Configuration with Dynatrace Auto-configuration

In the minimal configuration <> (no URI or API token), the v2 registry tries to retrieve the endpoint provided by the Dynatrace Kubernetes operator.
If the operator is not set up or does not provide this information, the exporter tries to send metrics to the https://docs.dynatrace.com/docs/shortlink/local-api[local OneAgent metrics ingest endpoint].
Note that this only works if a OneAgent is running on the host and the https://docs.dynatrace.com/docs/shortlink/local-api#enable-the-oneagent-metric-api[local OneAgent Metric API] is available.
If the ingestion port for the local OneAgent was changed to a custom one, the full endpoint URI has to be provided for the URI property (with API token left empty).

Configuration with URI and API Token

If no auto-configuration is available or the metrics should be sent to a different endpoint (such as a different tenant), you can configure the Dynatrace v2 exporter with an explicit endpoint URI and an https://docs.dynatrace.com/docs/shortlink/api-authentication[API token].
The https://docs.dynatrace.com/docs/shortlink/api-authentication[API token] must have the https://docs.dynatrace.com/docs/dynatrace-api/basics/dynatrace-api-authentication#token-scopes["Ingest metrics"] (metrics.ingest) permission set.
We recommend limiting the scope to only this permission.

You must specify the entire Metrics v2 API endpoint URI, including its path -- that is, with the /api/v2/metrics/ingest path on SaaS and managed deployments, or /metrics/ingest for OneAgent endpoints as mentioned in the https://docs.dynatrace.com/docs/shortlink/api-metrics-v2-post-datapoints[documentation].

Properties Available in the v2 Exporter [[bookmark-available-properties]]

When using the https://docs.dynatrace.com/docs/shortlink/api-metrics-v2[Dynatrace metrics API v2], you can set the following properties:

[source,java]

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
@Override
public DynatraceApiVersion apiVersion() {
return DynatraceApiVersion.V2;
}

text

@Override
public String uri() {
    // The endpoint of the Dynatrace Metrics API v2 including path, e.g.:
    // "https://{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest".
    String endpoint = System.getenv("ENVVAR_METRICS_INGEST_URL");
    return endpoint != null ? endpoint : DynatraceConfig.super.uri();
}

@Override
public String apiToken() {
    // should be read from a secure source
    String token = System.getenv("ENVVAR_METRICS_INGEST_TOKEN");
    return token != null ? token : "";
}

@Override
public String metricKeyPrefix() {
    // will be prepended to all metric keys
    return "your.desired.prefix";
}

@Override
public boolean enrichWithDynatraceMetadata() {
    return true;
}

@Override
public Map<String, String> defaultDimensions() {
    // create and return a map containing the desired key-value pairs.
    Map<String, String> dims = new HashMap<>();
    dims.put("dimensionKey", "dimensionValue");
    return dims;
}

// Only available in Micrometer 1.9.0 and above
@Override
public boolean useDynatraceSummaryInstruments() {
    return false;
}

// Only available in Micrometer 1.12.0 and above
@Override
public boolean exportMeterMetadata() {
    return true;
}

@Override
public @Nullable String get(String k) {
    return null; // accept the rest of the defaults
}

};

You can also set these properties in Spring Boot configuration files:

[source,yml]

management.dynatrace.metrics.export:
# Required only if not using the OneAgent endpoint
# For SaaS: https://{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest
# For managed deployments: https://{your-domain}/e/{your-environment-id}/api/v2/metrics/ingest
uri: YOUR_METRICS_INGEST_URL

text

# should be read from a secure source
api-token: YOUR_METRICS_INGEST_TOKEN

# These properties can only be used with the v2 exporter.
v2:
    # Sets a prefix that is prepended to each exported metric key.
    metric-key-prefix: my.metric.key.prefix

    # If set to true and a local OneAgent or operator is running, retrieves metadata
    # and adds it as additional dimensions to all data points (default: true)
    enrich-with-dynatrace-metadata: true

    # Sets an arbitrary number of key-value pairs as default dimensions.
    # Micrometer tags will overwrite these dimensions, if they have the same key.
    # Each exported metric will contain these dimensions.
    default-dimensions:
        key1: "value1"
        key2: "value2"

    # (since 1.9.0) Whether or not to use the Dynatrace-specific summary instruments. (default: true)
    # This should only be disabled if problems with existing instrumentation are discovered after upgrading to 1.9.0.
    # Set to false, this will restore the previous (1.8.x) behavior for Timers and DistributionSummaries.
    use-dynatrace-summary-instruments: true

    # (since 1.12.0) Determines whether meter metadata (unit, description) should be exported.
    export-meter-metadata: true

# The export interval in which metrics are sent to Dynatrace (default: 60s).
step: 60s

For more information about the metadata picked up by the Dynatrace metadata enrichment feature, see https://docs.dynatrace.com/docs/shortlink/enrichment-files[the Dynatrace documentation].

In Micrometer 1.9.0, Dynatrace-specific summary instruments (DynatraceTimer and DynatraceDistributionSummary) were introduced.
These specialized instruments are tailored to the Dynatrace metrics ingest, and prevent the creation of invalid metrics.
They are available from version 1.9.0 and are used, by default, as a drop-in replacement.
No action is needed from users upgrading to 1.9.0. If there is a discrepancy in the observed metrics, you can return to the previous behavior by setting the useDynatraceSummaryInstruments toggle to false.

=== API v1 (Legacy) [[bookmark-apiv1]]

CAUTION: The Dynatrace v1 API is deprecated. It will be removed in the future. Please use the v2 API instead. See https://docs.dynatrace.com/docs/discover-dynatrace/references/dynatrace-api/environment-api/metric-v1[Dynatrace API Docs].

When the apiVersion is configured to be v1, the registry sends data by using the https://docs.dynatrace.com/docs/shortlink/api-custom-metrics[Dynatrace Timeseries API v1 for Custom Metrics].
If a deviceId is specified, it defaults to v1 for backwards compatibility with earlier setups.
The device-id property is required for v1 and not used in v2.
Existing setups continue to work when updating to newer versions of Micrometer.
The reported metrics are assigned to https://docs.dynatrace.com/docs/shortlink/api-custom-device-report-metric[Custom Devices] in Dynatrace.

For the v1 API, do not specify the ingest path, but only the base URL of your environment, such as uri: https://\{your-environment-id}.live.dynatrace.com:

[source,java]

DynatraceConfig dynatraceConfig = new DynatraceConfig() {
@Override
public String uri() {
// The Dynatrace environment URI without any path, e.g.:
// https://{your-environment-id}.live.dynatrace.com
return MY_DYNATRACE_URI;
}

text

@Override
public String apiToken() {
    // should be read from a secure source
    return MY_TOKEN;
}

@Override
public String deviceId() {
    return MY_DEVICE_ID;
}

@Override
public @Nullable String get(String k) {
    return null; // accept the rest of the defaults
}

}; MeterRegistry registry = new DynatraceMeterRegistry(dynatraceConfig, Clock.SYSTEM);

[source,yml]

management.dynatrace.metrics.export:
# For v1 export, do not append a path to the endpoint URL, e.g.:
# For SaaS: https://{your-environment-id}.live.dynatrace.com
# For managed deployments: https://{your-domain}/e/{your-environment-id}
uri: https://{your-environment-id}.live.dynatrace.com

text

# should be read from a secure source
api-token: MY_TOKEN

# When setting the device id, metrics will be exported to the v1 timeseries endpoint
# Using just device-id (without the v1 prefix) is deprecated, but will work to maintain backwards compatibility.
v1:
    device-id: sample

# To disable Dynatrace publishing, e.g. in a local development profile, use:
# enabled: false

# The interval at which metrics are sent to Dynatrace. The default is 1 minute.
step: 1m

Modules/ROOT/Pages/Implementations/Elastic

= Micrometer Elastic
:sectnums:
:system: elastic

Elasticsearch is an open source search and analytics platform. Metrics stored in Elasticsearch can be visualized in Kibana.

include::_install.adoc[]

== Configuring

The following example configures an ElasticSearch instance:

[source,java]

ElasticConfig elasticConfig = new ElasticConfig() {
@Override
public @Nullable String get(String k) {
return null;
}
};

MeterRegistry registry = new ElasticMeterRegistry(elasticConfig, Clock.SYSTEM);

ElasticConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.elastic directly to the ElasticConfig:

[source,yml]

management.metrics.export.elastic:
# You will probably want disable Elastic publishing in a local development profile.
enabled: true

text

# The interval at which metrics are sent to Elastic. The default is 1 minute.
step: 1m

# The index to store metrics in, defaults to "micrometer-metrics"
index: micrometer-metrics

== Elastic APM agent integration

If you are using the Elastic APM agent, it can automatically collect metrics from Micrometer MeterRegistry instances. If you want only metrics collected by the Elastic APM agent and not shipped anywhere else, you can use the SimpleMeterRegistry. See the https://www.elastic.co/guide/en/apm/agent/java/current/metrics.html#metrics-micrometer[Elastic docs] for more details.

Modules/ROOT/Pages/Implementations/Ganglia

= Micrometer Ganglia
:sectnums:
:system: ganglia

Ganglia is an aging hierarchical metrics system which enjoyed wide popularity in Linux system monitoring and is still in place in many organizations. It originated in the early 2000s at the University of California, Berkeley.

NOTE: The micrometer-registry-ganglia module uses the https://github.com/ganglia/gmetric4j[gmetric4j] library, which contains classes generated by the LGPL licensed https://sourceforge.net/projects/remotetea/[remotetea project].

include::_install.adoc[]

== Configuring

The following example configures a Ganglia instance:

[source,java]

GangliaConfig gangliaConfig = new GangliaConfig() {
@Override
public String host() {
return "mygraphitehost";
}

text

@Override
public String get(String k) {
    return null; // accept the rest of the defaults
}

};

MeterRegistry registry = new GangliaMeterRegistry(gangliaConfig, Clock.SYSTEM);

Micrometer uses Dropwizard Metrics as the underlying instrumentation library when recording metrics destined for Ganglia. GangliaConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.ganglia directly to the GangliaConfig:

[source,yml]

management.metrics.export.ganglia:
# The location of your Ganglia server
host: mygraphitehost

text

# You will probably want to conditionally disable Ganglia publishing in local development.
enabled: true

# The interval at which metrics are sent to Ganglia. The default is 1 minute.
step: 1m

include::hierarchical-name-mapping.adoc[]

== Graphing

This section serves as a quick start to rendering useful representations in Ganglia for metrics originating in Micrometer.

=== Counters

Ganglia counters measure mean throughput and one-, five-, and fifteen-minute exponentially-weighted moving average throughputs.

.A Ganglia rendered graph of the random walk counter.
image::implementations/ganglia-counter.png[Ganglia-rendered counter]

Modules/ROOT/Pages/Implementations/Graphite

= Micrometer Graphite
:sectnums:
:system: graphite

Graphite is one of the most popular current hierarchical metrics systems backed by a fixed-size database, similar in design and purpose to RRDtool. It originated at Orbitz in 2006 and was open sourced in 2008.

include::_install.adoc[]

== Configuring

The following example configures a Graphite instance:

[source,java]

GraphiteConfig graphiteConfig = new GraphiteConfig() {
@Override
public String host() {
return "mygraphitehost";
}

text

@Override
public String get(String k) {
    return null; // accept the rest of the defaults
}

};

MeterRegistry registry = new GraphiteMeterRegistry(graphiteConfig, Clock.SYSTEM, HierarchicalNameMapper.DEFAULT);

Micrometer uses Dropwizard Metrics as the underlying instrumentation library when recording metrics destined for Graphite. GraphiteConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Spring Boot's Micrometer support binds its application properties directly to the GraphiteConfig. See https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.export.graphite[Spring Boot Reference Documentation] for details.

== Graphite Tag Support

As of Micrometer version 1.4.0, Micrometer supports exporting Graphite metrics by using tags instead of the traditional hierarchical format. By default, metrics are exported by using the tag format, unless any tagsAsPrefix values are configured.
https://graphite.readthedocs.io/en/latest/tags.html[Tag support] was added to Graphite in the 1.1.0 Graphite release.
If you wish to revert to the traditional hierarchical format, ensure that the graphiteTagsEnabled config value is set to false.
The following documentation sections on hierarchical name mapping and metrics prefixing are only applicable if tag support is disabled.

include::hierarchical-name-mapping.adoc[]

== Prefixing Your Metrics

To add a prefix to all metrics that go to graphite, use the GraphiteConfig#tagsAsPrefix configuration option. This option applies the tag value of a set of common tags as a prefix. For example, if tagsAsPrefix contains application, and a meter named myTimer is created with a tag of application=APPNAME, it appears in Graphite as APPNAME.myTimer.

Generally, when you use tagsAsPrefix, you should add common tags to the registry so that the tags are present on all meters that belong to that registry:

[source,java]

@Bean public MeterRegistryCustomizer commonTags() { return r -> r.config().commonTags("application", "APPNAME"); }

We do it this way because, generally, a tag prefix in Graphite is correlated to a common tag elsewhere. Prefixes tend to be something like app name or host. By applying those values as common tags, you make your metrics more portable (if you ever switch to a dimensional monitoring system, you are set).

You can use this when the order of the prefix matters. Micrometer always sorts tags, but the order of tag keys in tagsAsPrefix is preserved, so adding host and application to tagsAsPrefix results in a prefixed metric, such as HOST.APP.myCounter.

To meet your specific naming needs, you can also provide a custom hierarchical name mapper when creating GraphiteMeterRegistry, as follows:

[source,java]

GraphiteMeterRegistry r = new GraphiteMeterRegistry(
GraphiteConfig.DEFAULT,
Clock.SYSTEM,
(id, convention) -> "prefix." + HierarchicalNameMapper.DEFAULT.toHierarchicalName(id, convention));

NOTE: If you use a custom HierarchicalNameMapper, tagsAsPrefix is ignored.

== Further Customizing the GraphiteReporter

We give you the option to configure GraphiteReporter yourself if you need further customization. To do so, use this constructor and provide your own GraphiteReporter:

[source,java]

GraphiteMeterRegistry(GraphiteConfig config, Clock clock, HierarchicalNameMapper nameMapper,
MetricRegistry metricRegistry, GraphiteReporter reporter)

== Graphing

This section serves as a quick start to rendering useful representations in Graphite for metrics originating in Micrometer.

=== Counters

Graphite counters measure mean throughput and one-, five-, and fifteen-minute exponentially-weighted moving average throughputs.

.A Graphite rendered graph of the random walk counter.
image::implementations/graphite-counter.png[Graphite-rendered counter]

Modules/ROOT/Pages/Implementations/Hierarchical Name Mapping

== Hierarchical name mapping

Micrometer provides a HierarchicalNameMapper interface that governs how a dimensional meter ID is mapped to flat hierarchical names.

The default (HierarchicalNameMapper.DEFAULT) sorts tags alphabetically by key and appends tag key/value pairs to the base meter name with '.' -- for example, http_server_requests.method.GET.response.200. The name and tag keys have the registry's naming convention applied to them first.

If there is something special about your naming scheme that you need to honor, you can provide your own HierarchicalNameMapper implementation. The most common cause of a custom mapper comes from a need to prefix something to the front of every metric (generally something like app.<name>.http_server_requests.method.GET.response.200).

Modules/ROOT/Pages/Implementations/Humio

= Micrometer Humio
:sectnums:
:system: humio

Humio is a dimensional time-series SaaS with built-in dashboarding.

include::_install.adoc[]

== Configuring

The following example configures a Humio instance:

[source,java]

HumioConfig humioConfig = new HumioConfig() {
@Override
public String apiToken() {
return MY_TOKEN;
}

text

@Override
public @Nullable String get(String k) {
    return null;
}

}; MeterRegistry registry = new HumioMeterRegistry(humioConfig, Clock.SYSTEM);

HumioConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.humio directly to the HumioConfig:

[source,yml]

management.metrics.export.humio:
api-token: YOURKEY

text

# You will probably want disable Humio publishing in a local development profile.
enabled: true

# The interval at which metrics are sent to Humio. The default is 1 minute.
step: 1m

# The cluster Micrometer will send metrics to. The default is "https://cloud.humio.com"
uri: https://myhumiohost

== Graphing

This section serves as a quick start to rendering useful representations in Humio for metrics originating in Micrometer.

=== Timers

The Humio implementation of Timer produces four fields in Humio:

count: Rate of total time/second.
sum: Rate of calls/second.
max: A sliding window that shows the maximum amount recorded.
avg: A non-aggregable average for only this set of tag values.

The following query constructs a dimensionally aggregable average latency per URI:

[source, text]

name = http_server_requests | timechart(uri, function=max(avg))

.Timer over a simulated service.
image::implementations/humio-timer.png[Humio-rendered timer]

Modules/ROOT/Pages/Implementations/Influx

= Micrometer Influx
:sectnums:
:system: influx

The InfluxData suite of tools supports real-time stream processing and storage of time-series data. It supports downsampling, automatically expiring and deleting unwanted data, as well as backup and restore.

The InfluxMeterRegistry supports the 1.x InfluxDB API as well as the v2 API.

== Installation and Configuration

Micrometer supports shipping metrics to InfluxDB directly or through Telegraf through the StatsD registry.

It is recommended to use the BOM provided by Micrometer (or your framework if any), you can see how to configure it xref:../installing.adoc[here]. The examples below assume you are using a BOM.

=== Direct to InfluxDB

==== Gradle

After the BOM is xref:../installing.adoc[configured], add the following dependency:

[source,groovy]

implementation 'io.micrometer:micrometer-registry-influx'

NOTE: The version is not needed for this dependency since it is defined by the BOM.

==== Maven

After the BOM is xref:../installing.adoc[configured], add the following dependency:

[source,xml]

io.micrometer micrometer-registry-influx ----

NOTE: The version is not needed for this dependency since it is defined by the BOM.

Metrics are rate-aggregated and pushed to InfluxDB on a periodic interval. Rate aggregation performed by the registry yields datasets that are quite similar to those produced by Telegraf. The following example configures a meter registry for InfluxDB:

.InfluxDB 1.x configuration example [source, java]

InfluxConfig config = new InfluxConfig() {
@Override
public Duration step() {
return Duration.ofSeconds(10);
}

text

@Override
public String db() {
    return "mydb";
}

@Override
public String get(String k) {
    return null; // accept the rest of the defaults
}

}; MeterRegistry registry = new InfluxMeterRegistry(config, Clock.SYSTEM);

To ship metrics to InfluxDB 2.x, make sure to configure the org and bucket to which to write the metrics, as well as the authentication token:

.InfluxDB v2 configuration example [source, java]

InfluxConfig config = new InfluxConfig() {

text

@Override
public String org() {
    return "myorg";
}

@Override
public String bucket() {
    return "app-metrics";
}

@Override
public String token() {
    return "auth_token_here"; // FIXME: This should be securely bound rather than hard-coded, of course.
}

@Override
public String get(String k) {
    return null; // accept the rest of the defaults
}

}; MeterRegistry registry = new InfluxMeterRegistry(config, Clock.SYSTEM);

InfluxConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.influx directly to the InfluxConfig:

[source, yaml]

management.metrics.export.influx:
api-version: v2 # API version of InfluxDB to use. Defaults to 'v1' unless an org is configured. If an org is configured, defaults to 'v2'.
auto-create-db: true # Whether to create the InfluxDB database if it does not exist before attempting to publish metrics to it. InfluxDB v1 only. (Default: true)
batch-size: 10000 # Number of measurements per request to use for this backend. If more measurements are found, then multiple requests will be made. (Default: 10000)
bucket: mybucket # Bucket for metrics. Use either the bucket name or ID. Defaults to the value of the db property if not set. InfluxDB v2 only.
compressed: true # Whether to enable GZIP compression of metrics batches published to InfluxDB. (Default: true)
connect-timeout: 1s # Connection timeout for requests to this backend. (Default: 1s)
consistency: one # Write consistency for each point. (Default: one)
db: mydb # Database to send metrics to. InfluxDB v1 only. (Default: mydb)
enabled: true # Whether exporting of metrics to this backend is enabled. (Default: true)
num-threads: 2 # Number of threads to use with the metrics publishing scheduler. (Default: 2)
org: myorg # Org to write metrics to. InfluxDB v2 only.
password: mysecret # Login password of the InfluxDB server. InfluxDB v1 only.
read-timeout: 10s # Read timeout for requests to this backend. (Default: 10s)
retention-policy: my_rp # Retention policy to use (InfluxDB writes to the DEFAULT retention policy if one is not specified). InfluxDB v1 only.
step: 1m # Step size (i.e. reporting frequency) to use. (Default: 1m)
token: AUTH_TOKEN_HERE # Authentication token to use with calls to the InfluxDB backend. For InfluxDB v1, the Bearer scheme is used. For v2, the Token scheme is used.
uri: http://localhost:8086 # URI of the InfluxDB server. (Default: http://localhost:8086)
user-name: myusername # Login user of the InfluxDB server. InfluxDB v1 only.

=== Through Telegraf

Telegraf is a StatsD agent that expects a modified flavor of the StatsD line protocol.

==== Gradle

After the BOM is xref:../installing.adoc[configured], add the following dependency:

[source,groovy]

implementation 'io.micrometer:micrometer-registry-statsd'

NOTE: The version is not needed for this dependency since it is defined by the BOM.

==== Maven

After the BOM is xref:../installing.adoc[configured], add the following dependency:

[source,xml]

io.micrometer micrometer-registry-statsd ----

NOTE: The version is not needed for this dependency since it is defined by the BOM.

Metrics are shipped immediately over UDP to Telegraf by using Telegraf's flavor of the StatsD line protocol:

[source,java]

StatsdConfig config = new StatsdConfig() {
@Override
public String get(String k) {
return null;
}

text

@Override
public StatsdFlavor flavor() {
    return StatsdFlavor.Telegraf;
}

};

MeterRegistry registry = new StatsdMeterRegistry(config, Clock.SYSTEM);

Modules/ROOT/Pages/Implementations/Instana

= Micrometer Instana
Fabian Lange [email protected]
:sectnums:
:system: instana

Instana is an automatic application performance management and infrastructure monitoring system.

== Installation and Configuration

Instana automatically detects and reports all metrics without the need of any additional dependency or configuration.
It does so by detecting all instances of io.micrometer.core.instrument.MeterRegistry and collecting all registered io.micrometer.core.instrument.Meter instances from them.

You can run the Instana agent alongside your application by using Micrometer, and the Instana agent automatically monitors it.

== Supported Metrics

Timer: The total time of recorded events, scaled to milliseconds.
Counter: The cumulative count since this counter was created.
Gauge: The current value.
DistributionSummary: The total number of all recorded events.
LongTaskTimer: The current number of tasks being executed.
FunctionCounter: The cumulative count since this counter was created.
FunctionTimer: The total time of all occurrences of the timed event.
TimeGauge: The current value, scaled to the appropriate base unit.

The metrics show up on the Java Virtual Machine dashboard in Instana. You can configure alerting based on these metrics.

Modules/ROOT/Pages/Implementations/Jmx

= Micrometer JMX
:sectnums:
:system: jmx

Micrometer provides a hierarchical mapping to JMX, primarily as a cheap and portable way to view metrics locally. Where JMX exporting is found in production, the same metrics are generally exported to another, more purpose-fit monitoring system.

include::_install.adoc[]

Micrometer also sometimes scrapes data from JMX beans for use in reporting metrics. This registry implementation is not needed for these uses. Rather, this module is strictly used to export data to JMX.

include::hierarchical-name-mapping.adoc[]

== Counters

JMX counters measure mean throughput and one-, five-, and fifteen-minute exponentially-weighted moving average throughputs.

.The JMX rendered values of the random walk counter.
image::implementations/jmx-counter.png[JMX-rendered counter]

Modules/ROOT/Pages/Implementations/Kairos

= Micrometer KairosDB
:sectnums:
:system: kairos

KairosDB is a dimensional time-series database built on top of Cassandra. Charting can be accomplished in Grafana by using a link:https://docs.grafana.org/v4.0/datasources/kairosdb/[Kairos datasource].

include::_install.adoc[]

== Configuring

The following example configures KairosDB:

[source,java]

KairosConfig kairosConfig = new KairosConfig() {
@Override
public @Nullable String get(String k) {
return null;
}
};

MeterRegistry registry = new KairosMeterRegistry(kairosConfig, Clock.SYSTEM);

KairosConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.kairos directly to the KairosConfig:

[source,yml]

management.metrics.export.kairos:
# You will probably want disable Kairos publishing in a local development profile.
enabled: true

text

# The interval at which metrics are sent to Kairos. The default is 1 minute.
step: 1m

# Authentication may be required, depending on how you have Kairos configured
user-name: MYUSER
password: MYPASSWORD

Modules/ROOT/Pages/Implementations/New Relic

= Micrometer New Relic
:sectnums:
:system: new-relic

New Relic offers a dimensional monitoring system product called Insights. It includes a full UI and a query language called NRQL. New Relic Insights operates on a push model. Some features of NRQL assume that Insights receives a distinct event payload for every timing, count, and so on. Micrometer instead ships aggregates at a prescribed interval, letting your app's throughput scale without concern for event propagation to Insights becoming a bottleneck.

NOTE: New Relic provides its own Micrometer MeterRegistry implementation based on dimensional metrics.
It intends to supersede Micrometer's NewRelicMeterRegistry (which uses custom events in New Relic), because New Relic's dimensional metrics are a better fit for metrics than custom events.
You can find more details in https://github.com/newrelic/micrometer-registry-newrelic[its GitHub repository].

include::_install.adoc[]

== Configuring

The following example configures New Relic:

[source,java]

NewRelicConfig newRelicConfig = new NewRelicConfig() {
@Override
public String accountId() {
return "MYACCOUNT";
}

text

@Override
public String apiKey() {
    return "MY_INSIGHTS_API_KEY";
}

@Override
public String get(String k) {
    return null; // accept the rest of the defaults
}

};

MeterRegistry registry = new NewRelicMeterRegistry(newRelicConfig, Clock.SYSTEM);

There are two distinct sources of API keys in New Relic.

NewRelicConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.newrelic directly to the NewRelicConfig:

[source,yml]

management.metrics.export.newrelic:
account-id: MYACCOUNT
api-key: MY_INSIGHTS_API_KEY

text

# The interval at which metrics are sent to New Relic. See Duration.parse for the expected format.
# The default is 1 minute.
step: 1m

== Graphing

This section serves as a quick start to rendering useful representations in New Relic for metrics originating in Micrometer. See the https://docs.newrelic.com/docs/insights/nrql-new-relic-query-language/using-nrql/introduction-nrql[New Relic NRQL docs] for a far more complete reference of what is possible in New Relic.

=== Timers

At each publishing interval, the New Relic Timer produces a single event with the timer's name and several attributes:

avg: A mean latency for the publishing interval.
count: Throughput per second over the publishing interval.
totalTime: Total time per second over the publishing interval (used with count) to create aggregable means.

Additionally, if any percentiles or SLO buckets are defined on the timer, additional events are produced:

$\{name}.percentiles: Micrometer calculated percentiles for the publishing interval. One event is produced for each percentile, with a tag of phi in the range of [0,1].
$\{name}.histogram: One event is produced for each SLO boundary with a tag of 'le', indicating that it represents a cumulative count of events less than or equal to SLO boundaries over the publishing interval.

To generate an aggregable view of latency in New Relic, divide totalTime by count:

[source,sql]

SELECT sum(totalTime)/sum(count) as 'Average Latency', max(max) as 'Max' FROM timer since 30 minutes ago TIMESERIES auto

.Timer latency.
image::implementations/new-relic-timer-latency.png[New Relic-rendered timer]

The following example generates a throughput chart:

[source,sql]

SELECT average(count) as 'Average Throughput' FROM timer since 30 minutes ago TIMESERIES auto

.Timer throughput.
image::implementations/new-relic-timer-throughput.png[New Relic-rendered timer throughput]

The following example generates a plot of client-side percentiles:

[source,sql]

.Timer Percentiles.
image::implementations/new-relic-timer-percentiles.png[New Relic-rendered percentiles]

Note how these percentiles are not aggregable. We have selected the latest(value) function to display this chart (it is not correct to average(value) on a percentile value). The more dimensions you add to a timer, the less useful these values become.

Finally, if you define SLO boundaries with the fluent builder for Timer, you can view throughput below certain SLO boundaries. In this example, we set SLO boundaries at 275 (yellow), 300 (red), and 500 (blue) milliseconds for a simulated Timer that is recording samples normally distributed around 250 ms. These counts represent the rate/second of samples less than or equal to each SLO boundary.

[source,sql]

.Timer SLO boundaries.
image::implementations/new-relic-timer-sla.png[New Relic-rendered SLO boundaries]

Where the lines converge at various points, it is evident that no sample exceeded the 275 ms SLO boundary.

Modules/ROOT/Pages/Implementations/Otlp

= Micrometer OTLP
:sectnums:
:system: otlp

OpenTelemetry is a CNCF incubating project for providing standards for telemetry data. OpenTelemetry protocol (OTLP) is a vendor-neutral protocol that you can use to send data to various backends that support it. You can read the corresponding docs on how the metrics are ingested and can be visualized in the respective vendor docs.

include::_install.adoc[]

== Configuring
The following example configures an OTLP registry:

[source,java]

OtlpConfig otlpConfig = new OtlpConfig() {
@Override
public String get(final String key) {
return null;
}
};

MeterRegistry registry = new OtlpMeterRegistry(otlpConfig, Clock.SYSTEM);

OtlpConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source (for example, a simple Map can work), you can override the default configuration through properties. For example, Micrometer's Spring Boot support binds properties prefixed with management.otlp.metrics.export directly to the OtlpConfig:

[source, yaml]

management:
otlp:
metrics:
export:
# Supported configs
url: "https://otlp.example.com:4318/v1/metrics"
batchSize: 15000
aggregationTemporality: "cumulative"
compressionMode: "gzip"
headers:
header1: value1
step: 30s
resourceAttributes:
key1: value1

url - The address to which metrics are published. Environment variables OTEL_EXPORTER_OTLP_METRICS_ENDPOINT and OTEL_EXPORTER_OTLP_ENDPOINT are also supported in the default implementation. If a value is not provided, it defaults to http://localhost:4318/v1/metrics
batchSize - number of Meters to include in a single payload sent to the backend. The default is 10,000.
aggregationTemporality - https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality[Aggregation temporality, window=_blank] determines how the additive quantities are expressed, in relation to time. The environment variable OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is supported by the default implementation. The supported values are cumulative or delta. Defaults to cumulative.
compressionMode - Compression mode for metrics payload. The environment variables OTEL_EXPORTER_OTLP_METRICS_COMPRESSION and OTEL_EXPORTER_OTLP_COMPRESSION are supported by the default implementation. If both are set, OTEL_EXPORTER_OTLP_METRICS_COMPRESSION takes precedence. The supported values are gzip or none. Defaults to none.
headers - Additional headers to send with exported metrics. This can be used for authorization headers. By default, headers are loaded from the config. If that is not set, they can be taken from the environment variables OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS. If a header is set in both the environmental variables, the header in the latter overrides the former.
step - the interval at which metrics will be published. The environment variable OTEL_METRIC_EXPORT_INTERVAL is supported by the default implementation. If a value is not provided, defaults to 1 minute.
resourceAttributes - https://opentelemetry.io/docs/specs/otel/resource/semantic_conventions/#service[Resource attributes, window=_blank] are used for all metrics published. By default, Micrometer adds the following resource attributes:

[%autowidth]
|===
|Key | Default value

|telemetry.sdk.name
|io.micrometer

|telemetry.sdk.language
|java

|telemetry.sdk.version
| (e.g.: 1.11.0)

|service.name
|unknown_service
|===

If this config is empty, resource attributes are loaded from the OTEL_RESOURCE_ATTRIBUTES environmental variable. You can override service.name by setting the OTEL_SERVICE_NAME environmental variable, and this takes precedence over other configs.

== Customize metrics sender

The OtlpMeterRegistry has an OtlpMetricsSender abstraction for sending batches of metrics in OTLP protobuf format.

By default, metrics from the OTLP registry are sent via HTTP to the URL specified by OtlpConfig#url using an HttpUrlConnectionSender.
You may use a different HttpSender implementation by creating and configuring an instance of OtlpHttpMetricsSender such as in the following example using the OkHttpSender instead of the default HttpUrlConnectionSender:

[source,java,subs=+attributes]

include::{include-java}/metrics/OtlpMeterRegistryCustomizationTest.java[tags=customizeHttpSender, indent=0]

You can also provide a custom implementation of OtlpMetricsSender that does not use HttpSender at all. OtlpConfig#url will be used as the address when the sender is called in the OtlpMeterRegistry publish method.
For instance, if you made a gRPC implementation, you could configure it in the following way.
Micrometer does not currently provide a gRPC implementation of OtlpMetricsSender.

[source,java,subs=+attributes]

include::{include-java}/metrics/OtlpMeterRegistryCustomizationTest.java[tags=customGrpcSender, indent=0]

== Supported metrics
https://opentelemetry.io/docs/specs/otel/metrics/data-model/#metric-points[Metric points, window=_blank] define the different data points that are supported in OTLP. Micrometer supports exporting the below data points in OTLP format,

https://opentelemetry.io/docs/specs/otel/metrics/data-model/#sums[Sums, window=_blank]
https://opentelemetry.io/docs/specs/otel/metrics/data-model/#gauge[Gauge, window=_blank]
https://opentelemetry.io/docs/specs/otel/metrics/data-model/#histogram[Histogram, window=_blank]
https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram[Exponential Histogram, window=_blank]
https://opentelemetry.io/docs/specs/otel/metrics/data-model/#summary-legacy[Summary, window=_blank]

The following table maps OTLP data points and the Micrometer meters:

[%autowidth]
|===
|OTLP data point | Micrometer meter type

|Sums
|Counter, FunctionCounter

|Gauge
|Gauge, TimeGauge, MultiGauge

|Histogram
|Timer, DistributionSummary, LongTaskTimer, FunctionTimer (only sum and count are set)

|Exponential Histogram^*^
|Timer, DistributionSummary

|Summary
|Timer, DistributionSummary, LongTaskTimer
|===
* - Histogram or Exponential Histogram is determined based on the configuration of the OtlpMeterRegistry. See below section for more info.

Note:

max on the Histogram data point is supported only in delta aggregation temporality. This is because the values represented by cumulative min and max stabilize as more events are recorded and are less useful when recorded over application's lifecycle.
Currently, Micrometer only exports metadata for type Meter to OTLP.

== Histograms and Percentiles
Micrometer Timer and DistributionSummary support configuring xref:../concepts/histogram-quantiles.adoc[client-side percentiles and percentile histograms]. The OTLP specification terms the Summary data point (client-side percentiles) as legacy and not recommended for new applications. The Summary data point also cannot have min/max associated with it. Due to these reasons, Micrometer prefers exporting Timers and DistributionSummary as a Histogram data point. By default, a Timer/DistributionSummary without any additional percentile/histogram config is exported as a Histogram data point. However, by configuring the timer to generate only client-side percentiles, by using publishPercentiles, you can change this to a Summary data point that exports pre-calculated percentiles. When both publishPercentiles and (publishPercentileHistogram or serviceLevelObjectives) are configured, the Histogram data point is preferred and pre-calculated percentiles are not generated. The following table describes which data point is used with different configurations:

[%autowidth]
|===
|Configuration | OTLP data point

| publishPercentiles
| Summary

| publishPercentileHistogram
| Histogram/Exponential Histogram ^*^

| publishPercentiles and publishPercentileHistogram
| Histogram/Exponential Histogram ^*^

| serviceLevelObjectives
| Histogram

| publishPercentiles and serviceLevelObjectives
| Histogram
|===
* The configuration histogramFlavorPerMeter and histogramFlavor determine whether the OTLP DataPoint is a Histogram/Exponential Histogram.

OtlpMeterRegistry supports 2 types of Histogram implementations (1.Explicit Bucket Histogram (or simply called Histogram), 2. Exponential Histogram) when publishPercentileHistogram is configured.
The type is determined by histogramFlavorPerMeter and histogramFlavor in OtlpConfig used by the registry.
When the implementation is the exponential histogram, additional configuration applies:

maxScale used to cap the maximum https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponential-scale[scale, window=_blank] used by the Exponential Histogram (defaults to 20).
maxBuckets determines the maximum number of https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponential-buckets[buckets, window=_blank] to be used for exponential histograms (defaults to 160).
maxBucketsPerMeter overrides maxBuckets for specific meter names, giving more fine-grained configurability.

Since Exponential Histogram cannot have custom SLO's specified, an explicit bucket histogram is used whenever serviceLevelObjectives are configured.

=== Configuration with Spring Boot
If you use Spring Boot, you can use the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.customizing.per-meter-properties[per-meter properties, window=_blank] to configure this behavior.

If you want to generate a Histogram data point for a Timer with name test.timer and default buckets generated by Micrometer, use:

[source,properties]

management.metrics.distribution.percentiles-histogram.test.timer=true

For buckets with customized SLO, use:

[source,properties]

management.metrics.distribution.slo.test.timer=10.0,100.0,500.0,1000.0

Alternatively, if you want to generate Summary data point for a timer with name test.timer and 90th and 99th percentiles, you can use:

[source,properties]

management.metrics.distribution.percentiles.test.timer=0.9,0.99

== Exemplars

Exemplars are metadata that you can attach to the value of your metrics time series.
They can reference data outside of your metrics.
A common use case is cross-referencing tracing information (traceId, spanId) but any arbitrary key-value pairs can be attached.
Exemplars are not tags/dimensions (attributes in OTLP terminology); they will not increase cardinality since they belong to the values of the time series. Using them allows you to correlate metrics with traces/spans (or other concepts).

Micrometer supports exemplars for Counter, Timer, and DistributionSummary.

To enable exemplars, you need to provide an ExemplarContextProvider to the OtlpMeterRegistry using its builder:

[source,java,subs=+attributes]

include::{include-java}/metrics/OtlpMeterRegistryCustomizationTest.java[tags=customizeExemplarContextProvider, indent=0]

The implementation of ExemplarContextProvider is responsible for providing an OtlpExemplarContext which contains the necessary information (traceId, spanId, and arbitrary key-value pairs) to be attached to the exemplar.
If the exemplar information is not available (there is no current Span or the Span is not sampled) or the implementor does not want this particular Span to be used as an exemplar, the ExemplarContextProvider should return null.

NOTE: The traceId and spanId passed to OtlpExemplarContext must be case-insensitive hex-encoded strings.

Modules/ROOT/Pages/Implementations/Prometheus

= Micrometer Prometheus
:sectnums:

Prometheus is a dimensional time series database with a built-in UI, a custom query language, and math operations.
Prometheus is designed to operate on a pull model, periodically scraping metrics from application instances, based on service discovery.

Micrometer uses the Prometheus Java Client under the hood; there are two versions of it and Micrometer supports both. If you want to use the "new" client (1.x), use micrometer-registry-prometheus but if you want to use the "legacy" client (0.x), use micrometer-registry-prometheus-simpleclient.

:system: prometheus
include::_install.adoc[]

:system: prometheus-simpleclient
include::_install.adoc[]

== Configuring

Prometheus expects to scrape or poll individual application instances for metrics.
In addition to creating a Prometheus registry, you also need to expose an HTTP endpoint to Prometheus's scraper.
In a Spring Boot application, a https://docs.spring.io/spring-boot/docs/current/actuator-api/htmlsingle/#prometheus[Prometheus actuator endpoint] is auto-configured in the presence of Spring Boot Actuator.
Otherwise, you can use any JVM-based HTTP server implementation to expose scrape data to Prometheus.

The following example uses the JDK's com.sun.net.httpserver.HttpServer to expose a scrape endpoint:

[source,java]

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);

try {
HttpServer server = HttpServer.create(new InetSocketAddress(8080), 0);
server.createContext("/prometheus", httpExchange -> {
String response = prometheusRegistry.scrape(); <1>
httpExchange.sendResponseHeaders(200, response.getBytes().length);
try (OutputStream os = httpExchange.getResponseBody()) {
os.write(response.getBytes());
}
});

text

new Thread(server::start).start();

}
catch (IOException e) {
throw new RuntimeException(e);

}

<1> The PrometheusMeterRegistry has a scrape() function that knows how to supply the String data necessary for the scrape. All you have to do is wire it to an endpoint.

If you use the "new" client (micrometer-registry-prometheus), you can alternatively use io.prometheus.metrics.exporter.httpserver.HTTPServer, which you can find in io.prometheus:prometheus-metrics-exporter-httpserver (you need to add it as a dependency if you want to use it):

[source,java]

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
HTTPServer.builder()
.port(8080)
.registry(prometheusRegistry.getPrometheusRegistry())
.buildAndStart();

If you use the "legacy" client (micrometer-registry-prometheus-simpleclient), you can alternatively use io.prometheus.client.exporter.HTTPServer, which you can find in io.prometheus:simpleclient_httpserver:

[source,java]

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); // you can set the daemon flag to false if you want the server to block new HTTPServer(new InetSocketAddress(8080), prometheusRegistry.getPrometheusRegistry(), true);

If you use the "new" client (micrometer-registry-prometheus), another alternative can be io.prometheus.metrics.exporter.servlet.jakarta.PrometheusMetricsServlet, which you can find in io.prometheus:prometheus-metrics-exporter-servlet-jakarta in case your app is running in a servlet container (such as Tomcat):

[source,java]

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); HttpServlet servlet = new PrometheusMetricsServlet(prometheusRegistry.getPrometheusRegistry());

If you use the "legacy" client (micrometer-registry-prometheus-simpleclient), another alternative can be io.prometheus.client.exporter.MetricsServlet, which you can find in io.prometheus:simpleclient_servlet in case your app is running in a servlet container (such as Tomcat):

[source,java]

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); HttpServlet servlet = new MetricsServlet(prometheusRegistry.getPrometheusRegistry());

=== Scrape Format

By default, the PrometheusMeterRegistry scrape() method returns the https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format[Prometheus text format].

The https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md[OpenMetrics] format can also be produced.
To specify the format to be returned, you can pass a content type to the scrape method.
For example, to get the OpenMetrics 1.0.0 format scrape, you could use the Content-Type for it, as follows in case of the "new" client (micrometer-registry-prometheus):

[source,java]

String openMetricsScrape = registry.scrape("application/openmetrics-text");

If you use the "legacy" client (micrometer-registry-prometheus-simpleclient), you could use the Prometheus Java client constant for it:

[source,java]

String openMetricsScrape = registry.scrape(TextFormat.CONTENT_TYPE_OPENMETRICS_100);

In Spring Boot applications, the https://docs.spring.io/spring-boot/docs/current/actuator-api/htmlsingle/#prometheus[Prometheus Actuator endpoint] supports scraping in either format, defaulting to the Prometheus text format in the absence of a specific Accept header.

=== The Prometheus Rename Filter

In some cases, Micrometer provides instrumentation that overlaps with the commonly used Prometheus simple client modules but has chosen a different naming scheme for consistency and portability.
If you wish to use the Prometheus "standard" names, add the following filter:

[source,java]

prometheusRegistry.config().meterFilter(new PrometheusRenameFilter());

=== Prometheus Client Properties

If you use the "new" client (`micrometer-registry-prometheus`), you can use some of the properties that the Prometheus Java Client supports, see the https://prometheus.github.io/client_java/config/config/[Prometheus Java Client Config docs]. These properties can be loaded from any source that is supported by the Prometheus Java Client (Properties file, System properties, etc.) or they can be obtained through Micrometer using `PrometheusConfig`: [source,java]

PrometheusConfig config = new PrometheusConfig() {
@Override
public String get(String key) {
return null;
}

text

@Override
public Properties prometheusProperties() {
    Properties properties = new Properties();
    properties.putAll(PrometheusConfig.super.prometheusProperties()); <1>
    properties.setProperty("io.prometheus.exemplars.sampleIntervalMilliseconds", "1"); <2>
    return properties;
}

};

PrometheusMeterRegistry registry = new PrometheusMeterRegistry(config, new PrometheusRegistry(), Clock.SYSTEM);

<1> You can reuse the "default" properties defined in PrometheusConfig.
<2> You can set any property from any property source.

Micrometer passes these properties to the Exporters and the Exemplar Sampler of the Prometheus client, so you can use the https://prometheus.github.io/client_java/config/config/#exporter-properties[exporter], and the https://prometheus.github.io/client_java/config/config/#exemplar-properties[exemplar] properties of the Prometheus Client.

== Graphing

This section serves as a quick start to rendering useful representations in Prometheus for metrics originating in Micrometer.
See the https://prometheus.io/docs/querying/basics[Prometheus docs] for a far more complete reference of what is possible in Prometheus.

=== Grafana Dashboard

There are many third-party Grafana dashboards publicly available on https://grafana.com/grafana/dashboards/?search=micrometer[GrafanaHub].
See an example https://grafana.com/grafana/dashboards/4701-jvm-micrometer/[here].

NOTE: The dashboards are maintained by the community in their external GitHub repositories, so if you have an issue, it should be created in their respective GitHub repository.

=== Counters

The query that generates a graph for the random-walk counter is
rate(counter[10s]).

.A Grafana rendered graph of the random walk counter.
image::implementations/prometheus-counter.png[Grafana-rendered Prometheus counter]

Representing a counter without rate normalization over some time window is rarely useful, as the representation is a function of both the rapidity with which the counter is incremented and the longevity of the service. It is generally most useful to rate-normalize these time series to reason about them. Since Prometheus keeps track of discrete events across all time, it has the advantage of allowing for the selection of an arbitrary time window across which to normalize at query time (for example, rate(counter[10s]) provides a notion of requests per second over 10 second windows). The rate-normalized graph in the preceding image would return back to a value around 55 as soon as the new instance (say on a production deployment) was in service.

.Counter over the same random walk, no rate normalization.
image::implementations/prometheus-counter-norate.png[Grafana-rendered Prometheus counter (no rate)]

In contrast, without rate normalization, the counter drops back to zero on service restart, and the count increases without bound for the duration of the service's uptime.

=== Timers

The Prometheus Timer produces the following time series:

$\{name}_seconds_count: Total number of all calls.
$\{name}_seconds_sum: Total time of all calls.
$\{name}_seconds_max: Maximum recorded value within a configurable time window.

Enabling xref:../concepts/histogram-quantiles.adoc[percentile histograms] adds a $\{name}_seconds_bucket time series; the count, sum, and max series are unchanged.

Again, representing a counter without rate normalization over some time window is rarely useful, as the representation is a function of both the rapidity with which the counter is incremented and the longevity of the service.

Using the following Prometheus queries, we can graph the most commonly used statistics about timers:

Average latency: rate(timer_seconds_sum[10s])/rate(timer_seconds_count[10s])
Throughput (requests per second): rate(timer_seconds_count[10s])

.Timer over a simulated service.
image::implementations/prometheus-timer.png[Grafana-rendered Prometheus timer]

=== Long task timers

Long task timers measure tasks that are still running when Prometheus scrapes the registry.
This differs from a Timer, which records completed events.

The exposed Prometheus series depend on whether publishPercentileHistogram() is enabled:

|===
| Default Prometheus output | With percentile histograms enabled

| $\{name}_seconds_count, $\{name}_seconds_sum, $\{name}_seconds_max
| $\{name}_seconds_gcount, $\{name}_seconds_gsum, $\{name}_seconds_bucket, $\{name}_seconds_max
|===

The time series with xref:../concepts/histogram-quantiles.adoc[percentile histograms] enabled are defined by the https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#gaugehistogram[GaugeHistogram] in the OpenMetrics specification.

With the micrometer-registry-prometheus module, $\{name}_seconds_max is exposed as a separate metric family.
With the micrometer-registry-prometheus-simpleclient module, it is part of the same metric family as the other time series.

A LongTaskTimer only samples tasks that are running at scrape time, so its values return to zero when no tasks are in progress. If you need bucket-based queries such as histogram_quantile() over in-progress task duration, or if alert rules need histogram buckets, enable xref:../concepts/histogram-quantiles.adoc[percentile histograms].

For a serial task, long_task_timer_seconds_sum plots the current cumulative duration of in-progress tasks. In Grafana, we can set an alert threshold at some fixed point.

.Simulated back-to-back long tasks with a fixed alert threshold.
image::implementations/prometheus-long-task-timer.png[Grafana-rendered Prometheus long task timer]

== Warning on same name with different set of tag keys

Prometheus https://prometheus.io/docs/instrumenting/writing_clientlibs/#labels:[strongly discourages] users from creating meters having the same name with a different set of tag keys. You should register meters with the same name with a consistent set of tag keys.

This means that you should avoid the following:

[source,java]

// Please don't do this registry.counter("test", "first", "1").increment(); registry.counter("test", "second", "2").increment();

Instead, you should prefer doing the following: [source,java]

registry.counter("test", "first", "1", "second", "none").increment(); registry.counter("test", "first", "none", "second", "2").increment();

== Exemplars

Exemplars are metadata that you can attach to the value of your time series. They can reference data outside of your metrics. A common use case is storing tracing information (traceId, spanId). Exemplars are not tags/dimensions (labels in Prometheus terminology), they will not increase cardinality since they belong to the values of the time series.

In order to setup Exemplars for PrometheusMeterRegistry, you will need a component that provides you the tracing information. If you use the "new" client (micrometer-registry-prometheus), this component is the io.prometheus.metrics.tracer.common.SpanContext while if you use the "legacy" client (micrometer-registry-prometheus-simpleclient), it is the SpanContextSupplier.

Setting them up are somewhat similar, if you use the "new" client (`micrometer-registry-prometheus`): [source,java]

PrometheusMeterRegistry registry = new PrometheusMeterRegistry(
PrometheusConfig.DEFAULT,
new PrometheusRegistry(),
Clock.SYSTEM,
new MySpanContext() <1>
);
registry.counter("test").increment();

System.out.println(registry.scrape("application/openmetrics-text"));

<1> You need to implement SpanContext (class MySpanContext implements SpanContext { ... }) or use an implementation that already exists.

But if you use the "legacy" client (`micrometer-registry-prometheus-simpleclient`): [source,java]

PrometheusMeterRegistry registry = new PrometheusMeterRegistry(
PrometheusConfig.DEFAULT,
new CollectorRegistry(),
Clock.SYSTEM,
new DefaultExemplarSampler(new MySpanContextSupplier()) <1>
);
registry.counter("test").increment();

System.out.println(registry.scrape(TextFormat.CONTENT_TYPE_OPENMETRICS_100));

<1> You need to implement SpanContextSupplier (class MySpanContextSupplier implements SpanContextSupplier { ... }) or use an implementation that already exists.

If your configuration is correct, you should get something like this, the `# {span_id="321",trace_id="123"} ...` section is the Exemplar right after the value: [source]

TYPE test counter

HELP test

test_total 1.0 # {span_id="321",trace_id="123"} 1.0 1713310767.908

EOF

Exemplars are only supported in the OpenMetrics format (they will not show up in the Prometheus text format). You might need to explicitly ask for the OpenMetrics format, for example: [source,shell]

curl --silent -H 'Accept: application/openmetrics-text; version=1.0.0' localhost:8080/prometheus

Modules/ROOT/Pages/Implementations/SignalFx

= Micrometer SignalFx
:sectnums:
:system: signalfx

CAUTION: This module has been deprecated in favor of the xref:./otlp.adoc[OTLP Registry] because the https://github.com/signalfx/signalfx-java[SignalFX Java client library] that this module depends on has been deprecated.

SignalFx is a dimensional monitoring system SaaS with a full UI that operates on a push model. It has a rich set of alert "detectors".

include::_install.adoc[]

== Configuring

The following example configures SignalFx:

[source,java]

SignalFxConfig signalFxConfig = new SignalFxConfig() {
@Override
public String accessToken() {
return "MYTOKEN";
}

text

@Override
public String get(String k) {
    return null; // accept the rest of the defaults
}

};

MeterRegistry registry = new SignalFxMeterRegistry(signalFxConfig, Clock.SYSTEM);

There are two distinct sources of API keys in SignalFx.

SignalFxConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.signalfx directly to the SignalFxConfig:

[source,yml]

management.metrics.export.signalfx:
access-token: MYTOKEN

text

# The interval at which metrics are sent to Ganglia. See Duration.parse for the expected format.
# The default is 1 minute.
step: 1m

== Graphing

This section serves as a quick start to rendering useful representations in SignalFx for metrics originating in Micrometer. See the https://docs.signalfx.com/en/latest/charts/index.html[SignalFx docs] for a far more complete reference of what is possible in SignalFx.

=== Timers

At each publishing interval, the SignalFx Timer produces several time series in SignalFx:

$\{name}.avg: A mean latency for the publishing interval.
$\{name}.count: Throughput per second over the publishing interval.
$\{name}.totalTime: Total time per second over the publishing interval (used with count) to create aggregable means.
$\{name}.percentiles: Micrometer calculated percentiles for the publishing interval. One time series is produced for each percentile, with a tag of phi in the range of [0,1].
$\{name}.histogram: One event is produced for each SLO boundary with a tag of 'le', indicating that it represents a cumulative count of events less than or equal to SLO boundaries over the publishing interval.

To generate an aggregable view of latency in SignalFx, divide totalTime by count:

image::implementations/signalfx-timer-latency-query.png[SignalFx-aggregable latency query]

This is accomplished by adding signals for $\{name}.totalTime and $\{name}.count, adding a formula that divides them, and hiding the inputs to the formula.

.Timer latency.
image::implementations/signalfx-timer-latency.png[SignalFx-rendered timer]

To generate a throughput chart, use the $\{name}.count signal:

.Timer throughput.
image::implementations/signalfx-timer-throughput.png[SignalFx-rendered timer throughput]

To generate a plot of client-side percentiles, use the $\{name}.percentiles signal:

.Timer Percentiles.
image::implementations/signalfx-timer-percentiles.png[SignalFx-rendered percentiles]

Note that these percentiles are not aggregable. The more dimensions you add to a timer, the less useful these values become.

Finally, if you define SLO boundaries with the fluent builder for Timer, you can view throughput below certain SLO boundaries by using the $\{name}.histogram signal. In this example, we set SLO boundaries at 275 (green), 300 (blue), and 500 (purple) milliseconds for a simulated Timer that is recording samples normally distributed around 250 ms. These counts represent the rate/second of samples less than or equal to each SLO boundary.

.Timer SLO boundaries.
image::implementations/signalfx-timer-sla.png[SignalFx-rendered SLO boundaries]

Where the lines converge at various points, it is evident that no sample exceeded the 275 ms SLO boundary.

Modules/ROOT/Pages/Implementations/Stackdriver

= Micrometer Stackdriver Monitoring
Ray Tsang [email protected]
:sectnums:
:system: stackdriver

Stackdriver Monitoring is a dimensional time-series SaaS with built-in dashboarding.

include::_install.adoc[]

== Configuring

The following example configures Stackdriver:

[source,java]

StackdriverConfig stackdriverConfig = new StackdriverConfig() {
@Override
public String projectId() {
return MY_PROJECT_ID;
}

text

@Override
public String get(String key) {
    return null;
}

}

MeterRegistry registry = StackdriverMeterRegistry.builder(stackdriverConfig).build();

StackdriverConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.stackdriver directly to the StackdriverConfig:

[source,yml]

management.metrics.export.stackdriver:
project-id: MY_PROJECT_ID
resource-type: global

text

# You will probably want to disable Stackdriver Monitoring publishing in a local development profile.
enabled: true

# The interval at which metrics are sent to Stackdriver Monitoring. The default is 1 minute.
step: 1m

For most environments, you need to create and configure credentials to push metrics to Stackdriver Monitoring.
In most cases, you should create a service account with Stackdriver Monitoring permissions and configure a
GOOGLE_APPLICATION_CREDENTIALS environmental variable to the path of the service account key file.
The following example shows how to do so:

[source]

export PROJECT_ID=MY_PROJECT_ID
export APP_NAME=MY_APP

Create a service account

gcloud iam service-accounts create $APP_NAME

Grant the service account Stackdriver Monitoring writer permission

gcloud projects add-iam-policy-binding $PROJECT_ID
--member serviceAccount:$APP_NAME@$PROJECT_ID.iam.gserviceaccount.com
--role roles/monitoring.metricWriter

Create a key JSON file

gcloud iam service-accounts keys create $HOME/$APP_NAME-key.json
--iam-account $APP_NAME@$PROJECT_ID.iam.gserviceaccount.com

Configure GOOGLE_APPLICATION_CREDENTIALS env var

export GOOGLE_APPLICATION_CREDENTIALS=$HOME/$APP_NAME-key.json

When running in managed environments (such as Google App Engine, Google Cloud Run, and Google Cloud Function),
you need not configure this environmental variable. In those environments, a service account is
automatically associated with the application instance. The underlying Stackdriver Monitoring client
library can automatically detect and use those credentials.

== Stackdriver Labels

Micrometer metrics tags are mapped to https://cloud.google.com/monitoring/api/v3/metrics-details#intro-time-series[Stackdriver metrics labels]. With tags and labels, you can further filter or group
by the tag or label. See xref:../concepts/naming.adoc#_tag_naming[Micrometer Concepts] for more information on tags.
The following example filters by tags:

[source,java]

MeterRegistry registry = StackdriverMeterRegistry.builder(stackdriverConfig).build(); registry.config().commonTags("application", "my-application");

You can also configure resource labels with the StackdriverConfig method, resourceLabels. Depending on the configured resourceType, there will be required resource labels. See the documentation on https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource[choosing a monitored resource type].

IMPORTANT: When using Micrometer across multiple applications or instances, it is necessary that Stackdriver labels are unique per application or instance. Otherwise, you will see errors such as One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured. If using a resource type other than global, the resource labels may already make metrics unique per application instance. If not, a common tag with the hostname or platform-provided instance ID may be a good candidate for achieving this.

== Spring Boot

Spring Boot provides auto-configuration for Micrometer's StackdriverMeterRegistry. For more information, see the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-metrics-export-stackdriver[Spring Boot documentation].

You can manually configure or register the StackdriverMeterRegistry.
In addition to using Spring Boot Actuator, make sure you create the StackdriverMeterRegistry bean:

[source,java]

@Bean
StackdriverConfig stackdriverConfig() {
return new StackdriverConfig() {
@Override
public String projectId() {
return MY_PROJECT_ID;
}

text

@Override
    public String get(String key) {
        return null;
    }
}

}

@Bean
StackdriverMeterRegistry meterRegistry(StackdriverConfig stackdriverConfig) {
return StackdriverMeterRegistry.builder(stackdriverConfig).build();

}

You can also use https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html#production-ready-metrics-common-tags[Spring Boot Actuator Common Tags configuration] to configure common tags:

[source]

spring.application.name=my-application management.metrics.tags.application=${spring.application.name}

Modules/ROOT/Pages/Implementations/StatsD

= Micrometer StatsD
:sectnums:
:system: statsd

StatsD is a UDP-based sidecar-driven metrics collection system. The maintainer of the original StatsD line protocol specification is Etsy. Datadog's DogStatsD and Influx's Telegraf each accept a modified version of the line protocol, having each enriched the original specification with dimensionality in different ways.

If you intend to use the Datadog or Telegraf flavors, see the documentation for Micrometer's xref:./datadog.adoc[Datadog] or xref:./influx.adoc[Influx] support.

NOTE: Starting a StatsdMeterRegistry doesn't block waiting for a connection to the backend to be established, so recorded measurements could be lost until the connection is established.

include::_install.adoc[]

== Configuring

This configuration is used to ship metrics to a StatsD agent that is compatible with the original Etsy protocol. Metrics are shipped immediately over UDP to the agent.

[source,java]

StatsdConfig config = new StatsdConfig() {
@Override
public String get(String k) {
return null;
}

text

@Override
public StatsdFlavor flavor() {
    return StatsdFlavor.Etsy;
}

};

MeterRegistry registry = new StatsdMeterRegistry(config, Clock.SYSTEM);

NOTE: You can also configure Telegraf to accept the dogstatsd format. If you use Telegraf, configuring Micrometer to ship Telegraf-formatted StatsD lines eases the requirements of your Telegraf configuration.

StatsdConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with management.metrics.export.statsd directly to the StatsdConfig:

[source,yml]

management.metrics.export.statsd:
flavor: etsy

text

# You will probably want to conditionally disable StatsD publishing in local development.
enabled: true

# The interval at which metrics are sent to StatsD. The default is 1 minute.
step: 1m

== Customizing the Metrics Sink

By default, Micrometer publishes the StatsD line protocol over UDP, as the vast majority of existing StatsD agents are UDP servers. You can fully customize how the line protocol is shipped by modifying the builder for StatsdMeterRegistry:

[source,java]

Consumer lineLogger = line -> logger.info(line); <1>

MeterRegistry registry = StatsdMeterRegistry.builder(StatsdConfig.DEFAULT) <2>
.clock(clock)
.lineSink(lineLogger)
.build();

<1> Define what to do with lines.
<2> The flavor configuration option determines the structure of the line for the default line builder. It has no effect if you override the line builder with a customization.

=== Unix Domain Socket (UDS) Support

The StatsD registry supports sending metrics over Unix domain socket datagrams.

To enable this transport, configure the protocol as uds_datagram and set the host to the absolute path of the socket file.

[NOTE]

The `host` value must be the plain absolute path (for example, `/var/run/datadog/dsd.socket`). Do not prefix it with `unix://`.

Java configuration example:

[source,java]

StatsdConfig config = new StatsdConfig() {
@Override
public String get(String k) {
return null; // accept defaults
}

text

@Override
public StatsdProtocol protocol() {
    return StatsdProtocol.UDS_DATAGRAM;
}

@Override
public String host() {
    return "/var/run/datadog/dsd.socket";
}

};

MeterRegistry registry = new StatsdMeterRegistry(config, Clock.SYSTEM);

=== Using Apache Kafka for Line Sink

You can also use Apache Kafka for line sink, as follows:

[source,java]

Properties properties = new Properties();
properties.setProperty(BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
properties.setProperty(KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
properties.setProperty(VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

Producer<String, String> producer = new KafkaProducer<>(properties);

StatsdMeterRegistry.builder(statsdConfig)
.lineSink((line) -> producer.send(new ProducerRecord<>("my-metrics", line)))
.build();

Now Micrometer produces lines for metrics to the my-metrics topic and you can consume the lines on the topic.

== Customizing the Line Format

The built-in Etsy, dogstatsd, and Telegraf flavors cover most known public StatsD agents, but you can completely customize the line format to satisfy closed, proprietary agents. Again, we use the StatsdMeterRegistry builder to establish a line builder for each ID. Providing an instance of the builder per ID offers you the opportunity to eagerly cache the serialization of the ID's name and tags to optimize the serialization of a StatsD line based on that ID as samples are recorded. The following listing defines a fictional format:

[source,java]

Function<Meter.Id, StatsdLineBuilder> nameAndUnits = id -> new StatsdLineBuilder() {
String name = id.getName() + "/" + (id.getBaseUnit() == null ? "unknown" : id.getBaseUnit());

text

@Override
public String count(long amount, Statistic stat) {
   return name + ":" + amount + "|c";
}

... // implement gauge, histogram, and timing similarly

}

MeterRegistry registry = StatsdMeterRegistry.builder(StatsdConfig.DEFAULT) <1>
.clock(clock)
.lineBuilder(nameAndUnits)
.build();

<1> Because you have taken control of line building, the flavor is ignored.

Modules/ROOT/Pages/Implementations/Wavefront

= Micrometer Wavefront
:sectnums:
:system: wavefront

CAUTION: This module has been deprecated due to https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/release-announcements/0/25153[Wavefront's End of Life Announcement].

Wavefront is a dimensional monitoring system offered as a SaaS with a full UI, custom query language, and advanced math operations. Wavefront operates on a push model. Metrics can be pushed either through a sidecar process running on the same host (called the Wavefront proxy) or directly to the Wavefront API.

include::_install.adoc[]

== Configuring

This section describes how to configure Wavefront when you send data:

[[configuring-directly-to-wavefront]]
=== Directly to Wavefront

The following example configures sending directly to Wavefront:

[source,java]

WavefrontConfig config = new WavefrontConfig() {
@Override
public String uri() {
return "https://longboard.wavefront.com"; <1>
}

text

@Override
public String apiToken() {
    return "MYAPIKEY"; <2>
}

@Override
public String get(String key) {
    return null; <3>
}

}; MeterRegistry registry = new WavefrontMeterRegistry(config, Clock.SYSTEM);

<1> longboard is the name of the co-tenant instance on which most organizations start. Once you reach a sufficient scale, Wavefront may move you
to a dedicated host.
<2> This is required when pushing directly to Wavefront's API.
<3> Accept the rest of the defaults.

[[configuring-through-a-wavefront-proxy-sidecar]]
=== Through a Wavefront Proxy Sidecar

The following example configures sending through a Wavefront proxy sidecar:

[source,java]

MeterRegistry registry = new WavefrontMeterRegistry(WavefrontConfig.DEFAULT_PROXY, Clock.SYSTEM);

The default proxy configuration pushes metrics and histogram distributions to a Wavefront proxy sitting on localhost:2878.

NOTE: If publishing metrics to a Wavefront proxy, the URI must be expressed in the form of proxy://HOST:PORT.

== Graphing

This section serves as a quick start to rendering useful representations in Wavefront for metrics originating in Micrometer. See the https://docs.wavefront.com/query_language_getting_started.html[Wavefront docs] for a far more complete reference of what is possible in Wavefront.

=== Counters

The query that generates a graph for a random-walk counter is rate(ts(counter)).

.A Wavefront rendered graph of the random walk counter.
image::implementations/wavefront-counter-rate.png[Wavefront-rendered counter]

Because Wavefront keeps track of cumulative counts across all time, it has the
advantage of allowing for the selection of a particular time function at query time (for example,
rate(ts(counter)) to compute the per-second rate of change).

=== Timers

The Wavefront Timer produces different time series depending on whether or not
publishPercentileHistogram is enabled.

If publishPercentileHistogram is enabled, the Wavefront Timer produces histogram distributions
that let you query for the latency at any percentile using hs() queries. For example, you can
visualize latency at the 95th percentile (percentile(95, hs(timer.m))) or the 99.9th percentile
(percentile(99.9, hs(timer.m))). For more information on histogram distributions, see
<>, later in this section.

If publishPercentileHistogram is disabled, the Wavefront Timer produces several
time series:

$\{name}.avg: Mean latency across all calls.
$\{name}.count: Total number of all calls.
$\{name}.sum: Total time of all calls.
$\{name}.max: Max latency over the publishing interval.
$\{name}.percentiles: Micrometer-calculated percentiles for the publishing interval. These
cannot be aggregated across dimensions.

You can use these time series to generate a quick view of latency in Wavefront:

.Timer latency.
image::implementations/wavefront-timer-latency.png[Wavefront-rendered timer]

The preceding chart shows the average latency (rate(ts(timer.sum))/rate(ts(timer.count)) in
green), 95th percentile (ts(timer.percentile, phi="0.95") in orange), and max (ts(timer.max)
in blue).

Additionally, rate(ts(timer.count)) represents a rate/second throughput of events being timed:

.Timer throughput.
image::implementations/wavefront-timer-throughput.png[Wavefront-rendered timer throughput]

[[wavefront-histograms]]
=== Wavefront Histograms

Wavefront's histogram implementation stores an actual distribution of metrics, as opposed to single metrics. This lets you apply any percentile and aggregation function on the distribution at query time without having to specify specific percentiles and metrics to keep during metric collection.

Wavefront histogram distributions are collected and reported for any Timer or DistributionSummary that has publishPercentileHistogram enabled.

By default, distributions that are reported to Wavefront get aggregated by the minute, providing you with a histogram distribution for each minute. You also have the option of aggregating by hour or day. You can customize this with the following configuration options:

reportMinuteDistribution: Boolean specifying whether to aggregate by minute. Enabled by default. Metric name in Wavefront has .m suffix.
reportHourDistribution: Boolean specifying whether to aggregate by hour. Disabled by default. Metric name in Wavefront has .h suffix.
reportDayDistribution: Boolean specifying whether to aggregate by day. Disabled by default. Metric name in Wavefront has .d suffix.

If you are sending to a Wavefront proxy, by default, both metrics and histogram distributions are published to the same port: 2878 in the default proxy configuration. If your proxy is configured to listen for histogram distributions on a different port, you can specify the port to which to publish by using the distributionPort configuration option.

You can query histogram distributions in Wavefront by using hs() queries. For example, percentile(98, hs($\{name}.m)) returns the 98th percentile for a particular histogram aggregated over each minute. Each histogram metric name has a suffix (.m, .h, or .d), depending on the histogram's aggregation interval.

See the https://docs.wavefront.com/proxies_histograms.html[Wavefront Histograms documentation] for more information.

Modules/ROOT/Pages/Observation/Components

[[micrometer-observation-components]]
= Observation Components

In this section we will describe main components related to Micrometer Observation.

<<micrometer-observation-context, Observation Context>>
<<micrometer-observation-handler, Observation Handler>>
<<micrometer-observation-default-meter-handler, DefaultMeterObservationHandler>>
<<micrometer-observation-events, Signaling Errors and Arbitrary Events>>
<<micrometer-observation-convention-example, Observation Convention>>
<<micrometer-observation-predicates-filters, Observation Predicates and Filters>>
<<micrometer-observation-annotations, Using Annotations With @Observed and @ObservationKeyValue>>

Micrometer Observation basic flow

Observation through ObservationRegistry gets created with a mutable Observation.Context. On each Micrometer Observation lifecycle action (e.g. start()) a corresponding ObservationHandler method is called (e.g. onStart) with the mutable Observation.Context as argument.

// https://arthursonzogni.com/Diagon/#GraphDAG // ObservationRegistry->Observation // Context->Observation // Observation->Handler [source,subs=+attributes]

┌───────────────────┐┌───────┐ │ObservationRegistry││Context│ └┬──────────────────┘└┬──────┘ ┌▽────────────────────▽┐ │Observation │ └┬─────────────────────┘ ┌▽──────┐ │Handler│ └───────┘

Micrometer Observation detailed flow

// https://arthursonzogni.com/Diagon/#GraphDAG // ObservationRegistry->Observation // Context->Observation // ObservationConvention->Observation // ObservationPredicate->Observation // Observation->Handler // Handler->ObservationFilter [source,subs=+attributes]

┌───────────────────┐┌───────┐┌─────────────────────┐┌────────────────────┐ │ObservationRegistry││Context││ObservationConvention││ObservationPredicate│ └┬──────────────────┘└┬──────┘└┬────────────────────┘└┬───────────────────┘ ┌▽────────────────────▽────────▽──────────────────────▽┐ │Observation │ └┬─────────────────────────────────────────────────────┘ ┌▽──────┐ │Handler│ └┬──────┘ ┌▽────────────────┐ │ObservationFilter│ └─────────────────┘

Observation through ObservationRegistry gets created with a mutable Observation.Context. To allow name and key-value customization, an ObservationConvention can be used instead of direct name setting. List of ObservationPredicate is run to verify if an Observation should be created instead of a no-op version. On each xref:./introduction.adoc[Micrometer Observation lifecycle] action (e.g. start()) a corresponding ObservationHandler method is called (e.g. onStart) with the mutable Observation.Context as argument. On Observation stop, before calling the ObservationHandler onStop methods, list of ObservationFilter is called to optionally further modify the Observation.Context.

[[micrometer-observation-context]]
== Observation.Context

To pass information between the instrumented code and the handler (or between handler methods, such as onStart and onStop), you can use an Observation.Context. An Observation.Context is a Map-like container that can store values for you while your handler can access the data inside the context.

[[micrometer-observation-handler]]
== Observation Handler

Observation Handler allows adding capabilities to existing instrumentations (i.e. you instrument code once and depending on the Observation Handler setup, different actions, such as create spans, metrics, logs will happen). In other words, if you have instrumented code and want to add metrics around it, it's enough for you to register an Observation Handler in the Observation Registry to add that behaviour.

Let's look at the following example of adding a timer behaviour to an existing instrumentation.

A popular way to record Observations is storing the start state in a Timer.Sample instance and stopping it when the event has ended.
Recording such measurements could look like this:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=timer,indent=0]

If you want to have more observation options (such as metrics and tracing -- already included in Micrometer -- plus anything else you will plug in), you need to rewrite that code to use the Observation API.

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=observation,indent=0]

Starting with Micrometer 1.10, you can register "handlers" (ObservationHandler instances) that are notified about the xref:./introduction.adoc[lifecycle event] of an observation (for example, you can run custom code when an observation is started or stopped).
Using this feature lets you add tracing capabilities to your existing metrics instrumentation (see: DefaultTracingObservationHandler). The implementation of these handlers does not need to be tracing related. It is completely up to you how you are going to implement them (for example, you can add logging capabilities).

[[micrometer-observation-handler-example]]
=== ObservationHandler Example

Based on this, we can implement a simple handler that lets the users know about its invocations by printing them out to stdout:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=simple_handler,indent=0]

You need to register the handler to the ObservationRegistry:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=register_handler,indent=0]

You can use the observe method to instrument your codebase:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=instrumenting_code,indent=0]

You can also take full control of the scoping mechanism:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=manual_scoping,indent=0]

[[micrometer-observation-default-meter-handler]]
=== DefaultMeterObservationHandler

Micrometer provides DefaultMeterObservationHandler, an ObservationHandler implementation that automatically creates metrics from Observations.
When registered on an ObservationRegistry, it translates Observation lifecycle events into the following meter types:

Timer (<observation-name>) -- Created on stop. Records the duration between start and stop. Tagged with all low cardinality key values and an error tag (set to the exception class name if an error was recorded, or none otherwise).
LongTaskTimer (<observation-name>.active) -- Created on start. Tracks the duration of in-progress observations. Stopped when the observation is stopped. Only tagged with low cardinality key values that are available at the time of start.
Counter (<observation-name>.<event-name>) -- Incremented when an event is signaled via Observation.event(Event). Tagged with all low cardinality key values available at the time of the event.

NOTE: Only low cardinality key values are used as meter tags. High cardinality key values are not included in meters because they can cause a cardinality explosion in the metrics backend.

WARNING: Since the LongTaskTimer is created in the onStart method, it can only contain tags that are available by that time. Any low cardinality key values added after calling start on the Observation will not be included as tags on the LongTaskTimer.

The following example shows how to register DefaultMeterObservationHandler and the metrics it creates:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=default_meter_handler,indent=0]

If you do not need the LongTaskTimer (for example, to reduce the number of created meters), you can disable it using IgnoredMeters:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=default_meter_handler_ignore_ltt,indent=0]

[[micrometer-observation-events]]
== Signaling Errors and Arbitrary Events

When instrumenting code, we might want to signal that an error happened or signal that an arbitrary event happened. The observation API lets us do so through its error and event methods.

One use-case for signaling an arbitrary event can be attaching annotations to Span for Distributed Tracing, but you can also process them however you want in your own handler, such as emitting log events based on them:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=error_and_event,indent=0]

[[micrometer-observation-convention-example]]
== Observation.ObservationConvention Example

When instrumenting code, we want to provide sensible defaults for tags, but also we want to let users easily change those defaults. An ObservationConvention interface is a description of what tags and name we should create for an Observation.Context:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=tax_example,indent=0]

For a more detailed example, see the full usage example of an instrumentation together with overriding the default tags.

The following example puts the whole code together:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=observation_convention_example,indent=0]

[[micrometer-observation-predicates-filters]]
== Observation Predicates and Filters

To globally disable observations under given conditions, you can use an ObservationPredicate. To mutate the Observation.Context, you can use an ObservationFilter.

To set these, call the ObservationRegistry#observationConfig()#observationPredicate() and ObservationRegistry#observationConfig()#observationFilter() methods, respectively.

The following example uses predicates and filters:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationConfiguringTests.java[tags=predicate_and_filter,indent=0]

[[micrometer-observation-annotations]]
== Using Annotations With @Observed and @ObservationKeyValue

If you have turned on Aspect Oriented Programming (for example, by using org.aspectj:aspectjweaver), you can use the @Observed annotation to create observations. You can put that annotation either on a method to observe it or on a class to observe all the methods in it.

The following example shows an ObservedService that has an annotation on a method:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=observed_service,indent=0]

The following test asserts whether the proper observation gets created when a proxied ObservedService instance gets called:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=observed_aop,indent=0]

Also, you can use @ObservationKeyValue annotation to add key-values via method parameters.

The following example shows an ObservedServiceWithParameter that has an annotation on a method:

[source,java,subs=+attributes]

include::{include-java}/observation/ObservationHandlerTests.java[tags=observed_service_with_parameter,indent=0]

The following test asserts whether the proper observation gets created when a proxied ObservedServiceWithParameter instance gets called:

Modules/ROOT/Pages/Concepts/Counters Copy Section

[source,java] Copy Chapter

[source, java] Copy Chapter

[source,java] Copy Chapter

} Copy Chapter

[source,java] Copy Chapter

} Copy Chapter

[source,java,subs=+attributes] Copy Chapter

include::{include-java}/metrics/CountedAspectTest.java[tags=meter_tag_annotation_handler,indent=0] Copy Chapter

[source,java,subs=+attributes] Copy Chapter

include::{include-java}/metrics/CountedAspectTest.java[tags=interface,indent=0] Copy Chapter

[source,java,subs=+attributes] Copy Chapter

// Example for using multiple @MeterTag annotations on the same parameter // @MeterTags({ @MeterTag(...), @MeterTag(...) }) can be also used include::{include-java}/metrics/CountedAspectTest.java[tags=example_multi_annotations,indent=0] Copy Chapter

[source, java] Copy Chapter

Cache cache = ...; // suppose we have a Guava cache with stats recording on registry.more().counter("evictions", tags, cache, c -> c.stats().evictionCount()); <1> c.stats().evictionCount());')" title="Copy chapter prompt for LLMs"> Copy Chapter

[source, java] Copy Chapter

[source, java] Copy Chapter

Attempting to "re-register" a function-tracking counter will result in a warning like this: [source] Copy Chapter

WARNING: This FunctionCounter has been already registered (MeterId{name='my.fc', tags=[]}), the registration will be ignored. Note that subsequent logs will be logged at debug level. Copy Chapter

Modules/ROOT/Pages/Concepts/Distribution Summaries Copy Section

[source, java] Copy Chapter

DistributionSummary summary = registry.summary("response.size"); Copy Chapter

[source, java] Copy Chapter

[source,java] Copy Chapter

DistributionSummary.builder("my.ratio").scale(100).register(registry) Copy Chapter

[source,java] Copy Chapter

DistributionSummary.builder("my.ratio") .scale(100) .serviceLevelObjectives(70, 80, 90) .register(registry) Copy Chapter

Modules/ROOT/Pages/Concepts/Gauges Copy Section

[source, java] Copy Chapter

List list = registry.gauge("listGauge", Collections.emptyList(), new ArrayList<>(), List::size); <1> List list2 = registry.gaugeCollectionSize("listSize2", Tags.empty(), new ArrayList<>()); <2> Map map = registry.gaugeMapSize("mapGauge", Tags.empty(), new HashMap<>()); Copy Chapter

[source,java] Copy Chapter

// ... elsewhere you can update the value it holds using the object reference myGauge.set(27); myGauge.set(11); Copy Chapter

Attempting to "re-register" a gauge will result in a warning like this: [source] Copy Chapter

WARNING: This Gauge has been already registered (MeterId{name='my.gauge', tags=[]}), the registration will be ignored. Note that subsequent logs will be logged at debug level. Copy Chapter

[source, java] Copy Chapter

[source, java] Copy Chapter

[source, java] Copy Chapter

); Copy Chapter

Modules/ROOT/Pages/Concepts/High Cardinality Tags Detector Copy Section

[source,java,subs=+attributes] Copy Chapter

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_default,indent=0] Copy Chapter

[source,java,subs=+attributes] Copy Chapter

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_factory,indent=0] Copy Chapter

[source,java,subs=+attributes] Copy Chapter

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_builder,indent=0] Copy Chapter

[source,java,subs=+attributes] Copy Chapter

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=one_time_check,indent=0] Copy Chapter

[source,java,subs=+attributes] Copy Chapter

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=custom_consumer_config,indent=0] Copy Chapter

And here's an example implementation: [source,java,subs=+attributes] Copy Chapter

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=custom_consumer,indent=0] Copy Chapter

[source,java] Copy Chapter

registry.config().meterFilter(MeterFilter.ignoreTags("userId")); Copy Chapter

[source,java] Copy Chapter

observation.highCardinalityKeyValue("userId", userId); Copy Chapter

[source,java] Copy Chapter

registry.config().meterFilter(MeterFilter.maximumAllowableTags("http.requests", "uri", 100, MeterFilter.deny())); Copy Chapter

Modules/ROOT/Pages/Concepts/Histogram Quantiles Copy Section

[source,java] Copy Chapter

Timer.builder("my.timer") .publishPercentiles(0.5, 0.95) // median and 95th percentile <1> .publishPercentileHistogram() // <2> .serviceLevelObjectives(Duration.ofMillis(100)) // <3> .minimumExpectedValue(Duration.ofMillis(1)) // <4> .maximumExpectedValue(Duration.ofSeconds(10)) Copy Chapter

[source,java] Copy Chapter

registry.timer("myservice.http.requests").record(..); registry.timer("myservice.db.requests").record(..); Copy Chapter

[source,java] Copy Chapter

Modules/ROOT/Pages/Concepts/Implementations Copy Section

Modules/ROOT/Pages/Concepts/Long Task Timers Copy Section

[source, java] Copy Chapter

} Copy Chapter

[source, java] Copy Chapter

} Copy Chapter

[source, java] Copy Chapter

Modules/ROOT/Pages/Concepts/Meter Convention Copy Section

Modules/ROOT/Pages/Concepts/Meter Filters Copy Section

==== [source, java] Copy Chapter

==== [source, java] Copy Chapter

} Copy Chapter

==== [source, java] Copy Chapter

==== [source, java] Copy Chapter

} Copy Chapter

[IMPORTANT] .`MeterFilter` implementations of `map` should be "static"

==== [source, java] Copy Chapter

Modules/ROOT/Pages/Concepts/Counters

[source,java]

[source, java]

[source,java]

}

[source,java]

}

[source,java,subs=+attributes]

include::{include-java}/metrics/CountedAspectTest.java[tags=meter_tag_annotation_handler,indent=0]

[source,java,subs=+attributes]

include::{include-java}/metrics/CountedAspectTest.java[tags=interface,indent=0]

[source,java,subs=+attributes]

// Example for using multiple @MeterTag annotations on the same parameter // @MeterTags({ @MeterTag(...), @MeterTag(...) }) can be also used include::{include-java}/metrics/CountedAspectTest.java[tags=example_multi_annotations,indent=0]

[source, java]

Cache cache = ...; // suppose we have a Guava cache with stats recording on registry.more().counter("evictions", tags, cache, c -> c.stats().evictionCount()); <1>

[source, java]

[source, java]

Attempting to "re-register" a function-tracking counter will result in a warning like this: [source]

WARNING: This FunctionCounter has been already registered (MeterId{name='my.fc', tags=[]}), the registration will be ignored. Note that subsequent logs will be logged at debug level.

Modules/ROOT/Pages/Concepts/Distribution Summaries

[source, java]

DistributionSummary summary = registry.summary("response.size");

[source, java]

[source,java]

DistributionSummary.builder("my.ratio").scale(100).register(registry)

[source,java]

DistributionSummary.builder("my.ratio") .scale(100) .serviceLevelObjectives(70, 80, 90) .register(registry)

Modules/ROOT/Pages/Concepts/Gauges

[source, java]

List list = registry.gauge("listGauge", Collections.emptyList(), new ArrayList<>(), List::size); <1> List list2 = registry.gaugeCollectionSize("listSize2", Tags.empty(), new ArrayList<>()); <2> Map map = registry.gaugeMapSize("mapGauge", Tags.empty(), new HashMap<>());

[source,java]

// ... elsewhere you can update the value it holds using the object reference myGauge.set(27); myGauge.set(11);

Attempting to "re-register" a gauge will result in a warning like this: [source]

WARNING: This Gauge has been already registered (MeterId{name='my.gauge', tags=[]}), the registration will be ignored. Note that subsequent logs will be logged at debug level.

[source, java]

[source, java]

[source, java]

);

Modules/ROOT/Pages/Concepts/High Cardinality Tags Detector

[source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_default,indent=0]

[source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_factory,indent=0]

[source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_builder,indent=0]

[source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=one_time_check,indent=0]

[source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=custom_consumer_config,indent=0]

And here's an example implementation: [source,java,subs=+attributes]

include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=custom_consumer,indent=0]

[source,java]

registry.config().meterFilter(MeterFilter.ignoreTags("userId"));

[source,java]

observation.highCardinalityKeyValue("userId", userId);

[source,java]

registry.config().meterFilter(MeterFilter.maximumAllowableTags("http.requests", "uri", 100, MeterFilter.deny()));

Modules/ROOT/Pages/Concepts/Histogram Quantiles

[source,java]

Timer.builder("my.timer") .publishPercentiles(0.5, 0.95) // median and 95th percentile <1> .publishPercentileHistogram() // <2> .serviceLevelObjectives(Duration.ofMillis(100)) // <3> .minimumExpectedValue(Duration.ofMillis(1)) // <4> .maximumExpectedValue(Duration.ofSeconds(10))

[source,java]

registry.timer("myservice.http.requests").record(..); registry.timer("myservice.db.requests").record(..);

[source,java]

Modules/ROOT/Pages/Concepts/Implementations

Modules/ROOT/Pages/Concepts/Long Task Timers

[source, java]

}

[source, java]

}

[source, java]

Modules/ROOT/Pages/Concepts/Meter Convention

Modules/ROOT/Pages/Concepts/Meter Filters

==== [source, java]

==== [source, java]

}

==== [source, java]

==== [source, java]

}

==== [source, java]

};