### Modules/ROOT/Pages/Concepts/Counters [[counters]] = Counters Counters report a single metric: a count. The `Counter` interface lets you increment by a fixed amount, which must be positive. TIP: Never count something you can time with a `Timer` or summarize with a `DistributionSummary`! Both `Timer` and `DistributionSummary` always publish a count of events in addition to other measurements. When building graphs and alerts off of counters, you should generally be most interested in measuring the rate at which some event occurs over a given time interval. Consider a simple queue. You could use counters to measure various things, such as the rate at which items are being inserted and removed. It is tempting, at first, to conceive of visualizing absolute counts rather than a rate, but the absolute count is usually both a function of the rapidity with which something is used *and* the longevity of the application instance under instrumentation. Building dashboards and alerts of the rate of a counter per some interval of time disregards the longevity of the app, letting you see aberrant behavior long after the application has started. NOTE: Be sure to read through the timer section before jumping into using counters, as timers record a count of timed events as part of the suite of metrics that go into timing. For those pieces of code you intend to time, you do NOT need to add a separate counter. The following code simulates a real counter whose rate exhibits some perturbation over a short time window: [source,java] ---- Normal rand = ...; // a random generator MeterRegistry registry = ... Counter counter = registry.counter("counter"); <1> Flux.interval(Duration.ofMillis(10)) .doOnEach(d -> { if (rand.nextDouble() + 0.1 > 0) { <2> counter.increment(); <3> } }) .blockLast(); ---- <1> Most counters can be created off of the registry itself with a name and, optionally, a set of tags. <2> A slightly positively biased random walk. <3> This is how you interact with a counter. You could also call `counter.increment(n)` to increment by more than one in a single operation. A fluent builder for counters on the `Counter` interface itself provides access to less frequently used options, such as base units and description. You can register the counter as the last step of its construction by calling `register`: [source, java] ---- Counter counter = Counter .builder("counter") .baseUnit("beans") // optional .description("a description of what this counter does") // optional .tags("region", "test") // optional .register(registry); ---- == The `@Counted` Annotation The `micrometer-core` module contains a `@Counted` annotation that frameworks can use to add counting support to either specific types of methods such as those serving web request endpoints or, more generally, to all methods. Also, an incubating AspectJ aspect is included in `micrometer-core`. You can use it in your application either through compile/load time AspectJ weaving or through framework facilities that interpret AspectJ aspects and proxy targeted methods in some other way, such as Spring AOP. Here is a sample Spring AOP configuration: [source,java] ---- @Configuration public class CountedConfiguration { @Bean public CountedAspect countedAspect(MeterRegistry registry) { return new CountedAspect(registry); } } ---- Applying `CountedAspect` makes `@Counted` usable on any arbitrary method in an AspectJ proxied instance, as the following example shows: [source,java] ---- @Service public class ExampleService { @Counted public void sync() { // @Counted will record the number of executions of this method ... } @Async @Counted public CompletableFuture async() { // @Counted will record the number of executions of this method return CompletableFuture.supplyAsync(...); } } ---- === @MeterTag on Method Parameters To support using the `@MeterTag` annotation on method parameters, you need to configure the `@CountedAspect` to add the `CountedMeterTagAnnotationHandler`. [source,java,subs=+attributes] ----- include::{include-java}/metrics/CountedAspectTest.java[tags=resolvers,indent=0] include::{include-java}/metrics/CountedAspectTest.java[tags=meter_tag_annotation_handler,indent=0] ----- Let's assume that we have the following interface. [source,java,subs=+attributes] ----- include::{include-java}/metrics/CountedAspectTest.java[tags=interface,indent=0] ----- When its implementations would be called with different arguments (remember that the implementation needs to be annotated with `@Counted` annotation too), the following counters would be created: [source,java,subs=+attributes] ----- // Example for returning on the parameter include::{include-java}/metrics/CountedAspectTest.java[tags=example_value_to_string,indent=0] // Example for calling the provided on the parameter include::{include-java}/metrics/CountedAspectTest.java[tags=example_value_resolver,indent=0] // Example for calling the provided include::{include-java}/metrics/CountedAspectTest.java[tags=example_value_spel,indent=0] // Example for using multiple @MeterTag annotations on the same parameter // @MeterTags({ @MeterTag(...), @MeterTag(...) }) can be also used include::{include-java}/metrics/CountedAspectTest.java[tags=example_multi_annotations,indent=0] ----- NOTE: `CountedAspect` doesn't support meta-annotations with `@Counted`. == Function-tracking Counters Micrometer also provides a more infrequently used counter pattern that tracks a monotonically increasing function (a function that stays the same or increases over time but never decreases). Some monitoring systems, such as Prometheus, push cumulative values for counters to the backend, but others publish the rate at which a counter is incrementing over the push interval. By employing this pattern, you let the Micrometer implementation for your monitoring system choose whether to rate-normalize the counter, and your counter remains portable across different types of monitoring systems. [source, java] ----- Cache cache = ...; // suppose we have a Guava cache with stats recording on registry.more().counter("evictions", tags, cache, c -> c.stats().evictionCount()); <1> ----- <1> `evictionCount()` is a monotonically increasing function that increments with every cache eviction from the beginning of its life. The function-tracking counter, in concert with the monitoring system's rate normalizing functionality (whether this is an artifact of the query language or the way data is pushed to the system), adds a layer of richness on top of the cumulative value of the function itself. You can reason about the _rate_ at which the value is increasing, whether that rate is within an acceptable bound, is increasing or decreasing over time, and so on. WARNING: Micrometer cannot guarantee the monotonicity of the function for you. By using this signature, you assert its monotonicity based on what you know about its definition. A fluent builder for function counters on the `FunctionCounter` interface itself provides access to less frequently used options, such as base units and description. You can register the counter as the last step of its construction by calling `register(MeterRegistry)`: [source, java] ---- MyCounterState state = ...; FunctionCounter counter = FunctionCounter .builder("counter", state, state -> state.count()) .baseUnit("beans") // optional .description("a description of what this counter does") // optional .tags("region", "test") // optional .register(registry); ---- If the function tracks a time value, you can specify the time unit of the value returned by the function when calling `FunctionCounter.builder(...)`. This creates a `FunctionCounter` that automatically scales its time-based count to the registry's base time unit. This is analogous to how `TimeGauge` works, and provides a way to register monotonically increasing time values (e.g. `RuntimeMXBean::getUptime`) in a way that automatically scales to the preferred time unit of the metrics backend. [source, java] ---- MyCounterState state = ...; FunctionCounter.builder("counter.time", state, state -> state.totalTime(), TimeUnit.NANOSECONDS) .description("a description of what this counter does") // optional .tags("region", "test") // optional .register(registry); ---- WARNING: Attempting to construct a function-tracking counter with a primitive number or one of its `java.lang` object forms is always incorrect. These numbers are immutable. Thus, the function-tracking counter cannot ever be changed. Attempting to "re-register" the function-tracking counter with a different number does not work, as the registry maintains only one meter for each unique combination of name and tags. "Re-registering" a function-tracking counter can happen indirectly for example as the result of a `MeterFilter` modifying the name and/or the tags of two different function-tracking counters so that they will be the same after the filter is applied. Attempting to "re-register" a function-tracking counter will result in a warning like this: [source] ---- WARNING: This FunctionCounter has been already registered (MeterId{name='my.fc', tags=[]}), the registration will be ignored. Note that subsequent logs will be logged at debug level. ---- --- ### Modules/ROOT/Pages/Concepts/Distribution Summaries [[distribution-summaries]] = Distribution Summaries A distribution summary tracks the distribution of events. It is structurally similar to a timer but records values that do not represent a unit of time. For example, you could use a distribution summary to measure the payload sizes of requests hitting a server. The following example creates a distribution summary: [source, java] ---- DistributionSummary summary = registry.summary("response.size"); ---- The interface contains a fluent builder for distribution summaries: [source, java] ---- DistributionSummary summary = DistributionSummary .builder("response.size") .description("a description of what this summary does") // optional .baseUnit("bytes") // optional <1> .tags("region", "test") // optional .scale(100) // optional <2> .register(registry); ---- <1> Add base units for maximum portability. Base units are part of the naming convention for some monitoring systems. Leaving it off and violating the naming convention has no adverse effect if you forget. <2> Optionally, you can provide a scaling factor by which each recorded sample is multiplied as it is recorded. NOTE: The maximum (which is named `max`) for basic `DistributionSummary` implementations, such as `CumulativeDistributionSummary` and `StepDistributionSummary`, is a time window maximum (`TimeWindowMax`). It means that its value is the maximum value during a time window. If no new values are recorded for the time window length, the maximum is reset to 0 as a new time window starts. The time window size until values are fully expired is the `expiry` multiplied by the `bufferLength` in `DistributionStatisticConfig`. `expiry` defaults to the step size of the meter registry unless it's explicitly set to a different value, and `bufferLength` defaults to `3`. A time window max is used to capture the maximum latency in a subsequent interval after heavy resource pressure triggers the latency and prevents metrics from being published. Percentiles are also time window percentiles (`TimeWindowPercentileHistogram`). == Scaling and Histograms Micrometer's preselected percentile histogram buckets are all integers from 1 to `Long.MAX_VALUE`. Currently, `minimumExpectedValue` and `maximumExpectedValue` serve to control the cardinality of the bucket set. If we try to detect that your min/max yields a small range and scale the preselected bucket domain to your summary's range, we do not have another lever to control bucket cardinality. Instead, if your summary's domain is more constrained, scale your summary's range by a fixed factor. The use case we have heard so far is for summaries of ratios whose domain is [0,1]. Given that scenario, we can use the following code to create values from 0 to 100: [source,java] ---- DistributionSummary.builder("my.ratio").scale(100).register(registry) ---- This way, the ratio winds up in the range [0,100] and we can set `maximumExpectedValue` to 100. You can pair this with custom SLO boundaries if you care about particular ratios: [source,java] ---- DistributionSummary.builder("my.ratio") .scale(100) .serviceLevelObjectives(70, 80, 90) .register(registry) ---- == Memory Footprint Estimation The total memory footprint of a distribution summary can vary dramatically, depending on which options you choose. The following table of memory consumption is based on the use of various features. These figures assume no tags and a ring buffer length of 3. Adding tags adds somewhat to the total, as does increasing the buffer length. Total storage can also vary somewhat depending on the registry implementation. * R = Ring buffer length. We assume the default of 3 in all examples. R is set with `DistributionSummary.Builder#distributionStatisticBufferLength`. * B = Total histogram buckets. It can be SLO boundaries or percentile histogram buckets. By default, summaries have NO minimum and maximum expected value, so we ship all 276 predetermined histogram buckets. You should always clamp distribution summaries with a `minimumExpectedValue` and `maximumExpectedValue` when you intend to ship percentile histograms. * M = Time-decaying max. 104 bytes. * Fb = Fixed boundary histogram. 8b * B * R. * Pp = Percentile precision. By default, it is 1. It is generally in the range of [0, 3]. Pp is set with `DistributionSummary.Builder#percentilePrecision`. * Hdr(Pp) = High dynamic range histogram. - When Pp = 0: 1.9kb * R + 0.8kb - When Pp = 1: 3.8kb * R + 1.1kb - When Pp = 2: 18.2kb * R + 4.7kb - When Pp = 3: 66kb * R + 33kb [width="80%",options="header"] |========================================================= |Client-side percentiles |Histogram and/or SLOs |Formula | Example |No |No |M| ~0.1kb |No |Yes |M + Fb|For percentile histogram clamped to 66 buckets, ~6kb |Yes |Yes |M + Hdr(Pp)|For the addition of a 0.95 percentile with defaults otherwise, ~12.6kb |========================================================= NOTE: For Prometheus, R is _always_ equal to 1, regardless of how you attempt to configure it through `DistributionSummary.Builder`. This special case exists for Prometheus because it expects cumulative histogram data that never rolls over. --- ### Modules/ROOT/Pages/Concepts/Gauges [[gauges]] = Gauges A gauge is a handle to get the current value. Typical examples for gauges would be the size of a collection or map or the number of threads in a running state. TIP: Gauges are useful for monitoring things with natural upper bounds. We do not recommend using a gauge to monitor things like request count, as they can grow without bound for the duration of an application instance's life. TIP: Never gauge something you can count with a `Counter`! Micrometer takes the stance that gauges should be sampled and not be set, so there is no information about what might have occurred between samples. Any intermediate values set on a gauge are lost by the time the gauge value is reported to a metrics backend, so there is little value in setting those intermediate values in the first place. Think of a `Gauge` as a "`heisen-gauge`": a meter that changes only when it is observed. Every other meter type accumulates intermediate counts toward the point where the data is sent to the metrics backend. The `MeterRegistry` interface contains methods for building gauges to observe numeric values, functions, collections, and maps: [source, java] ---- List list = registry.gauge("listGauge", Collections.emptyList(), new ArrayList<>(), List::size); <1> List list2 = registry.gaugeCollectionSize("listSize2", Tags.empty(), new ArrayList<>()); <2> Map map = registry.gaugeMapSize("mapGauge", Tags.empty(), new HashMap<>()); ---- <1> A slightly more common form of gauge is one that monitors some non-numeric object. The last argument establishes the function that is used to determine the value of the gauge when the gauge is observed. <2> A more convenient form of (1) for when you want to monitor collection size. All of the different forms of creating a gauge maintain only a _weak reference_ to the object being observed, so as not to prevent garbage collection of the object. == Manually Incrementing or Decrementing a Gauge A gauge can be made to track any `java.lang.Number` subtype that is settable, such as `AtomicInteger` and `AtomicLong` found in `java.util.concurrent.atomic` and similar types, such as Guava's `AtomicDouble`: [source,java] ---- // maintain a reference to myGauge AtomicInteger myGauge = registry.gauge("numberGauge", new AtomicInteger(0)); // ... elsewhere you can update the value it holds using the object reference myGauge.set(27); myGauge.set(11); ---- Note that, in this form, unlike other meter types, you do not get a reference to the `Gauge` when creating one. Rather, you get a reference to the thing being observed. This is because of the "`heisen-gauge`" principal: The gauge is self-sufficient once created, so you should never need to interact with it. This lets us give you back only the instrumented object, which allows for quick one-liners that both create the object to be observed and set up metrics around it. This pattern should be less common than the `DoubleFunction` form. Remember that frequent setting of the observed `Number` results in a lot of intermediate values that never get published. Only the _instantaneous value_ of the gauge at publish time is ever sent to the monitoring system. WARNING: Attempting to construct a gauge with a primitive number or one of its `java.lang` object forms is always incorrect. These numbers are immutable. Thus, the gauge cannot ever be changed. Attempting to "re-register" the gauge with a different number does not work, as the registry maintains only one meter for each unique combination of name and tags. "Re-registering" a gauge can happen indirectly for example as the result of a `MeterFilter` modifying the name and/or the tags of two different gauges so that they will be the same after the filter is applied. Attempting to "re-register" a gauge will result in a warning like this: [source] ---- WARNING: This Gauge has been already registered (MeterId{name='my.gauge', tags=[]}), the registration will be ignored. Note that subsequent logs will be logged at debug level. ---- == Gauge Fluent Builder The interface contains a fluent builder for gauges: [source, java] ---- Gauge gauge = Gauge .builder("gauge", myObj, myObj::gaugeValue) .description("a description of what this gauge does") // optional .tags("region", "test") // optional .register(registry); ---- Generally the returned `Gauge` instance is not useful except in testing, as the gauge is already set up to track a value automatically upon registration. == Why is My Gauge Reporting NaN or Disappearing? It is your responsibility to hold a strong reference to the state object that you are measuring with a `Gauge`. Micrometer is careful to not create strong references to objects that would otherwise be garbage collected. Once the object being gauged is de-referenced and is garbage collected, Micrometer starts reporting a NaN or nothing for a gauge, depending on the registry implementation. If you see your gauge reporting for a few minutes and then disappearing or reporting NaN, it almost certainly suggests that the underlying object being gauged has been garbage collected. == `TimeGauge` `TimeGauge` is a specialized gauge that tracks a time value, to be scaled to the base unit of time expected by each registry implementation. `TimeGauge` can be registered with `TimeUnit` as follows: [source, java] ---- AtomicInteger msTimeGauge = new AtomicInteger(4000); AtomicInteger usTimeGauge = new AtomicInteger(4000); TimeGauge.builder("my.gauge", msTimeGauge, TimeUnit.MILLISECONDS, AtomicInteger::get).register(registry); TimeGauge.builder("my.other.gauge", usTimeGauge, TimeUnit.MICROSECONDS, AtomicInteger::get).register(registry); ---- And for example, if the registry is Prometheus, they will be converted in seconds as follows: ``` # HELP my_gauge_seconds # TYPE my_gauge_seconds gauge my_gauge_seconds 4.0 # HELP my_other_gauge_seconds # TYPE my_other_gauge_seconds gauge my_other_gauge_seconds 0.004 ``` == Multi-gauge Micrometer supports one last special type of `Gauge`, called a `MultiGauge`, to help manage gauging a growing or shrinking list of criteria. This feature lets you select a set of well-bounded but slightly changing set of criteria from something like an SQL query and report each `Row` (see the example below) as a `Gauge`. The following example creates a `MultiGauge`: [source, java] ---- // SELECT count(*) from job group by status WHERE job = 'dirty' MultiGauge statuses = MultiGauge.builder("statuses") .tag("job", "dirty") .description("The number of widgets in various statuses") .baseUnit("widgets") .register(registry); ... // run this whenever you re-run your query statuses.register( resultSet.stream() .map(result -> Row.of(Tags.of("status", result.getAsString("status")), result.getAsInt("count"))) .collect(toList()), true // whether to overwrite the previous value or only record it once ); ---- The `overwrite` argument controls how rows with the same set of tags are handled on subsequent registrations. Use `overwrite=true` when each registration represents a new snapshot and previously registered rows should be replaced with the latest values. Use `overwrite=false` when existing rows should remain unchanged on later registrations with the same tags. Rows that are no longer present in the new registration are removed. --- ### Modules/ROOT/Pages/Concepts/High Cardinality Tags Detector [[high-cardinality-tags-detector]] = High Cardinality Tags Detector High cardinality tags can cause memory and performance issues in your application and your metrics backend. When tag values are unbounded (for example userID, requestID, traceID), each unique combination creates a new Meter, potentially leading to millions of time series and excessive memory consumption. `HighCardinalityTagsDetector` helps you to identify Meters that may have high cardinality tags by monitoring the number of Meters that share the same name. When the count exceeds a configurable threshold, it indicates that the `Meter` likely has high cardinality tags. NOTE: `HighCardinalityTagsDetector` detects potential high cardinality tags by counting Meters with the same name. It does not detect other issues, such as appending random values to Meter names. Its sole purpose is detecting the potential presence of high cardinality tags. == Understand High Cardinality High cardinality occurs when a tag has an unbounded or large number of possible values that does not fit in memory. Common examples include: - UserID, Email, RequestID, SessionID, TraceID - Timestamps - Full URLs (`/users/123`) - Any user input that is not validated/normalized In contrast, low cardinality tags have a bounded, typically "small" set of values: - HTTP methods (`method=GET`) - HTTP status codes (`status=200`) - Application names (`application=payments-app`) - Environment names (`env=prod`) - Templated URLs (`/users/\{id}`) TIP: For more information on tag naming best practices, see xref:./naming.adoc#_tag_naming[Tag Naming]. == How It Works The detector works by: 1. Counting how many meters exist with the same name in the registry 2. Comparing this count against a threshold 3. Notifying you (via logging or custom consumer) when the threshold is exceeded == Usage You can configure the detector through the `MeterRegistry`. In this case, the detector is automatically started and managed by the registry: === Configure with the defaults [source,java,subs=+attributes] ---- include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_default,indent=0] ---- === Configure with a Factory You can provide your own `Function` that receives a registry and returns a detector: [source,java,subs=+attributes] ---- include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_factory,indent=0] ---- === Configure with a Builder `HighCardinalityTagsDetector.Builder` can be used to conveniently set parameters: [source,java,subs=+attributes] ---- include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=registry_integration_builder,indent=0] ---- === One-Time Check You can perform a one-time check to see if any meters exceed the threshold: [source,java,subs=+attributes] ---- include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=one_time_check,indent=0] ---- This approach can be useful for: - Ad-hoc investigation of potential issues - Tests to verify your instrumentation === Scheduled Monitoring For continuous monitoring in production, you don't need to create a scheduled job that periodically calls the detector. The detector has a built in scheduler, but you don't need to manage that one either (by calling `start()` and `close()`). Instead you can configure the detector through the `MeterRegistry` so that the detector is automatically started and managed by the registry for you. See the various configuration options with the registry above. === Custom Consumer Instead of the default logging behavior, you can provide a `Consumer` to handle high cardinality notifications: [source,java,subs=+attributes] ---- include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=custom_consumer_config,indent=0] ---- And here's an example implementation: [source,java,subs=+attributes] ---- include::{include-java}/metrics/HighCardinalityTagsDetectorTests.java[tags=custom_consumer,indent=0] ---- This can be useful for: - Custom logging or reporting - Publishing alerts - Collecting metrics about high cardinality detections - Triggering any kind of custom actions == Preventing High Cardinality When the detector identifies high cardinality tags, consider the following possible solutions: === Remove Problematic Tags If a tag provides little value but high cardinality, you should remove it. If you control the instrumentation, you should update it to not add the high cardinality tag. If you don't control the instrumentation, you can remove the tag using a `MeterFilter`: [source,java] ---- registry.config().meterFilter(MeterFilter.ignoreTags("userId")); ---- See xref:./meter-filters.adoc[Meter Filters] for more information and configuration options. === Normalize Tag Values Instead of using "raw" values, normalize them to reduce cardinality: - Use templated URLs instead of actual URLs: `/users/\{id}` instead of `/users/123` - Group values into ranges: `age.group=20-30` instead of `age=25` - Use categories: `outcome=CLIENT_ERROR` (or `status=4xx`) instead of `status=404` - Limit unknown/unexpected values to known values: `status=OTHER` NOTE: `age` and `status` are not necessarily high cardinality data since they are usually fall into a finite set of values (except if input is not validated and `age=12345` and `status=54321` are possible to send to the app). The normalization techniques to reduce cardinality are still valid though. === Use High Cardinality data with the Observation API If you are using Micrometer's Observation API, you can mark certain metadata as high cardinality. These key-values should not be used for recording metrics. Typically they are only used in outputs that can handle high cardinality (for example: distributed tracing systems, logs): [source,java] ---- observation.highCardinalityKeyValue("userId", userId); ---- See xref:../observation/introduction.adoc[Observation API] for more information. === Use MeterFilters to Set Bounds If none of the above works, you can use `maximumAllowableTags` and `maximumAllowableMetrics` as a last resort to protect your application and your monitoring system. They are for situations in which you cannot fix the cardinality problem, for example with an HTTP client instrumentation that does not offer a way to tag the URI with a template (low cardinality) instead of the expanded URI (high cardinality). You can use `MeterFilter` to prevent high cardinality before it becomes a problem: [source,java] ---- registry.config().meterFilter(MeterFilter.maximumAllowableTags("http.requests", "uri", 100, MeterFilter.deny())); ---- TIP: In the code above, instead of `MeterFilter.deny()`, you can implement your own logic that not only denies the registration of the `Meter` but also logs the event out. These logs can be overwhelming, it's a good idea to limit them. See xref:./meter-filters.adoc[Meter Filters] for more information. == Best Practices 1. **Enable in Production**: Run the detector in production to catch issues that may not appear in testing. 2. **Monitor Regularly**: Use the scheduled monitoring mode rather than one-time checks. 3. **Act on Warnings**: When high cardinality is detected, investigate and fix the root cause rather than just increasing the threshold. 4. **Set Appropriate Thresholds**: The default threshold might work for most applications, but adjust it based on your monitoring system's capabilities and your application's scale as needed. 5. **Test Your Instrumentation**: Use the detector in integration tests to verify that your custom instrumentation doesn't introduce high cardinality. == Limitations * The detector only identifies high cardinality by counting Meters with the same name. It cannot detect: ** Random values appended to `Meter` names ** Memory leaks from other sources ** High cardinality that hasn't yet exceeded the threshold * The detector examines the first meter that exceeds the threshold. If multiple meters have high cardinality, you'll need to fix them iteratively. * The threshold is a heuristic based on memory and may need adjustment for your specific use case. == See Also - xref:./meter-filters.adoc[Meter Filters]: Using filters to control meter registration and tag cardinality - xref:../observation/introduction.adoc[Observation API]: Using high- and low cardinality tags with Observations --- ### Modules/ROOT/Pages/Concepts/Histogram Quantiles [[histograms]] = Histograms and Percentiles Timers and distribution summaries support collecting data to observe their percentile distributions. There are two main approaches to viewing percentiles: * *Percentile histograms*: Micrometer accumulates values to an underlying histogram and ships a predetermined set of buckets to the monitoring system. The monitoring system's query language is responsible for calculating percentiles off of this histogram. Currently, only Prometheus, Atlas, and Wavefront support histogram-based percentile approximations, through `histogram_quantile`, `:percentile`, and `hs()`, respectively. If you target Prometheus, Atlas, or Wavefront, prefer this approach, since you can aggregate the histograms across dimensions (by summing the values of the buckets across a set of dimensions) and derive an aggregable percentile from the histogram. * *Client-side percentiles*: Micrometer computes a percentile approximation for each meter ID (set of name and tags) and ships the percentile value to the monitoring system. This is not as flexible as a percentile histogram because it is not possible to aggregate percentile approximations across tags. Nevertheless, it provides some level of insight into percentile distributions for monitoring systems that do not support server-side percentile calculation based on a histogram. The following example builds a timer with a histogram: [source,java] ---- Timer.builder("my.timer") .publishPercentiles(0.5, 0.95) // median and 95th percentile <1> .publishPercentileHistogram() // <2> .serviceLevelObjectives(Duration.ofMillis(100)) // <3> .minimumExpectedValue(Duration.ofMillis(1)) // <4> .maximumExpectedValue(Duration.ofSeconds(10)) ---- <1> `publishPercentiles`: Used to publish percentile values computed in your application. These values are non-aggregable across dimensions. <2> `publishPercentileHistogram`: Used to publish a histogram suitable for computing aggregable (across dimensions) percentile approximations in Prometheus (by using `histogram_quantile`), Atlas (by using `:percentile`), and Wavefront (by using `hs()`). For Prometheus and Atlas, the buckets in the resulting histogram are preset by Micrometer based on a generator that has been determined empirically by Netflix to yield a reasonable error bound on most real world timers and distribution summaries. By default, the generator yields 276 buckets, but Micrometer includes only those that are within the range set by `minimumExpectedValue` and `maximumExpectedValue`, inclusive. Micrometer clamps timers by default to a range of 1 millisecond to 1 minute, yielding 73 histogram buckets per timer dimension. `publishPercentileHistogram` has no effect on systems that do not support aggregable percentile approximations. No histogram is shipped for these systems. <3> `serviceLevelObjectives`: Used to publish a cumulative histogram with buckets defined by your SLOs. When used in concert with `publishPercentileHistogram` on a monitoring system that supports aggregable percentiles, this setting adds additional buckets to the published histogram. When used on a system that does not support aggregable percentiles, this setting causes a histogram to be published with only these buckets. <4> `minimumExpectedValue`/`maximumExpectedValue`: Controls the number of buckets shipped by `publishPercentileHistogram` and controls the accuracy and memory footprint of the underlying HdrHistogram structure. NOTE: For those monitoring systems, where percentiles can be approximated using the histogram (see `publishPercentileHistogram` above), it is usually unnecessary to also publish client-side percentiles (`publishPercentiles`) since in those scenarios client-side percentiles are redundant and also non-aggregable across dimensions (unlike histograms). Prometheus Java Client (1.x) does not support having both under the same metric name. Since shipping percentiles to the monitoring system generates additional time series, it is generally preferable to *not* configure them in core libraries that are included as dependencies in applications. Instead, applications can turn on this behavior for some set of timers and distribution summaries by using a meter filter. For example, suppose we have a handful of timers in a common library. We have prefixed these timer names with `myservice`: [source,java] ---- registry.timer("myservice.http.requests").record(..); registry.timer("myservice.db.requests").record(..); ---- We can turn on client-side percentiles for both timers by using a meter filter: [source,java] ---- registry.config().meterFilter( new MeterFilter() { @Override public DistributionStatisticConfig configure(Meter.Id id, DistributionStatisticConfig config) { if(id.getName().startsWith("myservice")) { return DistributionStatisticConfig.builder() .percentiles(0.95) .build() .merge(config); } return config; } }); ---- --- ### Modules/ROOT/Pages/Concepts/Implementations [[monitoring-systems]] = Supported Monitoring Systems Micrometer contains a core module with an instrumentation https://en.wikipedia.org/wiki/Service_provider_interface[SPI], a set of modules containing implementations for various monitoring systems (each is called a registry), and a test kit. You need to understand three important characteristics of monitoring systems: * *Dimensionality*. Whether the system supports metric names to be enriched with tag key/value pairs. If a system is not _dimensional_, it is _hierarchical_, which means it supports only a flat metric name. When publishing metrics to hierarchical systems, Micrometer flattens the set of tag key/value pairs and adds them to the name. [cols=2*,options="header"] |=== |Dimensional |Hierarchical |AppOptics, Atlas, Azure Monitor, Cloudwatch, Datadog, Datadog StatsD, Dynatrace, Elastic, Humio, Influx, KairosDB, New Relic, Prometheus, SignalFx, Sysdig StatsD, Telegraf StatsD, Wavefront |Graphite, Ganglia, JMX, Etsy StatsD |=== * *xref:./rate-aggregation.adoc[Rate Aggregation]*. In this context, we mean aggregation of a set of samples over a prescribed time interval. Some monitoring systems expect some types of discrete samples (such as counts) to be converted to a rate by the application prior to being published. Other systems expect cumulative values to always be sent. Still others have no opinion on it either way. [cols=2*,options="header"] |=== |Client-side |Server-side |AppOptics, Atlas, Azure Monitor, Datadog, Dynatrace, Elastic, Graphite, Ganglia, Humio, Influx, JMX, Kairos, New Relic, all StatsD flavors, SignalFx |Prometheus, Wavefront footnote:[As of 1.2.0, Micrometer sends cumulative values to Wavefront.] |=== * *Publishing*. Some systems expect to poll applications for metrics at their leisure, while others expect metrics to be pushed to them on a regular interval. [cols=2*,options="header"] |=== |Client pushes |Server polls |AppOptics, Atlas, Azure Monitor, Datadog, Dynatrace, Elastic, Graphite, Ganglia, Humio, Influx, JMX, Kairos, New Relic, SignalFx, Wavefront |Prometheus, all StatsD flavors |=== There are other, more minor, variations in expectations from one monitoring system to another, such as their conception of base units of measurement (particularly time) and the canonical naming convention for metrics. Micrometer customizes your metrics to meet these demands on a per-registry basis. --- ### Modules/ROOT/Pages/Concepts/Long Task Timers [[long-task-timers]] = Long Task Timers The long task timer is a special type of timer that lets you measure time while an event being measured is *still running*. A normal Timer only records the duration *after* the task is complete. Long task timers publish at least the following statistics: * Active task count * Total duration of active tasks * The maximum duration of active tasks Unlike a regular `Timer`, a long task timer does not publish statistics about completed tasks. Consider a background process to refresh metadata from a data store. For example, https://github.com/Netflix/edda[Edda] caches AWS resources, such as instances, volumes, auto-scaling groups, and others. Normally all data can be refreshed in a few minutes. If the AWS services have problems, it can take much longer. A long task timer can be used to track the active time for refreshing the metadata. For example, in a Spring application, it is common for such long running processes to be implemented with `@Scheduled`. Micrometer provides a special `@Timed` annotation for instrumenting these processes with a long task timer: [source, java] ---- @Timed(value = "aws.scrape", longTask = true) @Scheduled(fixedDelay = 360000) void scrapeResources() { // find instances, volumes, auto-scaling groups, etc... } ---- It is up to the application framework to make something happen with `@Timed`. If your framework of choice does not support it, you can still use the long task timer: [source, java] ---- LongTaskTimer scrapeTimer = registry.more().longTaskTimer("scrape"); void scrapeResources() { scrapeTimer.record(() => { // find instances, volumes, auto-scaling groups, etc... }); } ---- If we wanted to alert when this process exceeds a threshold, with a long task timer, we receive that alert at the first reporting interval after we have exceeded the threshold. With a regular timer, we would not receive the alert until the first reporting interval after the process completed, over an hour later! The interface contains a fluent builder for long task timers: [source, java] ---- LongTaskTimer longTaskTimer = LongTaskTimer .builder("long.task.timer") .description("a description of what this timer does") // optional .tags("region", "test") // optional .register(registry); ---- The exact series exposed by a long task timer depend on the registry. For example, the Prometheus registry exposes a different set of series when `publishPercentileHistogram()` is enabled; see the xref:../implementations/prometheus.adoc#_long_task_timers[Long task timers section of the Prometheus implementation docs] for details. --- ### Modules/ROOT/Pages/Concepts/Meter Convention [[meter-convention]] = Meter Convention While `MeterFilter` can be used to customize the name and tags (collectively, the convention) of Meters in any instrumentation, it can be more convenient and robust to customize the convention for a common instrumentation more directly. This is the purpose of the `MeterConvention`. For examples of its usage in instrumentation provided by Micrometer as well as examples of different implementations of a convention for an instrumentation, see the xref:../reference/jvm.adoc#meter-conventions[JVM metrics]. --- ### Modules/ROOT/Pages/Concepts/Meter Filters [[meter-filters]] = Meter Filters You can configure each registry with meter filters, which give you greater control over how and when meters are registered and what kinds of statistics they emit. Meter filters serve three basic functions: 1. **Deny** (or **Accept**) meters being registered. 2. **Transform** meter IDs (for example, changing the name, adding or removing tags, and changing the description or base units). 3. **Configure** distribution statistics for some meter types. Implementations of `MeterFilter` are added to the registry programmatically: ==== [source, java] ---- registry.config() .meterFilter(MeterFilter.ignoreTags("too.much.information")) .meterFilter(MeterFilter.denyNameStartsWith("jvm")); ---- ==== Meter filters are applied in order, and the results of transforming or configuring a meter are chained. == Deny or Accept Meters The verbose form of an accept or deny filter is: ==== [source, java] ---- new MeterFilter() { @Override public MeterFilterReply accept(Meter.Id id) { if(id.getName().contains("test")) { return MeterFilterReply.DENY; } return MeterFilterReply.NEUTRAL; } } ---- ==== `MeterFilterReply` has three possible states: * `DENY`: Do not let this meter be registered. When you try to register a meter against a registry and the filter returns `DENY`, the registry returns a NOOP version of that meter (for example, `NoopCounter` or `NoopTimer`). Your code can continue to interact with the NOOP meter, but anything recorded to it is discarded immediately with minimal overhead. * `NEUTRAL`: If no other meter filter has returned `DENY`, registration of meters proceeds normally. * `ACCEPT`: If a filter returns `ACCEPT`, the meter is immediately registered without interrogating the accept methods of any further filters. === Convenience Methods `MeterFilter` provides several convenience static builders for deny and accept type filters: * `accept()`: Accept every meter, overriding the decisions of any filters that follow. * `accept(Predicate)`: Accept any meter matching the predicate. * `acceptNameStartsWith(String)`: Accept every meter with a matching prefix. * `deny()`: Deny every meter, overriding the decisions of any filters that follow. * `denyNameStartsWith(String)`: Deny every meter with a matching prefix. All `MeterBinder` implementations provided by Micrometer have names with common prefixes to allow for easy grouping visualization in UIs but also to make them easy to disable or enable as a group with a prefix. For example, you can deny all JVM metrics with `MeterFilter.denyNameStartsWith("jvm")`. * `deny(Predicate)`: Deny any meter that matches the predicate. * `maximumAllowableMetrics(int)`: Deny any meter after the registry has reached a certain number of meters. * `maximumAllowableTags(String meterNamePrefix, String tagKey, int maximumTagValues, MeterFilter onMaxReached)`: Places an upper bound on the number of tags produced by the matching series. **Whitelisting** only a certain group of metrics is a particularly common case for monitoring systems that are _expensive_. This can be achieved with a static call: * `denyUnless(Predicate)`: Deny all meters that _do not_ match the predicate. === Chaining Deny Accept Meters Meter filters are applied in the order in which they are configured on the registry, so it is possible to stack deny and accept filters to achieve more complex rules: ==== [source, java] ---- registry.config() .meterFilter(MeterFilter.acceptNameStartsWith("http")) .meterFilter(MeterFilter.deny()); ---- ==== This achieves another form of whitelisting by stacking two filters together. Only `http` metrics are allowed to exist in this registry. == Transforming Metrics The following example shows a transform filter: ==== [source, java] ---- new MeterFilter() { @Override public Meter.Id map(Meter.Id id) { if(id.getName().startsWith("test")) { return id.withName("extra." + id.getName()).withTag("extra.tag", "value"); } return id; } } ---- ==== This filter adds a name prefix and an additional tag conditionally to meters starting with a name of `test`. [IMPORTANT] .`MeterFilter` implementations of `map` should be "static" ==== The `id` parameter is the only dynamic input that changes over the lifecycle of the `MeterFilter` on which `MeterFilter` implementations should depend. Depending on a value that will be fixed at runtime such as an instance ID or hostname is fine because it does not change after application start-up. Use cases where dynamic behavior is desired, such as defining tags based on the context of a request etc., should be implemented in the instrumentation itself rather than in a `MeterFilter`. For example, see `MongoMetricsCommandListener` and the `MongoCommandTagsProvider` it takes in a constructor argument as well as the default implementation `DefaultMongoCommandTagsProvider`. See also xref:../observation/components.adoc#micrometer-observation-predicates-filters[ObservationFilter] which allows dynamic implementations. ==== === Convenience Methods `MeterFilter` provides convenience builders for many common transformation cases: * `commonTags(Iterable)`: Adds a set of tags to all metrics. Adding common tags for application name, host, region, and others is a highly recommended practice. * `ignoreTags(String...)`: Drops matching tag keys from every meter. This is particularly useful when a tag provably comes to have too high cardinality and starts stressing your monitoring system or costing too much but you cannot change all the instrumentation points quickly. * `replaceTagValues(String tagKey, Function replacement, String... exceptions)`: Replace tag values according to the provided mapping for all matching tag keys. You can use this to reduce the total cardinality of a tag by mapping some portion of tag values to something else. * `renameTag(String meterNamePrefix, String fromTagKey, String toTagKey)`: Rename a tag key for every metric that begins with a given prefix. == Configuring Distribution Statistics `Timer` and `DistributionSummary` contain a set of optional distribution statistics (in addition to the basics of count, total, and max) that you can configure through filters. These distribution statistics include pre-computed percentiles, SLOs, and histograms. ==== [source, java] ---- new MeterFilter() { @Override public DistributionStatisticConfig configure(Meter.Id id, DistributionStatisticConfig config) { if (id.getName().startsWith("prefix")) { return DistributionStatisticConfig.builder() .percentiles(0.9, 0.95) .build() .merge(config); } return config; } }; ---- ==== Generally, you should create a new `DistributionStatisticConfig` with only the pieces you wish to configure and then `merge` it with the input configuration. This lets you drop down on registry-provided defaults for distribution statistics and to chain multiple filters together, each configuring some part of the distribution statistics (for example, you might want a 100ms SLO for all HTTP requests but only percentile histograms on a few critical endpoints). `MeterFilter` provides convenience builders for: * `maxExpected(Duration/long)`: Governs the upper bound of percentile histogram buckets shipped from a timer or summary. * `minExpected(Duration/long)`: Governs the lower bound of percentile histogram buckets shipped from a timer or summary. Spring Boot offers property-based filters for configuring SLOs, percentiles, and percentile histograms by name prefix. == Apply MeterFilters conditionally It is a common use-case to apply MeterFilters only for a selection of Meters (for example, for ones that have a certain name or start with a given prefix). In order to do this, Micrometer provides a convenience method `forMeters(Predicate predicate, MeterFilter delegate)` that enables the provided (delegate) filter for the Meters selected by the predicate. See the following example: ==== [source, java] ---- registry.config() .meterFilter(MeterFilter.forMeters(startsWith("prefix"), MeterFilter.ignoreTags("extra"))); Predicate startsWith(String prefix) { return id -> id.getName().startsWith(prefix); } ---- ==== --- ### Modules/ROOT/Pages/Concepts/Meter Provider [[meter-provider]] = Meter Provider It's a common use-case to attach tags dynamically to a `Meter`. Let's say we execute a job and we want to use a `Timer` to instrument it: [source, java] ---- Timer.Sample sample = Timer.start(registry); Result result = job.execute(); Timer timer = Timer.builder("job.execution") .tag("job.name", "job") .tag("status", result.status()) .register(registry); sample.stop(timer); ---- This lets us dynamically determine the `status` tag from the end state of the operation we are timing. There are two drawbacks of doing this: 1. Every time the above is executed, a new `Timer.Builder` instance is created. This increases the amount of data that the GC needs to collect. 2. The code above is somewhat boilerplate, it does not let you define the common properties of a Timer and attach what is dynamically changing but everything is always present. NOTE: In some cases you can use `registry.timer("job.execution", "job.name", "my-job", "status", result.status())` instead of using `Timer.Builder` which can save you some extra objects but this is not always possible. You can resolve both of these issues by using a `MeterProvider`. It's a convenience interface to create new meters from tags using a common "template". NOTE: Not every `Meter` can do this, `MeterProvider` can be used with `Counter`, `Timer`, `LongTaskTimer`, and `DistributionSummary`. Here's what you can do instead of the above: [source, java] ---- private MeterProvider timerProvider = Timer.builder("job.execution") .tag("job.name", "my-job") .withRegistry(registry); <1> // ... Timer.Sample sample = Timer.start(registry); Result result = job.execute(); sample.stop(timerProvider.withTags("status", result.status())); <2> ---- <1> Definition of the `MeterProvider` for `Timer` with all the "static" fields necessary. Please note the `withRegistry` method call. <2> Definition of the dynamic tags. Note that only those tags are defined here that are dynamic and everything else is defined where the `MeterProvider` is created. The `withTags` method returns a `Timer` that is created using the tags defined in `withTags` plus everything else that is defined by the `MeterProvider`. This and the previous example produce the same output, the only difference is the amount of boilerplate in your code and the amount of builder objects created in the heap. --- ### Modules/ROOT/Pages/Concepts/Meters [[meters]] = Meters A `Meter` is the interface for collecting a set of measurements (which we individually call metrics) about your application. Micrometer supports a set of `Meter` primitives, including `Timer`, `Counter`, `Gauge`, `DistributionSummary`, `LongTaskTimer`, `FunctionCounter`, `FunctionTimer`, and `TimeGauge`. Different meter types result in a different number of time series metrics. For example, while there is a single metric that represents a `Gauge`, a `Timer` measures both the count of timed events and the total time of all timed events. TIP: Recording a measurement for a `Meter` is expected to be a relatively cheap operation and should not throw any exception. If the xref:./registry.adoc[registry] supports publishing metrics to a monitoring system, this is done in a separate thread and should not affect recording metrics. A meter is uniquely identified by its name and dimensions. We use the terms, "`dimensions`" and "`tags,`" interchangeably, and the Micrometer interface is `Tag` simply because it is shorter. As a general rule, it should be possible to use the name as a pivot. Dimensions let a particular named metric be sliced to drill down and reason about the data. This means that, if only the name is selected, you can drill down by using other dimensions and reason about the value being shown. --- ### Modules/ROOT/Pages/Concepts/Naming [[naming-meters]] = Naming Meters Micrometer employs a naming convention that separates lowercase words with a `.` (dot) character. Different monitoring systems have different recommendations regarding naming conventions, and some naming conventions may be incompatible between one system and another. Each Micrometer implementation for a monitoring system comes with a naming convention that transforms lowercase dot notation names to the monitoring system's recommended naming convention. Additionally, this naming convention implementation removes special characters that are disallowed by the monitoring system from the metric names and tags. You can override the default naming convention for a registry by implementing `NamingConvention` and setting it on the registry: ==== [source,java] ---- registry.config().namingConvention(myCustomNamingConvention); ---- ==== With naming conventions in place, the following timer registered in Micrometer looks good natively in a wide variety of monitoring systems: ==== [source,java] ---- registry.timer("http.server.requests"); ---- ==== . Prometheus - `http_server_requests_duration_seconds` . Atlas - `httpServerRequests` . Graphite - `http.server.requests` . InfluxDB - `http_server_requests` By adhering to Micrometer's lowercase dot notation convention, you guarantee the maximum degree of portability for your metric names across monitoring systems. == Tag Naming TIP: We recommend that you follow the same lowercase dot notation described for meter names when naming tags. Using this consistent naming convention for tags allows for better translation into the respective monitoring system's idiomatic naming schemes. Suppose we are trying to measure the number of http requests and the number of database calls. *Recommended Approach* ==== [source,java] ---- registry.counter("database.calls", "db", "users") registry.counter("http.requests", "uri", "/api/users") ---- ==== This variant provides enough context so that, if only the name is selected, the value can be reasoned about and is at least potentially meaningful. For example if we select `database.calls`, we can see the total number of calls to all databases. Then we can group by or select by `db` to drill down further or perform comparative analysis on the contribution of calls to each database. *Bad Approach* ==== [source,java] ---- registry.counter("calls", "class", "database", "db", "users"); registry.counter("calls", "class", "http", "uri", "/api/users"); ---- ==== In this approach, if we select `calls`, we get a value that is an aggregate of the number of calls to the database and to our API endpoint. This time series is not useful without further dimensional drill-down. == Common Tags You can define common tags at the registry level and add them to every metric reported to the monitoring system. This is generally used for dimensional drill-down on the operating environment, such as host, instance, region, stack, and others. The following lines set the same tag in two equivalent ways: ==== [source,java] ---- registry.config().commonTags("stack", "prod", "region", "us-east-1"); registry.config().commonTags(Arrays.asList(Tag.of("stack", "prod"), Tag.of("region", "us-east-1"))); // equivalently ---- ==== Further calls to `commonTags` append additional common tags. [IMPORTANT] ==== Common tags generally have to be added to the registry _before_ any (possibly autoconfigured) meter binders. Depending on your environment, there are different ways to achieve this. If you use Spring Boot, you have two options: * Add your common tags with https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.customizing.common-tags[configuration properties] * If you need more flexibility (for example, you have to add common tags to a registry defined in a shared library), register a `MeterRegistryCustomizer` callback interface as a bean to add your common tags. See the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.getting-started[Spring Boot Reference Documentation] for more information. ==== == Tag Values Tag values must be non-null. WARNING: Beware of the potential for tag values coming from user-supplied sources to blow up the cardinality of a metric. You should always carefully normalize and add bounds to user-supplied input. Sometimes, the cause is sneaky. Consider the URI tag for recording HTTP requests on service endpoints. If we do not constrain 404's to a value like NOT_FOUND, the dimensionality of the metric would grow with each resource that cannot be found. --- ### Modules/ROOT/Pages/Concepts/Rate Aggregation [[rate-aggregation]] = Rate Aggregation Micrometer is aware of whether a particular monitoring system expects rate aggregation to happen client-side before metrics are published or ad-hoc as part of the query on the server. It accumulates metrics according to which style the monitoring system expects. Not all measurements are reported or best viewed as a rate. For example, gauge values and the active task count long task timers are not rates. == Server-side Monitoring systems that perform server-side rate math expect absolute values to be reported at each publishing interval. For example, the absolute count of all increments to a counter since the beginning of the application is sent on each publishing interval. Suppose we have a slightly positively biased random walk that chooses to increment a counter once every 10 milliseconds. If we view the raw counter value in a system like Prometheus, we see a step-wise monotonically increasing function (the width of the step is the interval at which Prometheus is polling or scraping for data). image::concepts/prometheus-counter-norate.png[Absolute counter value] Representing a counter without rate aggregation over some time window is rarely useful, as the representation is a function of both the rapidity with which the counter is incremented and the longevity of the service. In the preceding example, the counter drops back to zero on service restart. The rate-aggregated graph would return back to a value of around 55 as soon as the new instance (say on a production deployment) was in service. image::concepts/prometheus-counter.png[Rate-aggregated counter] If you have achieved zero-downtime deployments (for example, through red-black deployments), you should be able to comfortably set _minimum_ alert thresholds on the rate-aggregated graph without service restarts causing dips in the counter value. IMPORTANT: For most production purposes, whether it be alerting, automated canary analysis, or other use cases, base your automation off of rate-aggregated data. == Client-side Two other classes of monitoring system either: * Expect rate-aggregated data. Given the key insight that, for most production purposes, we should be basing our decisions off of rates rather than absolute values. Such systems benefit from having to do less math to satisfy queries. * Have relatively little or no math operations that would let us rate-aggregate data through our queries. For these systems, publishing pre-aggregated data is the only way to build meaningful representations. Micrometer efficiently maintains rate data by means of a step value that accumulates data for the current publishing interval. When the step value is polled (when publishing, for example), if the step value detects that the current interval has elapsed, it moves current data to "`previous`" state. This previous state is what is reported until the next time current data overwrites it. The following image shows the interaction of current and previous state, along with polling: image::concepts/rate-normalizing.png[Behavior of a step value,width=1200] The value returned by the poll function is always _rate per second * interval_. If the step value shown in the preceding image represents the values of a counter, we could say that the counter saw "`0.3 increments per second`" in the first interval, which is reportable to the backend at any time during the second interval. Micrometer timers track at least a count and the total time as separate measurements. Suppose we configure publishing at 10-second intervals and we saw 20 requests that each took 100ms. Then, for the first interval: . `count` = 10 seconds * (20 requests / 10 seconds) = 20 requests . `totalTime` = 10 seconds * (20 * 100 ms / 10 seconds) = 2 seconds The `count` statistic is meaningful by itself: It is a measure of _throughput_. `totalTime` represents the total latency of all requests in the interval. Additionally, consider: `totalTime / count` = 2 seconds / 20 requests = 0.1 seconds / request = 100 ms / request This is a useful measure of _average latency_. When the same idea is applied to the `totalAmount` and `count` emanating from distribution summaries, the measure is called a _distribution average_. Average latency is just the distribution average for a distribution summary measured in time (a timer). Some monitoring systems (such as Atlas) provide facilities for computing the distribution average from these statistics, and Micrometer includes `totalTime` and `count` as separate statistics. Others, (such as Datadog) do not have this kind of operation built-in, and Micrometer calculates the distribution average client-side and ships that. Shipping the rate for the publishing interval is sufficient to reason about the rate over any time window greater than or equal to the publishing interval. In our example, if a service continues to receive 20 requests that each take 100ms for every 10 second interval in a given minute, we could say: . Micrometer reported "`20 requests`" for `count` on every 10-second interval. The monitoring system sums these six 10-second intervals and arrives at the conclusion that there are 120 requests / minute. Note that it is the monitoring system doing this summation, not Micrometer. . Micrometer reported "`2 seconds`" of `totalTime` on every 10 second interval. The monitoring system can sum all total time statistics over the minute to yield "`12 seconds`" of total time in the minute interval. Then, the average latency is as we expect: 12 seconds / 120 requests = 100 ms / request. --- ### Modules/ROOT/Pages/Concepts/Registry [[registry]] = Registry Meters in Micrometer are created from and held in a `MeterRegistry`. Each supported monitoring system has an implementation of `MeterRegistry`. How a registry is created varies for each implementation. Micrometer includes a `SimpleMeterRegistry` that holds the latest value of each meter in memory and does not export the data anywhere. If you do not yet have a preferred monitoring system, you can get started playing with metrics by using the simple registry: ==== [source,java] ---- MeterRegistry registry = new SimpleMeterRegistry(); ---- ==== NOTE: A `SimpleMeterRegistry` is autowired for you in Spring-based applications. == Composite Registries Micrometer provides a `CompositeMeterRegistry` to which you can add multiple registries, letting you publish metrics to more than one monitoring system simultaneously: ==== [source,java] ---- CompositeMeterRegistry composite = new CompositeMeterRegistry(); Counter compositeCounter = composite.counter("counter"); compositeCounter.increment(); <1> SimpleMeterRegistry simple = new SimpleMeterRegistry(); composite.add(simple); <2> compositeCounter.increment(); <3> ---- <1> Increments are NOOP'd until there is a registry in the composite. The counter's count still yields 0 at this point. <2> A counter named `counter` is registered to the simple registry. <3> The simple registry counter is incremented, along with counters for any other registries in the composite. ==== == Global Registry Micrometer provides a static global registry called `Metrics.globalRegistry` and a set of static builders for generating meters based on this registry (note that `globalRegistry` is a composite registry): ==== [source,java] ---- class MyComponent { Counter featureCounter = Metrics.counter("feature", "region", "test"); <1> void feature() { featureCounter.increment(); } void feature2(String type) { Metrics.counter("feature.2", "type", type).increment(); <2> } } class MyApplication { void start() { // wire your monitoring system to global static state Metrics.addRegistry(new SimpleMeterRegistry()); <3> } } ---- <1> Wherever possible (and especially where instrumentation performance is critical), store `Meter` instances in fields to avoid a lookup on their name or tags on each use. <2> When tags need to be determined from local context, you have no choice but to construct or lookup the Meter inside your method body. The lookup cost is just a single hash lookup, so it is acceptable for most use cases. <3> It is OK to add registries _after_ meters have been created with code like `Metrics.counter(...)`. These meters are added to each registry, as it is bound to the global composite. ==== --- ### Modules/ROOT/Pages/Concepts/Timers [[timers]] = Timers Timers are intended for measuring short-duration latencies and the frequency of such events. All implementations of `Timer` report at least the total time and the count of events as separate time series but other time series can also be reported depending on what is supported by the backend (max, percentiles, histograms). While you can use timers for other use cases, note that negative values are not supported, and recording many longer durations could cause overflow of the total time at `Long.MAX_VALUE` nanoseconds (292.3 years). As an example, consider a graph showing request latency to a typical web server. The server can be expected to respond to many requests quickly, so the timer gets updated many times per second. The appropriate base unit for timers varies by metrics backend, and for good reason. Micrometer is decidedly un-opinionated about this. However, because of the potential for confusion, Micrometer requires a `TimeUnit` when interacting with `Timer` implementations. Micrometer is aware of the preferences of each implementation and publishes your timing in the appropriate base unit, based on the implementation. The following listing shows part of the `Timer` interface: [source,java] ---- public interface Timer extends Meter { ... void record(long amount, TimeUnit unit); void record(Duration duration); double totalTime(TimeUnit unit); } ---- The interface contains a fluent builder for timers: [source,java] ---- Timer timer = Timer .builder("my.timer") .description("a description of what this timer does") // optional .tags("region", "test") // optional .register(registry); ---- NOTE: The maximum statistical value for basic `Timer` implementations, such as `CumulativeTimer` and `StepTimer`, is a time window maximum (`TimeWindowMax`). It means that its value is the maximum value during a time window. If no new values are recorded for the time window length, the max is reset to 0 as a new time window starts. The time window size until values are fully expired is the `expiry` multiplied by the `bufferLength` in `DistributionStatisticConfig`. `expiry` defaults to the step size of the meter registry unless it's explicitly set to a different value, and `bufferLength` defaults to `3`. A time window maximum is used to capture maximum latency in a subsequent interval after heavy resource pressure triggers the latency and prevents metrics from being published. Percentiles are also time window percentiles (`TimeWindowPercentileHistogram`). Histogram buckets usually behave like counters, so depending on the backend, they can be reported as cumulative values (for example in the case of Prometheus) or as a rate at which a counter increments over the push interval. == Recording Blocks of Code The `Timer` interface exposes several convenience overloads for recording timings inline, including the following: [source,java] ---- timer.record(() -> dontCareAboutReturnValue()); timer.recordCallable(() -> returnValue()); Runnable r = timer.wrap(() -> dontCareAboutReturnValue()); <1> Callable c = timer.wrap(() -> returnValue()); ---- <1> Wrap `Runnable` or `Callable` and return the instrumented version of it for use later. NOTE: A `Timer` is really a specialized distribution summary that is aware of how to scale durations to the base unit of time of each monitoring system and has an automatically determined base unit. In every case where you want to measure time, you should use a `Timer` rather than a `DistributionSummary`. == Storing Start State in `Timer.Sample` You may also store the start state in a sample instance that can be stopped later. The sample records a start time based on the registry's clock. After starting a sample, execute the code to be timed and finish the operation by calling `stop(Timer)` on the sample: [source, java] ---- Timer.Sample sample = Timer.start(registry); // do stuff Response response = ... sample.stop(registry.timer("my.timer", "response", response.status())); ---- Note how we do not decide the timer to which to accumulate the sample until it is time to stop the sample. This lets us dynamically determine certain tags from the end state of the operation we are timing. == The `@Timed` Annotation The `micrometer-core` module contains a `@Timed` annotation that frameworks can use to add timing support to either specific types of methods such as those serving web request endpoints or, more generally, to all methods. Also, an incubating AspectJ aspect is included in `micrometer-core`. You can use it in your application either through compile/load time AspectJ weaving or through framework facilities that interpret AspectJ aspects and proxy targeted methods in some other way, such as Spring AOP. Here is a sample Spring AOP configuration: [source,java] ---- @Configuration public class TimedConfiguration { @Bean public TimedAspect timedAspect(MeterRegistry registry) { return new TimedAspect(registry); } } ---- Applying `TimedAspect` makes `@Timed` usable on any arbitrary method in an AspectJ proxied instance, as the following example shows: [source,java] ---- @Service public class ExampleService { @Timed public void sync() { // @Timed will record the execution time of this method, // from the start and until it exits normally or exceptionally. ... } @Async @Timed public CompletableFuture async() { // @Timed will record the execution time of this method, // from the start and until the returned CompletableFuture // completes normally or exceptionally. return CompletableFuture.supplyAsync(...); } } ---- NOTE: `TimedAspect` doesn't support meta-annotations with `@Timed`. === @MeterTag on Method Parameters To support using the `@MeterTag` annotation on method parameters, you need to configure the `@TimedAspect` to add the `MeterTagAnnotationHandler`. [source,java,subs=+attributes] ----- include::{include-java}/metrics/TimedAspectTest.java[tags=resolvers,indent=0] include::{include-java}/metrics/TimedAspectTest.java[tags=meter_tag_annotation_handler,indent=0] ----- Let's assume that we have the following interface. [source,java,subs=+attributes] ----- include::{include-java}/metrics/TimedAspectTest.java[tags=interface,indent=0] ----- When its implementations would be called with different arguments (remember that the implementation needs to be annotated with `@Timed` annotation too), the following timers would be created: [source,java,subs=+attributes] ----- // Example for returning on the parameter include::{include-java}/metrics/TimedAspectTest.java[tags=example_value_to_string,indent=0] // Example for calling the provided on the parameter include::{include-java}/metrics/TimedAspectTest.java[tags=example_value_resolver,indent=0] // Example for calling the provided include::{include-java}/metrics/TimedAspectTest.java[tags=example_value_spel,indent=0] // Example for using multiple @MeterTag annotations on the same parameter // @MeterTags({ @MeterTag(...), @MeterTag(...) }) can be also used include::{include-java}/metrics/TimedAspectTest.java[tags=example_multi_annotations,indent=0] ----- == Function-tracking Timers Micrometer also provides a more infrequently used timer pattern that tracks two monotonically increasing functions (a function that stays the same or increases over time but never decreases): a count function and a total time function. Some monitoring systems, such as Prometheus, push cumulative values for counters (which apply to both the count and total time functions in this case) to the backend, but others publish the rate at which a counter increments over the push interval. By employing this pattern, you let the Micrometer implementation for your monitoring system choose whether to rate normalize the timer, and your timer remains portable across different types of monitoring systems. [source, java] ----- IMap cache = ...; // suppose we have a Hazelcast cache registry.more().timer("cache.gets.latency", Tags.of("name", cache.getName()), cache, c -> c.getLocalMapStats().getGetOperationCount(), <1> c -> c.getLocalMapStats().getTotalGetLatency(), TimeUnit.NANOSECONDS <2> ); ----- <1> `getGetOperationCount()` is a monotonically increasing function incrementing with every cache get from the beginning of its life. <2> This represents the unit of time represented by `getTotalGetLatency()`. Each registry implementation specifies what its expected base unit of time is, and the total time reported will be scaled to this value. The function-tracking timer, in concert with the monitoring system's rate normalizing functionality (whether this is an artifact of the query language or the way data is pushed to the system), adds a layer of richness to the cumulative value of the functions themselves. You can reason about the _rate_ of throughput and latency, whether that rate is within an acceptable bound, is increasing or decreasing over time, and so on. WARNING: Micrometer cannot guarantee the monotonicity of the count and total time functions for you. By using this signature, you are asserting their monotonicity based on what you know about their definitions. There is also a fluent builder for function timers on the `FunctionTimer` interface itself, providing access to less frequently used options, such as base units and description. You can register the timer as the last step of its construction by calling `register(MeterRegistry)`: [source, java] ---- IMap cache = ... FunctionTimer.builder("cache.gets.latency", cache, c -> c.getLocalMapStats().getGetOperationCount(), c -> c.getLocalMapStats().getTotalGetLatency(), TimeUnit.NANOSECONDS) .tags("name", cache.getName()) .description("Cache gets") .register(registry); ---- WARNING: Attempting to construct a function-tracking timer with a primitive number or one of its `java.lang` object forms is always incorrect. These numbers are immutable. Thus, the function-tracking timer cannot ever be changed. Attempting to "re-register" the function-tracking timer with a different number does not work, as the registry maintains only one meter for each unique combination of name and tags. "Re-registering" a function-tracking timer can happen indirectly for example as the result of a `MeterFilter` modifying the name and/or the tags of two different function-tracking timers so that they will be the same after the filter is applied. Attempting to "re-register" a function-tracking timer will result in a warning like this: [source] ---- WARNING: This FunctionTimer has been already registered (MeterId{name='my.ft', tags=[]}), the registration will be ignored. Note that subsequent logs will be logged at debug level. ---- == Pause Detection Micrometer uses the `LatencyUtils` package to compensate for https://highscalability.com/blog/2015/10/5/your-load-generator-is-probably-lying-to-you-take-the-red-pi.html[coordinated omission] -- extra latency arising from system and VM pauses that skew your latency statistics downward. Distribution statistics, such as percentiles and SLO counts, are influenced by a pause detector implementation that adds additional latency here and there to compensate for pauses. Micrometer supports two pause detector implementations: a clock-drift based detector and a no-op detector. The no-op detector is configured by default, but the clock-drift detector can be configured as shown in the next example. The clock-drift based detector has a configurable sleep interval and pause threshold. CPU consumption is inversely proportional to `sleepInterval`, as is pause detection accuracy. 100ms for both values is a reasonable default to offer decent detection of long pause events while consuming a negligible amount of CPU time. NOTE: You need to add LatencyUtils(`org.latencyutils:LatencyUtils`) to your runtime classpath to use the `ClockDriftPauseDetector`. Without it on the runtime classpath, a warning will be logged and no pause detection will happen. You can customize the pause detector as follows: [source,java] ---- registry.config().pauseDetector(new ClockDriftPauseDetector(sleepInterval, pauseThreshold)); registry.config().pauseDetector(new NoPauseDetector()); ---- In the future, we may provide further detector implementations. Some pauses may be able to be inferred from GC logging in some circumstances, for example, without requiring a constant CPU load, however minimal. Also, a future JDK may provide direct access to pause events. == Memory Footprint Estimation Timers are the most memory-consuming meter, and their total footprint can vary dramatically, depending on which options you choose. The following table of memory consumption is based on the use of various features. These figures assume no tags and a ring buffer length of 3. Adding tags adds somewhat to the total, as does increasing the buffer length. Total storage can also vary somewhat, depending on the registry implementation. * R = Ring buffer length. We assume the default of 3 in all examples. R is set with `Timer.Builder#distributionStatisticBufferLength`. * B = Total histogram buckets. Can be SLO boundaries or percentile histogram buckets. By default, timers are clamped to a minimum expected value of 1ms and a maximum expected value of 30 seconds, yielding 66 buckets for percentile histograms, when applicable. * I = Interval estimator for pause compensation. 1.7 kb. * M = Time-decaying max. 104 bytes. * Fb = Fixed boundary histogram. 8b * B * R. * Pp = Percentile precision. By default, it is 1. Generally in the range [0, 3]. Pp is set with `Timer.Builder#percentilePrecision`. * Hdr(Pp) = High dynamic range histogram. - When Pp = 0: 1.9kb * R + 0.8kb - When Pp = 1: 3.8kb * R + 1.1kb - When Pp = 2: 18.2kb * R + 4.7kb - When Pp = 3: 66kb * R + 33kb [width="80%",options="header"] |========================================================= |Pause detection |Client-side percentiles |Histogram and/or SLOs |Formula | Example |Yes |No |No |I + M| ~1.8kb |Yes |No |Yes |I + M + Fb|For default percentile histogram, ~7.7kb |Yes |Yes |Yes |I + M + Hdr(Pp)|For the addition of a 0.95 percentile with defaults otherwise, ~14.3kb |No |No |No |M| ~0.1kb |No |No |Yes |M + Fb|For default percentile histogram, ~6kb |No |Yes |Yes |M + Hdr(Pp)|For the addition of a 0.95 percentile with defaults otherwise, ~12.6kb |========================================================= NOTE: For Prometheus, specifically, R is _always_ equal to 1, regardless of how you attempt to configure it through `Timer.Builder`. This special case exists because Prometheus expects cumulative histogram data that never rolls over. --- ### Modules/ROOT/Pages/Guides/Console Reporter = Passing through to Dropwizard's Console Reporter This guide shows how to plug in less commonly used Dropwizard `Reporter` implementations -- in this case, the `ConsoleReporter`. [source,java] ---- @Bean public MetricRegistry dropwizardRegistry() { return new MetricRegistry(); } @Bean public ConsoleReporter consoleReporter(MetricRegistry dropwizardRegistry) { ConsoleReporter reporter = ConsoleReporter.forRegistry(dropwizardRegistry) .convertRatesTo(TimeUnit.SECONDS) .convertDurationsTo(TimeUnit.MILLISECONDS) .build(); reporter.start(1, TimeUnit.SECONDS); return reporter; } @Bean public MeterRegistry consoleLoggingRegistry(MetricRegistry dropwizardRegistry) { DropwizardConfig consoleConfig = new DropwizardConfig() { @Override public String prefix() { return "console"; } @Override public String get(String key) { return null; } }; return new DropwizardMeterRegistry(consoleConfig, dropwizardRegistry, HierarchicalNameMapper.DEFAULT, Clock.SYSTEM) { @Override protected Double nullGaugeValue() { return null; } }; } ---- Note that, in a Spring environment, this registry is added to other implementations in a composite and is used for all metrics, both custom and auto-configured. [source,java] ---- class MyComponent { private final MeterRegistry registry; public MyComponent(MeterRegistry registry) { this.registry = registry; } public void doSomeWork(String lowCardinalityInput) { registry.timer("my.latency", "input", lowCardinalityInput).record(() -> { // do work }); } } ---- The following listing shows typical output for this custom timer: [source,txt] ---- 3/2/18 10:14:27 AM ============================================================= -- Timers ---------------------------------------------------------------------- myLatency.lowCardinalityInput.INPUT count = 1 mean rate = 1.02 calls/second 1-minute rate = 0.00 calls/second 5-minute rate = 0.00 calls/second 15-minute rate = 0.00 calls/second min = 100.00 milliseconds max = 100.00 milliseconds mean = 100.00 milliseconds stddev = 0.00 milliseconds median = 100.00 milliseconds 75% <= 100.00 milliseconds 95% <= 100.00 milliseconds 98% <= 100.00 milliseconds 99% <= 100.00 milliseconds 99.9% <= 100.00 milliseconds ---- --- ### Modules/ROOT/Pages/Guides/Custom Meter Registry = Custom Meter Registry Micrometer includes support for popular meter registries, so you should check those first. For an existing meter registry, if you think that your requirements are generally useful, consider creating an issue or PR against the Micrometer issue tracker. For a non-existent meter registry, if it is widely-used, consider creating an issue or PR for it. If you need to create your own custom meter registry, you can create it by extending one of the base classes for meter registries: `MeterRegistry`, `PushMeterRegistry`, or `StepMeterRegistry`. The most common way is to extend `StepMeterRegistry`. You can create your own custom `StepMeterRegistry`. First, define your own meter registry configuration by extending `StepRegistryConfig`, as follows: [source,java] ---- public interface CustomRegistryConfig extends StepRegistryConfig { CustomRegistryConfig DEFAULT = k -> null; @Override default String prefix() { return "custom"; } } ---- Second, define your own meter registry by extending `StepMeterRegistry`, as follows: [source,java] ---- public class CustomMeterRegistry extends StepMeterRegistry { public CustomMeterRegistry(CustomRegistryConfig config, Clock clock) { super(config, clock); start(new NamedThreadFactory("custom-metrics-publisher")); } @Override protected void publish() { getMeters().stream().forEach(meter -> System.out.println("Publishing " + meter.getId())); } @Override protected TimeUnit getBaseTimeUnit() { return TimeUnit.MILLISECONDS; } } ---- Finally, create the meter registry configuration and the meter registry. If you use Spring Boot, you can do so as follows: [source,java] ---- @Configuration public class MetricsConfig { @Bean public CustomRegistryConfig customRegistryConfig() { return CustomRegistryConfig.DEFAULT; } @Bean public CustomMeterRegistry customMeterRegistry(CustomRegistryConfig customRegistryConfig, Clock clock) { return new CustomMeterRegistry(customRegistryConfig, clock); } } ---- --- ### Modules/ROOT/Pages/Guides/Http Sender Resilience4j Retry = HttpSender with Resilience4j Retry `HttpSender` is an interface for HTTP clients that are used in meter registries for HTTP-based metrics publication. There are two implementations: * `HttpUrlConnectionSender`: `HttpURLConnection`-based `HttpSender` * `OkHttpSender`: OkHttp-based `HttpSender` Micrometer does not include support for retry, but you can decorate it with Resilience4j retry, as follows: [source,java] ---- @Bean public DatadogMeterRegistry datadogMeterRegistry(DatadogConfig datadogConfig, Clock clock) { return DatadogMeterRegistry.builder(datadogConfig) .clock(clock) .httpClient(new RetryHttpClient()) .build(); } private static class RetryHttpClient extends HttpUrlConnectionSender { private final RetryConfig retryConfig = RetryConfig.custom() .maxAttempts(2) .waitDuration(Duration.ofSeconds(5)) .build(); private final Retry retry = Retry.of("datadog-metrics", this.retryConfig); @Override public Response send(Request request) { CheckedFunction0 retryableSupplier = Retry.decorateCheckedSupplier( this.retry, () -> super.send(request)); return Try.of(retryableSupplier).get(); } } ---- --- ### Modules/ROOT/Pages/Implementations/ Install [id=installing-micrometer-registry-{system}] == Installing micrometer-registry-{system} It is recommended to use the BOM provided by Micrometer (or your framework if any), you can see how to configure it xref:../installing.adoc[here]. The examples below assume you are using a BOM. === Gradle After the BOM is xref:../installing.adoc[configured], add the following dependency: [source,groovy,subs=+attributes] ---- implementation 'io.micrometer:micrometer-registry-{system}' ---- NOTE: The version is not needed for this dependency since it is defined by the BOM. === Maven After the BOM is xref:../installing.adoc[configured], add the following dependency: [source,xml,subs=+attributes] ---- io.micrometer micrometer-registry-{system} ---- NOTE: The version is not needed for this dependency since it is defined by the BOM. --- ### Modules/ROOT/Pages/Implementations/AppOptics = Micrometer AppOptics :sectnums: :system: appoptics AppOptics is a dimensional time-series SaaS with built-in dashboarding. include::_install.adoc[] == Configuring The following example configures an AppOptics instance: [source,java] ---- AppOpticsConfig appopticsConfig = new AppOpticsConfig() { @Override public String apiToken() { return MY_TOKEN; } @Override public @Nullable String get(String k) { return null; } }; MeterRegistry registry = new AppOpticsMeterRegistry(appopticsConfig, Clock.SYSTEM); ---- `AppOpticsConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.appoptics` directly to the `AppOpticsConfig`: [source,yml] ---- management.metrics.export.appoptics: api-token: YOURKEY # You will probably want disable AppOptics publishing in a local development profile. enabled: true # The interval at which metrics are sent to AppOptics. The default is 1 minute. step: 1m ---- == Graphing This section serves as a quick start to rendering useful representations in AppOptics for metrics that originate in Micrometer. === Timers The AppOptics implementation of `Timer` produces three fields in AppOptics: * `count`: Rate of number of calls/second. * `sum`: Rate of total time/second. * `max`: A sliding window showing the maximum amount recorded. .Dimensionally aggregable average in AppOptics. image::implementations/appoptics-timer-average.png[AppOptics timer average] AppOptics performs the `sum/count` division dimensionally to generate aggregable averages on your behalf. .Timer over a simulated service. image::implementations/appoptics-timer.png[AppOptics-rendered timer] --- ### Modules/ROOT/Pages/Implementations/Atlas = Micrometer Atlas :sectnums: :system: atlas Atlas is an in-memory dimensional time series database with built-in graphing, a custom stack-based query language, and advanced math operations. Atlas originated at Netflix, where it remains the operational metrics solution. include::_install.adoc[] == Configuring [source,java] ---- AtlasConfig atlasConfig = new AtlasConfig() { @Override public Duration step() { return Duration.ofSeconds(10); } @Override public String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new AtlasMeterRegistry(atlasConfig, Clock.SYSTEM); ---- Micrometer uses Netflix's https://github.com/netflix/spectator[Spectator] as the underlying instrumentation library when recording metrics destined for Atlas. `AtlasConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties prefixed with `management.metrics.export.atlas` directly to the `AtlasConfig`: [source,yml] ---- management.metrics.export.atlas: # The location of your Atlas server uri: http://localhost:7101/api/v1/publish # You will probably want to conditionally disable Atlas publishing in local development. enabled: true # The interval at which metrics are sent to Atlas. The default is 1 minute. step: 1m ---- == Graphing This section serves as a quick start to rendering useful representations in Atlas for metrics originating in Micrometer. See the https://github.com/netflix/atlas/wiki[Atlas wiki] for a far more complete reference of what is possible in Atlas. === Counters Atlas serves up graphs in the form of PNG images (and other https://github.com/Netflix/atlas/wiki/Output-Formats[output formats] as well). We use the following query to visualize the counter from Atlas. Note that the value is rate-normalized over the step interval rather than monotonically increasing. Atlas always expects xref:../concepts/rate-aggregation.adoc#_client_side[rate-aggregated] data for counters from Micrometer. .Counter over a positive-biased random walk. image::implementations/atlas-counter.png[Atlas-rendered counter] [source,http] ---- GET /api/v1/graph? q= name,counter,:eq, 2,:lw &tz=US/Central &s=e-15m <1> &w=400 <2> &l=0 <3> Host: localhost:7101 ---- <1> The range of time we want to visualize along the x-axis. `e` represents the end time or "`now`". This graph's axis is from 15 minutes ago until now. Atlas automatically chooses the finest grained step interval available from the data that would render at least 1px wide on the resultant image. <2> The overall width of the PNG image returned should be 400px. <3> Set the y-axis lower limit to 0 so that random perturbation in the walk does not look so dramatic. === Timers While reading directly from a `Timer` returns a `double`, the underlying value is stored in https://github.com/netflix/spectator[Spectator] as a nanosecond-precise `long`. What precision is lost by converting to a `double` in the `Timer` interface does not affect a system like Atlas, because it has been configured to read measurements from the underlying Spectator `Timer` that Micrometer is hiding from you. The Spectator Atlas `Timer` produces four time series, each with a different `statistic` tag: * `count`: Rate of calls per second. * `totalTime`: Rate of total time per second. * `totalOfSquares`: Rate of total time squared per second (useful for standard deviation). * `max`: The maximum amount recorded. Therefore, you can achieve a throughput (requests/second) line with the following query: ```http name,timer,:eq,statistic,count,:eq,:and ``` Notice that `statistic` is just a dimension that can be drilled down and selected like any other. Furthermore, `totalTime/count` represents average latency and can be selected with a short-hand `:dist-avg` query, which selects the `totalTime` and `count` time series and performs the division for us: ```http name,timer,:eq,:dist-avg ``` In the preceding example, you can see these two lines plotted on a single dual-axis graph. .Timer over a simulated service. image::implementations/atlas-timer.png[Atlas-rendered timer] === Long Task Timers Suppose we had a task that took two minutes to complete when it was expected to complete in less than 70 seconds. A key benefit of long task timers is the ability to receive an alert at the first reporting interval after we have exceeded the threshold. With a regular timer, we would not receive an alert until the first reporting interval after the process completed. If we had a ten-second publishing interval, the regular timer alert would arrive almost a minute after the long task timer alert. .Simulated back-to-back long tasks. image::implementations/atlas-long-task-timer.png[Atlas-rendered long task timer] [source, http] ---- GET /api/v1/graph? q= name,longTaskTimer,:eq,statistic,duration,:eq,:and, <1> :dup, 70,:gt,:vspan,f00,:color,40,:alpha,alerted,:legend, <2> 70,f00,:color,alert+threshold,:legend <3> &tz=US/Central &s=e-15m &w=400 &l=0 &title=Peaks+of+Long+Tasks &ylabel=time Host: localhost:7101 ---- <1> A representation of long tasks that are happening back-to-back. <2> A vertical span that appears whenever the long task exceeds our threshold of 70 seconds. So that it does not overwhelm the graph, we also decrease the opacity of the vspan. <3> Plot the threshold of 70 seconds as a separate line. --- ### Modules/ROOT/Pages/Implementations/Azure Monitor = Micrometer Azure Monitor :sectnums: :system: azure-monitor Azure Monitor is a dimensional time-series SaaS with built-in dashboarding. include::_install.adoc[] == Configuring The following example configures a Micrometer Azure Monitor: [source,java] ---- AzureMonitorConfig azureMonitorConfig = new AzureMonitorConfig() { @Override public String instrumentationKey() { return MY_KEY; } @Override public String get(String key) { return null; } }; MeterRegistry registry = new AzureMonitorMeterRegistry(azureMonitorConfig, Clock.SYSTEM); ---- `AzureMonitorConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.azure-monitor` directly to the `AzureMonitorConfig`: [source,yml] ---- management.metrics.export.azure-monitor: instrumentation-key: YOURKEY # You will probably want disable Azure Monitor publishing in a local development profile. enabled: true # The interval at which metrics are sent to Azure Monitor. The default is 1 minute. step: 1m ---- --- ### Modules/ROOT/Pages/Implementations/Cloudwatch = Micrometer CloudWatch Tommy Ludwig :sectnums: :system: cloudwatch2 https://aws.amazon.com/cloudwatch/[Amazon CloudWatch] is a dimensional time-series SaaS on Amazon's cloud. include::_install.adoc[] == Configuring The following example configures Micrometer CloudWatch: [source,java] ---- CloudWatchConfig cloudWatchConfig = new CloudWatchConfig() { @Override public String get(String s) { return null; } @Override public String namespace() { return "mynamespace"; } }; MeterRegistry meterRegistry = new CloudWatchMeterRegistry(cloudWatchConfig, Clock.SYSTEM, CloudWatchAsyncClient.create()); ---- You can provide your own `CloudWatchAsyncClient` to the constructor of the registry. `CloudWatchConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, https://docs.awspring.io/spring-cloud-aws/docs/3.1.0/reference/html/index.html#cloudwatch-metrics[Micrometer support in Spring Cloud AWS] binds properties prefixed with `management.metrics.export.cloudwatch` directly to the `CloudWatchConfig`: [source,yml] ---- management.metrics.export.cloudwatch: namespace: YOURNAMESPACE # You will probably want to disable publishing in a local development profile. enabled: true # The interval at which metrics are sent to CloudWatch. The default is 1 minute. step: 1m ---- --- ### Modules/ROOT/Pages/Implementations/Datadog = Micrometer Datadog :sectnums: :system: datadog Datadog is a dimensional time-series SaaS with built-in dashboarding and alerting. == Installation and Configuration Micrometer supports shipping metrics to Datadog directly by using its HTTP API or by using DogStatsD through the xref:./statsD.adoc[StatsD registry]. If you can choose between the two, the API approach is far more efficient. NOTE: If you encounter a rate limit problem with the Datadog API approach, try the DogStatsD approach or one of https://docs.datadoghq.com/metrics/guide/micrometer/[alternatives that are described in the Datadog documentation]. It is recommended to use the BOM provided by Micrometer (or your framework if any), you can see how to configure it xref:../installing.adoc[here]. The examples below assume you are using a BOM. === Direct to Datadog API Approach ==== Gradle After the BOM is xref:../installing.adoc[configured], add the following dependency: [source,groovy] ---- implementation 'io.micrometer:micrometer-registry-datadog' ---- NOTE: The version is not needed for this dependency since it is defined by the BOM. ==== Maven After the BOM is xref:../installing.adoc[configured], add the following dependency: [source,xml] ---- io.micrometer micrometer-registry-datadog ---- NOTE: The version is not needed for this dependency since it is defined by the BOM. Metrics are rate-aggregated and pushed to `datadoghq` on a periodic interval. Rate aggregation performed by the registry yields datasets that are similar to those produced by `dogstatsd`. [source, java] ---- DatadogConfig config = new DatadogConfig() { @Override public Duration step() { return Duration.ofSeconds(10); } @Override public String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new DatadogMeterRegistry(config, Clock.SYSTEM); ---- `DatadogConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration through properties. For example, Spring Boot's Micrometer support binds properties directly to the `DatadogConfig`. See the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.export.datadog[Datadog] section in the Spring Boot reference documentation. `DatadogConfig.hostTag()` specifies a tag key that is mapped to https://docs.datadoghq.com/api/v1/metrics/#submit-metrics[the `host` field] when shipping metrics to Datadog. For example, if `DatadogConfig.hostTag()` returns `host`, the tag having `host` as its key is used. You can set the tag by using common tags, as follows: [source,java] ---- registry.config().commonTags("host", "my-host"); ---- `uri` is an important property to configure. The default value is `https://api.datadoghq.com`. Depending on the Datadog site (region), the api endpoint will be different. To find your the correct `uri` for your account, do the following: 1. Read about the https://docs.datadoghq.com/getting_started/site/[Datadog site]. 2. Go to the https://docs.datadoghq.com/api/latest/metrics/[Metrics API reference] and select your own option from the "DATADOG SITE" dropdown. 3. Check any API request's endpoint. For example, for the `US5` site, the correct API endpoint is `https://api.us5.datadoghq.com`. For the `US3` site, it is `https://api.us3.datadoghq.com`. === Through DogStatsD Approach ==== Gradle After the BOM is xref:../installing.adoc[configured], add the following dependency: [source,groovy] ---- implementation 'io.micrometer:micrometer-registry-statsd' ---- NOTE: The version is not needed for this dependency since it is defined by the BOM. ==== Maven After the BOM is xref:../installing.adoc[configured]d, add the following dependency: [source,xml] ---- io.micrometer micrometer-registry-statsd ---- NOTE: The version is not needed for this dependency since it is defined by the BOM. Metrics are immediately shipped to DogStatsD using Datadog's flavor of the StatsD line protocol. `java-dogstatsd-client` is _not_ needed on the classpath for this to work, as Micrometer uses its own implementation. [source,java] ---- StatsdConfig config = new StatsdConfig() { @Override public String get(String k) { return null; } @Override public StatsdFlavor flavor() { return StatsdFlavor.DATADOG; } }; MeterRegistry registry = new StatsdMeterRegistry(config, Clock.SYSTEM); ---- If the `DD_ENTITY_ID` environment variable is properly set, Micrometer supports DogStatsD's https://docs.datadoghq.com/developers/dogstatsd/?tab=kubernetes#origin-detection-over-udp[origin detection over UDP] feature on Kubernetes. Micrometer, by default, publishes `Timer` meters to DogStatsD as the StatsD "`timing`" metric type, `ms`. These meters are sent to Datadog as https://docs.datadoghq.com/metrics/types/?tab=histogram#metric-types[histogram] type metrics. Also, by default, Micrometer publishes `DistributionSummary` meters as histogram type metrics. When `percentileHistogram` is enabled for the meter, Micrometer sends `Timer` and `DistributionSummary` meters as Datadog https://docs.datadoghq.com/metrics/distributions[Distributions] to DogStatsD. You can make a `DistributionSummary` with `percentileHistogram` enabled, as follows: [source,java] ---- DistributionSummary responseSizeSummary = DistributionSummary.builder("http.server.response.size") .baseUnit("bytes") .publishPercentileHistogram() .register(registry); ---- --- ### Modules/ROOT/Pages/Implementations/Dynatrace = Micrometer Dynatrace :sectnums: :system: dynatrace https://www.dynatrace.com/[*Dynatrace*] is a Software Intelligence Platform featuring application performance monitoring (APM), artificial intelligence for operations (AIOps), IT infrastructure monitoring, digital experience management (DEM), and digital business analytics capabilities. It can ingest multi-purpose dimensional time-series data and has built-in dashboarding. Both SaaS and self-hosted (Managed) deployments are offered. include::_install.adoc[] == Configuring For setting up new integrations with Dynatrace, we recommend using the latest version of the https://docs.dynatrace.com/docs/shortlink/api-metrics-v2[Dynatrace Metrics API] (currently version 2). If you use Micrometer with Spring Boot, see the https://docs.dynatrace.com/docs/shortlink/micrometer-metrics-ingest[Dynatrace documentation] and the https://docs.spring.io/spring-boot/docs/current/reference/html/actuator.html#actuator.metrics.export.dynatrace[Spring Boot documentation]. Dynatrace provides different ways of setting up integrations: === Using Dynatrace auto-configuration (preferred) [[bookmark-auto-configuration]] Dynatrace auto-configuration is available for hosts that are monitored by OneAgent or by the Dynatrace Operator for Kubernetes. If Dynatrace OneAgent is installed on the host running Micrometer, metrics can be exported directly by using OneAgent without having to specify an endpoint URI or API token. If running in Kubernetes with the Dynatrace operator installed, the registry picks up your endpoint URI and API token from the operator instead. In this case, there is no need to configure anything, so you can use the following code in your project to export Micrometer metrics to Dynatrace: [source,java] ---- DynatraceConfig dynatraceConfig = new DynatraceConfig() { @Override public @Nullable String get(String k) { // This method of the interface is used by the other configuration methods and needs to be // implemented here. Returning null accepts the defaults for the other configuration items. return null; } }; MeterRegistry registry = new DynatraceMeterRegistry(dynatraceConfig, Clock.SYSTEM); ---- If you use Micrometer 1.10.0 or above, you can also use the DEFAULT config to achieve the same with less code: [source,java] ---- MeterRegistry registry = new DynatraceMeterRegistry(DynatraceConfig.DEFAULT, Clock.SYSTEM); ---- You can also change other properties by creating an instance of the `DynatraceConfig` and overwriting the respective methods. For example, you can specify the exporter version, which defaults to `v2` unless a deviceId is set: [source,java] ---- DynatraceConfig dynatraceConfig = new DynatraceConfig() { @Override public DynatraceApiVersion apiVersion() { return DynatraceApiVersion.V2; } @Override public @Nullable String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new DynatraceMeterRegistry(dynatraceConfig, Clock.SYSTEM); ---- `DynatraceConfig` is an interface with a set of default methods. Spring Boot's Micrometer support binds properties prefixed with `management.dynatrace.metrics.export` directly to the `DynatraceConfig`. NOTE: Property names for binding attributes from Spring Boot have changed in Spring Boot version 3.0.0. If you use a Spring Boot version before 3.0.0, use `management.metrics.export.dynatrace` instead of `management.dynatrace.metrics.export`. Using Spring Boot Micrometer support allows configuring the Dynatrace exporter by using <>. When using Micrometer with Spring Boot, you need not instantiate the `DynatraceMeterRegistry` manually, as Spring Boot does it automatically for you. All configuration options that can be set by overwriting methods can also be set through Spring Boot properties, and adding a separate MeterRegistry can lead to metrics not being exported, as auto-configuration might break. To use the Dynatrace metrics exporter for Micrometer in your Spring Boot project, include the `runtimeOnly 'io.micrometer:micrometer-registry-dynatrace'` dependency. In this default configuration, metrics are exported to the local OneAgent or Kubernetes operator-provided endpoint. === Using a Custom Endpoint If auto-configuration is not available on the host, both the Dynatrace Metrics API v2 endpoint and an API token have to be specified. The https://docs.dynatrace.com/docs/shortlink/api-authentication[Dynatrace API token documentation] contains more information on how to create an API token. To ingest metrics, the 'Ingest metrics' (`metrics.ingest`) permission is required on the token. We recommend limiting scope to only this permission. [source,java] ---- DynatraceConfig dynatraceConfig = new DynatraceConfig() { @Override public DynatraceApiVersion apiVersion() { // not strictly required, but makes the code more clear/explicit return DynatraceApiVersion.V2; } @Override public String uri() { // The endpoint of the Dynatrace Metrics API v2 including path, e.g.: // "https://{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest" String endpoint = System.getenv("ENVVAR_METRICS_INGEST_URL"); return endpoint != null ? endpoint : DynatraceConfig.super.uri(); } @Override public String apiToken() { // should be read from a secure source String token = System.getenv("ENVVAR_METRICS_INGEST_TOKEN"); return token != null ? token : ""; } @Override public @Nullable String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new DynatraceMeterRegistry(dynatraceConfig, Clock.SYSTEM); ---- These properties can also be set through Spring Boot, by using property or yaml files. You can also reference environment variables by using the Spring property placeholders (such as `management.dynatrace.metrics.export.uri: $\{DT_METRICS_INGEST_URL}`). NOTE: `v2` is used as the default API version unless a `deviceId` is set, as described <>. [source,yml] ---- # For Spring Boot 3.0.0 and above: management.dynatrace.metrics.export: # For Spring Boot versions below 3.0.0, use the line below instead of the line above: # management.metrics.export.dynatrace: # for SaaS: https://\{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest # for managed deployments: https://{your-domain}/e/\{your-environment-id}/api/v2/metrics/ingest uri: YOUR_METRICS_INGEST_URL # should be read from a secure source api-token: YOUR_METRICS_INGEST_TOKEN ---- === Meter metadata Starting with Micrometer 1.12.0, the Dynatrace registry v2 exports meter metadata to Dynatrace. Currently supported types of metadata are *unit* (called "base unit" in Micrometer) and *description*. No changes are required to start exporting Metadata to Dynatrace - upgrading to version 1.12.0 or above is enough. Find more information about metrics metadata in the https://docs.dynatrace.com/docs/shortlink/metric-ingestion-protocol#metadata[Dynatrace documentation]. The export of metrics metadata can be disabled by setting the `exportMeterMetadata` property on the `DynatraceConfig` (see <> below) to `false`. == API Versions === API v2 [[bookmark-apiv2]] When the API version is configured to `v2`, the registry sends data by using the https://docs.dynatrace.com/docs/shortlink/api-metrics-v2[Metrics API v2]. To maintain backwards compatibility, when a `deviceId` is set (which is required for `v1` and not used in `v2`), `v1` is used as the default. Otherwise, the version defaults to `v2` and does not have to be set explicitly. With no endpoint URI and token set, metrics are exported to the local OneAgent endpoint or, if running in Kubernetes with the Dynatrace operator installed, to the endpoint provided by the operator. If you do not need auto-configuration, you can specify the endpoint and token explicitly, to export metrics to that specific endpoint. Explicitly specifying these will overwrite auto-configuration. *Minimal Configuration with Dynatrace Auto-configuration* In the minimal configuration <> (no URI or API token), the v2 registry tries to retrieve the endpoint provided by the Dynatrace Kubernetes operator. If the operator is not set up or does not provide this information, the exporter tries to send metrics to the https://docs.dynatrace.com/docs/shortlink/local-api[local OneAgent metrics ingest endpoint]. Note that this only works if a OneAgent is running on the host and the https://docs.dynatrace.com/docs/shortlink/local-api#enable-the-oneagent-metric-api[local OneAgent Metric API] is available. If the ingestion port for the local OneAgent was changed to a custom one, the full endpoint URI has to be provided for the URI property (with API token left empty). *Configuration with URI and API Token* If no auto-configuration is available or the metrics should be sent to a different endpoint (such as a different tenant), you can configure the Dynatrace v2 exporter with an explicit endpoint URI and an https://docs.dynatrace.com/docs/shortlink/api-authentication[API token]. The https://docs.dynatrace.com/docs/shortlink/api-authentication[API token] must have the https://docs.dynatrace.com/docs/dynatrace-api/basics/dynatrace-api-authentication#token-scopes["Ingest metrics"] (`metrics.ingest`) permission set. We recommend limiting the scope to only this permission. You must specify the entire Metrics v2 API endpoint URI, including its path -- that is, with the `/api/v2/metrics/ingest` path on SaaS and managed deployments, or `/metrics/ingest` for OneAgent endpoints as mentioned in the https://docs.dynatrace.com/docs/shortlink/api-metrics-v2-post-datapoints[documentation]. *Properties Available in the v2 Exporter* [[bookmark-available-properties]] When using the https://docs.dynatrace.com/docs/shortlink/api-metrics-v2[Dynatrace metrics API v2], you can set the following properties: [source,java] ---- DynatraceConfig dynatraceConfig = new DynatraceConfig() { @Override public DynatraceApiVersion apiVersion() { return DynatraceApiVersion.V2; } @Override public String uri() { // The endpoint of the Dynatrace Metrics API v2 including path, e.g.: // "https://{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest". String endpoint = System.getenv("ENVVAR_METRICS_INGEST_URL"); return endpoint != null ? endpoint : DynatraceConfig.super.uri(); } @Override public String apiToken() { // should be read from a secure source String token = System.getenv("ENVVAR_METRICS_INGEST_TOKEN"); return token != null ? token : ""; } @Override public String metricKeyPrefix() { // will be prepended to all metric keys return "your.desired.prefix"; } @Override public boolean enrichWithDynatraceMetadata() { return true; } @Override public Map defaultDimensions() { // create and return a map containing the desired key-value pairs. Map dims = new HashMap<>(); dims.put("dimensionKey", "dimensionValue"); return dims; } // Only available in Micrometer 1.9.0 and above @Override public boolean useDynatraceSummaryInstruments() { return false; } // Only available in Micrometer 1.12.0 and above @Override public boolean exportMeterMetadata() { return true; } @Override public @Nullable String get(String k) { return null; // accept the rest of the defaults } }; ---- You can also set these properties in Spring Boot configuration files: [source,yml] ---- management.dynatrace.metrics.export: # Required only if not using the OneAgent endpoint # For SaaS: https://{your-environment-id}.live.dynatrace.com/api/v2/metrics/ingest # For managed deployments: https://{your-domain}/e/{your-environment-id}/api/v2/metrics/ingest uri: YOUR_METRICS_INGEST_URL # should be read from a secure source api-token: YOUR_METRICS_INGEST_TOKEN # These properties can only be used with the v2 exporter. v2: # Sets a prefix that is prepended to each exported metric key. metric-key-prefix: my.metric.key.prefix # If set to true and a local OneAgent or operator is running, retrieves metadata # and adds it as additional dimensions to all data points (default: true) enrich-with-dynatrace-metadata: true # Sets an arbitrary number of key-value pairs as default dimensions. # Micrometer tags will overwrite these dimensions, if they have the same key. # Each exported metric will contain these dimensions. default-dimensions: key1: "value1" key2: "value2" # (since 1.9.0) Whether or not to use the Dynatrace-specific summary instruments. (default: true) # This should only be disabled if problems with existing instrumentation are discovered after upgrading to 1.9.0. # Set to false, this will restore the previous (1.8.x) behavior for Timers and DistributionSummaries. use-dynatrace-summary-instruments: true # (since 1.12.0) Determines whether meter metadata (unit, description) should be exported. export-meter-metadata: true # The export interval in which metrics are sent to Dynatrace (default: 60s). step: 60s ---- For more information about the metadata picked up by the Dynatrace metadata enrichment feature, see https://docs.dynatrace.com/docs/shortlink/enrichment-files[the Dynatrace documentation]. In Micrometer 1.9.0, Dynatrace-specific summary instruments (`DynatraceTimer` and `DynatraceDistributionSummary`) were introduced. These specialized instruments are tailored to the Dynatrace metrics ingest, and prevent the creation of invalid metrics. They are available from version 1.9.0 and are used, by default, as a drop-in replacement. No action is needed from users upgrading to 1.9.0. If there is a discrepancy in the observed metrics, you can return to the previous behavior by setting the `useDynatraceSummaryInstruments` toggle to `false`. === API v1 (Legacy) [[bookmark-apiv1]] CAUTION: The Dynatrace v1 API is deprecated. It will be removed in the future. Please use the v2 API instead. See https://docs.dynatrace.com/docs/discover-dynatrace/references/dynatrace-api/environment-api/metric-v1[Dynatrace API Docs]. When the `apiVersion` is configured to be `v1`, the registry sends data by using the https://docs.dynatrace.com/docs/shortlink/api-custom-metrics[Dynatrace Timeseries API v1 for Custom Metrics]. If a `deviceId` is specified, it defaults to `v1` for backwards compatibility with earlier setups. The `device-id` property is required for `v1` and not used in `v2`. Existing setups continue to work when updating to newer versions of Micrometer. The reported metrics are assigned to https://docs.dynatrace.com/docs/shortlink/api-custom-device-report-metric[Custom Devices] in Dynatrace. For the v1 API, do not specify the ingest path, but only the base URL of your environment, such as `uri: https://\{your-environment-id}.live.dynatrace.com`: [source,java] ---- DynatraceConfig dynatraceConfig = new DynatraceConfig() { @Override public String uri() { // The Dynatrace environment URI without any path, e.g.: // https://{your-environment-id}.live.dynatrace.com return MY_DYNATRACE_URI; } @Override public String apiToken() { // should be read from a secure source return MY_TOKEN; } @Override public String deviceId() { return MY_DEVICE_ID; } @Override public @Nullable String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new DynatraceMeterRegistry(dynatraceConfig, Clock.SYSTEM); ---- [source,yml] ---- management.dynatrace.metrics.export: # For v1 export, do not append a path to the endpoint URL, e.g.: # For SaaS: https://{your-environment-id}.live.dynatrace.com # For managed deployments: https://{your-domain}/e/{your-environment-id} uri: https://{your-environment-id}.live.dynatrace.com # should be read from a secure source api-token: MY_TOKEN # When setting the device id, metrics will be exported to the v1 timeseries endpoint # Using just device-id (without the v1 prefix) is deprecated, but will work to maintain backwards compatibility. v1: device-id: sample # To disable Dynatrace publishing, e.g. in a local development profile, use: # enabled: false # The interval at which metrics are sent to Dynatrace. The default is 1 minute. step: 1m ---- --- ### Modules/ROOT/Pages/Implementations/Elastic = Micrometer Elastic :sectnums: :system: elastic Elasticsearch is an open source search and analytics platform. Metrics stored in Elasticsearch can be visualized in Kibana. include::_install.adoc[] == Configuring The following example configures an ElasticSearch instance: [source,java] ---- ElasticConfig elasticConfig = new ElasticConfig() { @Override public @Nullable String get(String k) { return null; } }; MeterRegistry registry = new ElasticMeterRegistry(elasticConfig, Clock.SYSTEM); ---- `ElasticConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.elastic` directly to the `ElasticConfig`: [source,yml] ---- management.metrics.export.elastic: # You will probably want disable Elastic publishing in a local development profile. enabled: true # The interval at which metrics are sent to Elastic. The default is 1 minute. step: 1m # The index to store metrics in, defaults to "micrometer-metrics" index: micrometer-metrics ---- == Elastic APM agent integration If you are using the Elastic APM agent, it can automatically collect metrics from Micrometer `MeterRegistry` instances. If you want only metrics collected by the Elastic APM agent and not shipped anywhere else, you can use the `SimpleMeterRegistry`. See the https://www.elastic.co/guide/en/apm/agent/java/current/metrics.html#metrics-micrometer[Elastic docs] for more details. --- ### Modules/ROOT/Pages/Implementations/Ganglia = Micrometer Ganglia :sectnums: :system: ganglia Ganglia is an aging hierarchical metrics system which enjoyed wide popularity in Linux system monitoring and is still in place in many organizations. It originated in the early 2000s at the University of California, Berkeley. NOTE: The `micrometer-registry-ganglia` module uses the https://github.com/ganglia/gmetric4j[gmetric4j] library, which contains classes generated by the LGPL licensed https://sourceforge.net/projects/remotetea/[remotetea project]. include::_install.adoc[] == Configuring The following example configures a Ganglia instance: [source,java] ---- GangliaConfig gangliaConfig = new GangliaConfig() { @Override public String host() { return "mygraphitehost"; } @Override public String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new GangliaMeterRegistry(gangliaConfig, Clock.SYSTEM); ---- Micrometer uses Dropwizard Metrics as the underlying instrumentation library when recording metrics destined for Ganglia. `GangliaConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.ganglia` directly to the `GangliaConfig`: [source,yml] ---- management.metrics.export.ganglia: # The location of your Ganglia server host: mygraphitehost # You will probably want to conditionally disable Ganglia publishing in local development. enabled: true # The interval at which metrics are sent to Ganglia. The default is 1 minute. step: 1m ---- include::hierarchical-name-mapping.adoc[] == Graphing This section serves as a quick start to rendering useful representations in Ganglia for metrics originating in Micrometer. === Counters Ganglia counters measure mean throughput and one-, five-, and fifteen-minute exponentially-weighted moving average throughputs. .A Ganglia rendered graph of the random walk counter. image::implementations/ganglia-counter.png[Ganglia-rendered counter] --- ### Modules/ROOT/Pages/Implementations/Graphite = Micrometer Graphite :sectnums: :system: graphite Graphite is one of the most popular current hierarchical metrics systems backed by a fixed-size database, similar in design and purpose to RRDtool. It originated at Orbitz in 2006 and was open sourced in 2008. include::_install.adoc[] == Configuring The following example configures a Graphite instance: [source,java] ---- GraphiteConfig graphiteConfig = new GraphiteConfig() { @Override public String host() { return "mygraphitehost"; } @Override public String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new GraphiteMeterRegistry(graphiteConfig, Clock.SYSTEM, HierarchicalNameMapper.DEFAULT); ---- Micrometer uses Dropwizard Metrics as the underlying instrumentation library when recording metrics destined for Graphite. `GraphiteConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Spring Boot's Micrometer support binds its application properties directly to the `GraphiteConfig`. See https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.export.graphite[Spring Boot Reference Documentation] for details. == Graphite Tag Support As of Micrometer version 1.4.0, Micrometer supports exporting Graphite metrics by using tags instead of the traditional hierarchical format. By default, metrics are exported by using the tag format, unless any `tagsAsPrefix` values are configured. https://graphite.readthedocs.io/en/latest/tags.html[Tag support] was added to Graphite in the 1.1.0 Graphite release. If you wish to revert to the traditional hierarchical format, ensure that the `graphiteTagsEnabled` config value is set to `false`. The following documentation sections on hierarchical name mapping and metrics prefixing are only applicable if tag support is disabled. include::hierarchical-name-mapping.adoc[] == Prefixing Your Metrics To add a prefix to all metrics that go to graphite, use the `GraphiteConfig#tagsAsPrefix` configuration option. This option applies the tag value of a set of common tags as a prefix. For example, if `tagsAsPrefix` contains `application`, and a meter named `myTimer` is created with a tag of `application=APPNAME`, it appears in Graphite as `APPNAME.myTimer`. Generally, when you use `tagsAsPrefix`, you should add common tags to the registry so that the tags are present on all meters that belong to that registry: [source,java] ---- @Bean public MeterRegistryCustomizer commonTags() { return r -> r.config().commonTags("application", "APPNAME"); } ---- We do it this way because, generally, a tag prefix in Graphite is correlated to a common tag elsewhere. Prefixes tend to be something like app name or host. By applying those values as common tags, you make your metrics more portable (if you ever switch to a dimensional monitoring system, you are set). You can use this when the order of the prefix matters. Micrometer always sorts tags, but the order of tag keys in `tagsAsPrefix` is preserved, so adding `host` and `application` to `tagsAsPrefix` results in a prefixed metric, such as `HOST.APP.myCounter`. To meet your specific naming needs, you can also provide a custom hierarchical name mapper when creating `GraphiteMeterRegistry`, as follows: [source,java] ---- GraphiteMeterRegistry r = new GraphiteMeterRegistry( GraphiteConfig.DEFAULT, Clock.SYSTEM, (id, convention) -> "prefix." + HierarchicalNameMapper.DEFAULT.toHierarchicalName(id, convention)); ---- NOTE: If you use a custom `HierarchicalNameMapper`, `tagsAsPrefix` is ignored. == Further Customizing the `GraphiteReporter` We give you the option to configure `GraphiteReporter` yourself if you need further customization. To do so, use this constructor and provide your own `GraphiteReporter`: [source,java] ---- GraphiteMeterRegistry(GraphiteConfig config, Clock clock, HierarchicalNameMapper nameMapper, MetricRegistry metricRegistry, GraphiteReporter reporter) ---- == Graphing This section serves as a quick start to rendering useful representations in Graphite for metrics originating in Micrometer. === Counters Graphite counters measure mean throughput and one-, five-, and fifteen-minute exponentially-weighted moving average throughputs. .A Graphite rendered graph of the random walk counter. image::implementations/graphite-counter.png[Graphite-rendered counter] --- ### Modules/ROOT/Pages/Implementations/Hierarchical Name Mapping == Hierarchical name mapping Micrometer provides a `HierarchicalNameMapper` interface that governs how a dimensional meter ID is mapped to flat hierarchical names. The default (`HierarchicalNameMapper.DEFAULT`) sorts tags alphabetically by key and appends tag key/value pairs to the base meter name with '.' -- for example, `http_server_requests.method.GET.response.200`. The name and tag keys have the registry's naming convention applied to them first. If there is something special about your naming scheme that you need to honor, you can provide your own `HierarchicalNameMapper` implementation. The most common cause of a custom mapper comes from a need to prefix something to the front of every metric (generally something like `app..http_server_requests.method.GET.response.200`). --- ### Modules/ROOT/Pages/Implementations/Humio = Micrometer Humio :sectnums: :system: humio Humio is a dimensional time-series SaaS with built-in dashboarding. include::_install.adoc[] == Configuring The following example configures a Humio instance: [source,java] ---- HumioConfig humioConfig = new HumioConfig() { @Override public String apiToken() { return MY_TOKEN; } @Override public @Nullable String get(String k) { return null; } }; MeterRegistry registry = new HumioMeterRegistry(humioConfig, Clock.SYSTEM); ---- `HumioConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.humio` directly to the `HumioConfig`: [source,yml] ---- management.metrics.export.humio: api-token: YOURKEY # You will probably want disable Humio publishing in a local development profile. enabled: true # The interval at which metrics are sent to Humio. The default is 1 minute. step: 1m # The cluster Micrometer will send metrics to. The default is "https://cloud.humio.com" uri: https://myhumiohost ---- == Graphing This section serves as a quick start to rendering useful representations in Humio for metrics originating in Micrometer. === Timers The Humio implementation of `Timer` produces four fields in Humio: * `count`: Rate of total time/second. * `sum`: Rate of calls/second. * `max`: A sliding window that shows the maximum amount recorded. * `avg`: A non-aggregable average for only this set of tag values. The following query constructs a dimensionally aggregable average latency per URI: [source, text] ---- name = http_server_requests | timechart(uri, function=max(avg)) ---- .Timer over a simulated service. image::implementations/humio-timer.png[Humio-rendered timer] --- ### Modules/ROOT/Pages/Implementations/Influx = Micrometer Influx :sectnums: :system: influx The InfluxData suite of tools supports real-time stream processing and storage of time-series data. It supports downsampling, automatically expiring and deleting unwanted data, as well as backup and restore. The InfluxMeterRegistry supports the 1.x InfluxDB API as well as the v2 API. == Installation and Configuration Micrometer supports shipping metrics to InfluxDB directly or through Telegraf through the StatsD registry. It is recommended to use the BOM provided by Micrometer (or your framework if any), you can see how to configure it xref:../installing.adoc[here]. The examples below assume you are using a BOM. === Direct to InfluxDB ==== Gradle After the BOM is xref:../installing.adoc[configured], add the following dependency: [source,groovy] ---- implementation 'io.micrometer:micrometer-registry-influx' ---- NOTE: The version is not needed for this dependency since it is defined by the BOM. ==== Maven After the BOM is xref:../installing.adoc[configured], add the following dependency: [source,xml] ---- io.micrometer micrometer-registry-influx ---- NOTE: The version is not needed for this dependency since it is defined by the BOM. Metrics are rate-aggregated and pushed to InfluxDB on a periodic interval. Rate aggregation performed by the registry yields datasets that are quite similar to those produced by Telegraf. The following example configures a meter registry for InfluxDB: .InfluxDB 1.x configuration example [source, java] ---- InfluxConfig config = new InfluxConfig() { @Override public Duration step() { return Duration.ofSeconds(10); } @Override public String db() { return "mydb"; } @Override public String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new InfluxMeterRegistry(config, Clock.SYSTEM); ---- To ship metrics to InfluxDB 2.x, make sure to configure the `org` and `bucket` to which to write the metrics, as well as the authentication `token`: .InfluxDB v2 configuration example [source, java] ---- InfluxConfig config = new InfluxConfig() { @Override public String org() { return "myorg"; } @Override public String bucket() { return "app-metrics"; } @Override public String token() { return "auth_token_here"; // FIXME: This should be securely bound rather than hard-coded, of course. } @Override public String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new InfluxMeterRegistry(config, Clock.SYSTEM); ---- `InfluxConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.influx` directly to the `InfluxConfig`: [source, yaml] ---- management.metrics.export.influx: api-version: v2 # API version of InfluxDB to use. Defaults to 'v1' unless an org is configured. If an org is configured, defaults to 'v2'. auto-create-db: true # Whether to create the InfluxDB database if it does not exist before attempting to publish metrics to it. InfluxDB v1 only. (Default: true) batch-size: 10000 # Number of measurements per request to use for this backend. If more measurements are found, then multiple requests will be made. (Default: 10000) bucket: mybucket # Bucket for metrics. Use either the bucket name or ID. Defaults to the value of the db property if not set. InfluxDB v2 only. compressed: true # Whether to enable GZIP compression of metrics batches published to InfluxDB. (Default: true) connect-timeout: 1s # Connection timeout for requests to this backend. (Default: 1s) consistency: one # Write consistency for each point. (Default: one) db: mydb # Database to send metrics to. InfluxDB v1 only. (Default: mydb) enabled: true # Whether exporting of metrics to this backend is enabled. (Default: true) num-threads: 2 # Number of threads to use with the metrics publishing scheduler. (Default: 2) org: myorg # Org to write metrics to. InfluxDB v2 only. password: mysecret # Login password of the InfluxDB server. InfluxDB v1 only. read-timeout: 10s # Read timeout for requests to this backend. (Default: 10s) retention-policy: my_rp # Retention policy to use (InfluxDB writes to the DEFAULT retention policy if one is not specified). InfluxDB v1 only. step: 1m # Step size (i.e. reporting frequency) to use. (Default: 1m) token: AUTH_TOKEN_HERE # Authentication token to use with calls to the InfluxDB backend. For InfluxDB v1, the Bearer scheme is used. For v2, the Token scheme is used. uri: http://localhost:8086 # URI of the InfluxDB server. (Default: http://localhost:8086) user-name: myusername # Login user of the InfluxDB server. InfluxDB v1 only. ---- === Through Telegraf Telegraf is a StatsD agent that expects a modified flavor of the StatsD line protocol. ==== Gradle After the BOM is xref:../installing.adoc[configured], add the following dependency: [source,groovy] ---- implementation 'io.micrometer:micrometer-registry-statsd' ---- NOTE: The version is not needed for this dependency since it is defined by the BOM. ==== Maven After the BOM is xref:../installing.adoc[configured], add the following dependency: [source,xml] ---- io.micrometer micrometer-registry-statsd ---- NOTE: The version is not needed for this dependency since it is defined by the BOM. Metrics are shipped immediately over UDP to Telegraf by using Telegraf's flavor of the StatsD line protocol: [source,java] ---- StatsdConfig config = new StatsdConfig() { @Override public String get(String k) { return null; } @Override public StatsdFlavor flavor() { return StatsdFlavor.Telegraf; } }; MeterRegistry registry = new StatsdMeterRegistry(config, Clock.SYSTEM); ---- --- ### Modules/ROOT/Pages/Implementations/Instana = Micrometer Instana Fabian Lange :sectnums: :system: instana Instana is an automatic application performance management and infrastructure monitoring system. == Installation and Configuration Instana automatically detects and reports all metrics without the need of any additional dependency or configuration. It does so by detecting all instances of `io.micrometer.core.instrument.MeterRegistry` and collecting all registered `io.micrometer.core.instrument.Meter` instances from them. You can run the Instana agent alongside your application by using Micrometer, and the Instana agent automatically monitors it. == Supported Metrics * **Timer**: The total time of recorded events, scaled to milliseconds. * **Counter**: The cumulative count since this counter was created. * **Gauge**: The current value. * **DistributionSummary**: The total number of all recorded events. * **LongTaskTimer**: The current number of tasks being executed. * **FunctionCounter**: The cumulative count since this counter was created. * **FunctionTimer**: The total time of all occurrences of the timed event. * **TimeGauge**: The current value, scaled to the appropriate base unit. The metrics show up on the Java Virtual Machine dashboard in Instana. You can configure alerting based on these metrics. --- ### Modules/ROOT/Pages/Implementations/Jmx = Micrometer JMX :sectnums: :system: jmx Micrometer provides a hierarchical mapping to JMX, primarily as a cheap and portable way to view metrics locally. Where JMX exporting is found in production, the same metrics are generally exported to another, more purpose-fit monitoring system. include::_install.adoc[] Micrometer also sometimes scrapes data from JMX beans for use in reporting metrics. This registry implementation is not needed for these uses. Rather, this module is strictly used to _export_ data to JMX. include::hierarchical-name-mapping.adoc[] == Counters JMX counters measure mean throughput and one-, five-, and fifteen-minute exponentially-weighted moving average throughputs. .The JMX rendered values of the random walk counter. image::implementations/jmx-counter.png[JMX-rendered counter] --- ### Modules/ROOT/Pages/Implementations/Kairos = Micrometer KairosDB :sectnums: :system: kairos KairosDB is a dimensional time-series database built on top of Cassandra. Charting can be accomplished in Grafana by using a link:https://docs.grafana.org/v4.0/datasources/kairosdb/[Kairos datasource]. include::_install.adoc[] == Configuring The following example configures KairosDB: [source,java] ---- KairosConfig kairosConfig = new KairosConfig() { @Override public @Nullable String get(String k) { return null; } }; MeterRegistry registry = new KairosMeterRegistry(kairosConfig, Clock.SYSTEM); ---- `KairosConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.kairos` directly to the `KairosConfig`: [source,yml] ---- management.metrics.export.kairos: # You will probably want disable Kairos publishing in a local development profile. enabled: true # The interval at which metrics are sent to Kairos. The default is 1 minute. step: 1m # Authentication may be required, depending on how you have Kairos configured user-name: MYUSER password: MYPASSWORD ---- --- ### Modules/ROOT/Pages/Implementations/New Relic = Micrometer New Relic :sectnums: :system: new-relic New Relic offers a dimensional monitoring system product called Insights. It includes a full UI and a query language called NRQL. New Relic Insights operates on a push model. Some features of NRQL assume that Insights receives a distinct event payload for every timing, count, and so on. Micrometer instead ships aggregates at a prescribed interval, letting your app's throughput scale without concern for event propagation to Insights becoming a bottleneck. NOTE: New Relic provides its own Micrometer `MeterRegistry` implementation based on dimensional metrics. It intends to supersede Micrometer's `NewRelicMeterRegistry` (which uses custom events in New Relic), because New Relic's dimensional metrics are a better fit for metrics than custom events. You can find more details in https://github.com/newrelic/micrometer-registry-newrelic[its GitHub repository]. include::_install.adoc[] == Configuring The following example configures New Relic: [source,java] ---- NewRelicConfig newRelicConfig = new NewRelicConfig() { @Override public String accountId() { return "MYACCOUNT"; } @Override public String apiKey() { return "MY_INSIGHTS_API_KEY"; } @Override public String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new NewRelicMeterRegistry(newRelicConfig, Clock.SYSTEM); ---- There are two distinct sources of API keys in New Relic. `NewRelicConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.newrelic` directly to the `NewRelicConfig`: [source,yml] ---- management.metrics.export.newrelic: account-id: MYACCOUNT api-key: MY_INSIGHTS_API_KEY # The interval at which metrics are sent to New Relic. See Duration.parse for the expected format. # The default is 1 minute. step: 1m ---- == Graphing This section serves as a quick start to rendering useful representations in New Relic for metrics originating in Micrometer. See the https://docs.newrelic.com/docs/insights/nrql-new-relic-query-language/using-nrql/introduction-nrql[New Relic NRQL docs] for a far more complete reference of what is possible in New Relic. === Timers At each publishing interval, the New Relic `Timer` produces a single event with the timer's name and several attributes: * `avg`: A mean latency for the publishing interval. * `count`: Throughput per second over the publishing interval. * `totalTime`: Total time per second over the publishing interval (used with `count`) to create aggregable means. Additionally, if any percentiles or SLO buckets are defined on the timer, additional events are produced: * `$\{name}.percentiles`: Micrometer calculated percentiles for the publishing interval. One event is produced for each percentile, with a tag of `phi` in the range of [0,1]. * `$\{name}.histogram`: One event is produced for each SLO boundary with a tag of 'le', indicating that it represents a cumulative count of events less than or equal to SLO boundaries over the publishing interval. To generate an aggregable view of latency in New Relic, divide `totalTime` by `count`: [source,sql] ---- SELECT sum(totalTime)/sum(count) as 'Average Latency', max(max) as 'Max' FROM timer since 30 minutes ago TIMESERIES auto ---- .Timer latency. image::implementations/new-relic-timer-latency.png[New Relic-rendered timer] The following example generates a throughput chart: [source,sql] ---- SELECT average(count) as 'Average Throughput' FROM timer since 30 minutes ago TIMESERIES auto ---- .Timer throughput. image::implementations/new-relic-timer-throughput.png[New Relic-rendered timer throughput] The following example generates a plot of client-side percentiles: [source,sql] ---- SELECT latest(value) from timerPercentile FACET phi since 30 minutes ago TIMESERIES auto ---- .Timer Percentiles. image::implementations/new-relic-timer-percentiles.png[New Relic-rendered percentiles] Note how these percentiles are _not aggregable_. We have selected the `latest(value)` function to display this chart (it is not correct to `average(value)` on a percentile value). The more dimensions you add to a timer, the less useful these values become. Finally, if you define SLO boundaries with the fluent builder for `Timer`, you can view throughput below certain SLO boundaries. In this example, we set SLO boundaries at 275 (yellow), 300 (red), and 500 (blue) milliseconds for a simulated `Timer` that is recording samples normally distributed around 250 ms. These counts represent the rate/second of samples less than or equal to each SLO boundary. [source,sql] ---- SELECT sum(value) from timerHistogram FACET le since 30 minutes ago TIMESERIES auto ---- .Timer SLO boundaries. image::implementations/new-relic-timer-sla.png[New Relic-rendered SLO boundaries] Where the lines converge at various points, it is evident that no sample exceeded the 275 ms SLO boundary. --- ### Modules/ROOT/Pages/Implementations/Otlp = Micrometer OTLP :sectnums: :system: otlp OpenTelemetry is a CNCF incubating project for providing standards for telemetry data. OpenTelemetry protocol (OTLP) is a vendor-neutral protocol that you can use to send data to various backends that support it. You can read the corresponding docs on how the metrics are ingested and can be visualized in the respective vendor docs. include::_install.adoc[] == Configuring The following example configures an OTLP registry: [source,java] ---- OtlpConfig otlpConfig = new OtlpConfig() { @Override public String get(final String key) { return null; } }; MeterRegistry registry = new OtlpMeterRegistry(otlpConfig, Clock.SYSTEM); ---- `OtlpConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source (for example, a simple `Map` can work), you can override the default configuration through properties. For example, Micrometer's Spring Boot support binds properties prefixed with `management.otlp.metrics.export` directly to the `OtlpConfig`: [source, yaml] ---- management: otlp: metrics: export: # Supported configs url: "https://otlp.example.com:4318/v1/metrics" batchSize: 15000 aggregationTemporality: "cumulative" compressionMode: "gzip" headers: header1: value1 step: 30s resourceAttributes: key1: value1 ---- 1. `url` - The address to which metrics are published. Environment variables `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and `OTEL_EXPORTER_OTLP_ENDPOINT` are also supported in the default implementation. If a value is not provided, it defaults to `http://localhost:4318/v1/metrics` 2. `batchSize` - number of ``Meter``s to include in a single payload sent to the backend. The default is 10,000. 3. `aggregationTemporality` - https://opentelemetry.io/docs/specs/otel/metrics/data-model/#temporality[Aggregation temporality, window=_blank] determines how the additive quantities are expressed, in relation to time. The environment variable `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE` is supported by the default implementation. The supported values are `cumulative` or `delta`. Defaults to `cumulative`. 4. `compressionMode` - Compression mode for metrics payload. The environment variables `OTEL_EXPORTER_OTLP_METRICS_COMPRESSION` and `OTEL_EXPORTER_OTLP_COMPRESSION` are supported by the default implementation. If both are set, `OTEL_EXPORTER_OTLP_METRICS_COMPRESSION` takes precedence. The supported values are `gzip` or `none`. Defaults to `none`. 5. `headers` - Additional headers to send with exported metrics. This can be used for authorization headers. By default, headers are loaded from the config. If that is not set, they can be taken from the environment variables `OTEL_EXPORTER_OTLP_HEADERS` and `OTEL_EXPORTER_OTLP_METRICS_HEADERS`. If a header is set in both the environmental variables, the header in the latter overrides the former. 6. `step` - the interval at which metrics will be published. The environment variable `OTEL_METRIC_EXPORT_INTERVAL` is supported by the default implementation. If a value is not provided, defaults to 1 minute. 7. `resourceAttributes` - https://opentelemetry.io/docs/specs/otel/resource/semantic_conventions/#service[Resource attributes, window=_blank] are used for all metrics published. By default, Micrometer adds the following resource attributes: [%autowidth] |=== |Key | Default value |telemetry.sdk.name |io.micrometer |telemetry.sdk.language |java |telemetry.sdk.version | (e.g.: 1.11.0) |service.name |unknown_service |=== If this config is empty, resource attributes are loaded from the `OTEL_RESOURCE_ATTRIBUTES` environmental variable. You can override `service.name` by setting the `OTEL_SERVICE_NAME` environmental variable, and this takes precedence over other configs. == Customize metrics sender The `OtlpMeterRegistry` has an `OtlpMetricsSender` abstraction for sending batches of metrics in OTLP protobuf format. By default, metrics from the OTLP registry are sent via HTTP to the URL specified by `OtlpConfig#url` using an `HttpUrlConnectionSender`. You may use a different `HttpSender` implementation by creating and configuring an instance of `OtlpHttpMetricsSender` such as in the following example using the `OkHttpSender` instead of the default `HttpUrlConnectionSender`: [source,java,subs=+attributes] ----- include::{include-java}/metrics/OtlpMeterRegistryCustomizationTest.java[tags=customizeHttpSender, indent=0] ----- You can also provide a custom implementation of `OtlpMetricsSender` that does not use `HttpSender` at all. `OtlpConfig#url` will be used as the address when the sender is called in the `OtlpMeterRegistry` `publish` method. For instance, if you made a gRPC implementation, you could configure it in the following way. Micrometer does not currently provide a gRPC implementation of `OtlpMetricsSender`. [source,java,subs=+attributes] ----- include::{include-java}/metrics/OtlpMeterRegistryCustomizationTest.java[tags=customGrpcSender, indent=0] ----- == Supported metrics https://opentelemetry.io/docs/specs/otel/metrics/data-model/#metric-points[Metric points, window=_blank] define the different data points that are supported in OTLP. Micrometer supports exporting the below data points in OTLP format, 1. https://opentelemetry.io/docs/specs/otel/metrics/data-model/#sums[Sums, window=_blank] 2. https://opentelemetry.io/docs/specs/otel/metrics/data-model/#gauge[Gauge, window=_blank] 3. https://opentelemetry.io/docs/specs/otel/metrics/data-model/#histogram[Histogram, window=_blank] 4. https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponentialhistogram[Exponential Histogram, window=_blank] 5. https://opentelemetry.io/docs/specs/otel/metrics/data-model/#summary-legacy[Summary, window=_blank] The following table maps OTLP data points and the Micrometer meters: [%autowidth] |=== |OTLP data point | Micrometer meter type |Sums |Counter, FunctionCounter |Gauge |Gauge, TimeGauge, MultiGauge |Histogram |Timer, DistributionSummary, LongTaskTimer, FunctionTimer (only sum and count are set) |Exponential Histogram^*^ |Timer, DistributionSummary |Summary |Timer, DistributionSummary, LongTaskTimer |=== __* - Histogram or Exponential Histogram is determined based on the configuration of the `OtlpMeterRegistry`. See below section for more info.__ *Note*: 1. `max` on the Histogram data point is supported only in delta aggregation temporality. This is because the values represented by cumulative min and max stabilize as more events are recorded and are less useful when recorded over application's lifecycle. 2. Currently, Micrometer only exports metadata for type `Meter` to OTLP. == Histograms and Percentiles Micrometer `Timer` and `DistributionSummary` support configuring xref:../concepts/histogram-quantiles.adoc[client-side percentiles and percentile histograms]. The OTLP specification terms the Summary data point (client-side percentiles) as legacy and not recommended for new applications. The Summary data point also cannot have min/max associated with it. Due to these reasons, Micrometer prefers exporting Timers and DistributionSummary as a Histogram data point. By default, a Timer/DistributionSummary without any additional percentile/histogram config is exported as a Histogram data point. However, by configuring the timer to generate only client-side percentiles, by using `publishPercentiles`, you can change this to a Summary data point that exports pre-calculated percentiles. When both `publishPercentiles` and (`publishPercentileHistogram` or `serviceLevelObjectives`) are configured, the Histogram data point is preferred and pre-calculated percentiles are *not* generated. The following table describes which data point is used with different configurations: [%autowidth] |=== |Configuration | OTLP data point | publishPercentiles | Summary | publishPercentileHistogram | Histogram/Exponential Histogram ^*^ | publishPercentiles and publishPercentileHistogram | Histogram/Exponential Histogram ^*^ | serviceLevelObjectives | Histogram | publishPercentiles and serviceLevelObjectives | Histogram |=== __* The configuration `histogramFlavorPerMeter` and `histogramFlavor` determine whether the OTLP DataPoint is a Histogram/Exponential Histogram.__ `OtlpMeterRegistry` supports 2 types of Histogram implementations (1.Explicit Bucket Histogram (or simply called Histogram), 2. Exponential Histogram) when `publishPercentileHistogram` is configured. The type is determined by `histogramFlavorPerMeter` and `histogramFlavor` in `OtlpConfig` used by the registry. When the implementation is the exponential histogram, additional configuration applies: 1. `maxScale` used to cap the maximum https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponential-scale[scale, window=_blank] used by the Exponential Histogram (defaults to 20). 2. `maxBuckets` determines the maximum number of https://opentelemetry.io/docs/specs/otel/metrics/data-model/#exponential-buckets[buckets, window=_blank] to be used for exponential histograms (defaults to 160). 3. `maxBucketsPerMeter` overrides `maxBuckets` for specific meter names, giving more fine-grained configurability. Since Exponential Histogram cannot have custom SLO's specified, an explicit bucket histogram is used whenever `serviceLevelObjectives` are configured. === Configuration with Spring Boot If you use Spring Boot, you can use the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#actuator.metrics.customizing.per-meter-properties[per-meter properties, window=_blank] to configure this behavior. If you want to generate a Histogram data point for a Timer with name `test.timer` and default buckets generated by Micrometer, use: [source,properties] ---- management.metrics.distribution.percentiles-histogram.test.timer=true ---- For buckets with customized SLO, use: [source,properties] ---- management.metrics.distribution.slo.test.timer=10.0,100.0,500.0,1000.0 ---- Alternatively, if you want to generate Summary data point for a timer with name `test.timer` and 90th and 99th percentiles, you can use: [source,properties] ---- management.metrics.distribution.percentiles.test.timer=0.9,0.99 ---- == Exemplars Exemplars are metadata that you can attach to the value of your metrics time series. They can reference data outside of your metrics. A common use case is cross-referencing tracing information (`traceId`, `spanId`) but any arbitrary key-value pairs can be attached. Exemplars are not tags/dimensions (attributes in OTLP terminology); they will not increase cardinality since they belong to the values of the time series. Using them allows you to correlate metrics with traces/spans (or other concepts). Micrometer supports exemplars for `Counter`, `Timer`, and `DistributionSummary`. To enable exemplars, you need to provide an `ExemplarContextProvider` to the `OtlpMeterRegistry` using its builder: [source,java,subs=+attributes] ----- include::{include-java}/metrics/OtlpMeterRegistryCustomizationTest.java[tags=customizeExemplarContextProvider, indent=0] ----- The implementation of `ExemplarContextProvider` is responsible for providing an `OtlpExemplarContext` which contains the necessary information (traceId, spanId, and arbitrary key-value pairs) to be attached to the exemplar. If the exemplar information is not available (there is no current Span or the Span is not sampled) or the implementor does not want this particular Span to be used as an exemplar, the `ExemplarContextProvider` should return `null`. NOTE: The `traceId` and `spanId` passed to `OtlpExemplarContext` must be case-insensitive hex-encoded strings. --- ### Modules/ROOT/Pages/Implementations/Prometheus = Micrometer Prometheus :sectnums: Prometheus is a dimensional time series database with a built-in UI, a custom query language, and math operations. Prometheus is designed to operate on a pull model, periodically scraping metrics from application instances, based on service discovery. Micrometer uses the Prometheus Java Client under the hood; there are two versions of it and Micrometer supports both. If you want to use the "new" client (`1.x`), use `micrometer-registry-prometheus` but if you want to use the "legacy" client (`0.x`), use `micrometer-registry-prometheus-simpleclient`. :system: prometheus include::_install.adoc[] :system: prometheus-simpleclient include::_install.adoc[] == Configuring Prometheus expects to scrape or poll individual application instances for metrics. In addition to creating a Prometheus registry, you also need to expose an HTTP endpoint to Prometheus's scraper. In a Spring Boot application, a https://docs.spring.io/spring-boot/docs/current/actuator-api/htmlsingle/#prometheus[Prometheus actuator endpoint] is auto-configured in the presence of Spring Boot Actuator. Otherwise, you can use any JVM-based HTTP server implementation to expose scrape data to Prometheus. The following example uses the JDK's `com.sun.net.httpserver.HttpServer` to expose a scrape endpoint: [source,java] ---- PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); try { HttpServer server = HttpServer.create(new InetSocketAddress(8080), 0); server.createContext("/prometheus", httpExchange -> { String response = prometheusRegistry.scrape(); <1> httpExchange.sendResponseHeaders(200, response.getBytes().length); try (OutputStream os = httpExchange.getResponseBody()) { os.write(response.getBytes()); } }); new Thread(server::start).start(); } catch (IOException e) { throw new RuntimeException(e); } ---- <1> The `PrometheusMeterRegistry` has a `scrape()` function that knows how to supply the String data necessary for the scrape. All you have to do is wire it to an endpoint. If you use the "new" client (`micrometer-registry-prometheus`), you can alternatively use `io.prometheus.metrics.exporter.httpserver.HTTPServer`, which you can find in `io.prometheus:prometheus-metrics-exporter-httpserver` (you need to add it as a dependency if you want to use it): [source,java] ---- PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); HTTPServer.builder() .port(8080) .registry(prometheusRegistry.getPrometheusRegistry()) .buildAndStart(); ---- If you use the "legacy" client (`micrometer-registry-prometheus-simpleclient`), you can alternatively use `io.prometheus.client.exporter.HTTPServer`, which you can find in `io.prometheus:simpleclient_httpserver`: [source,java] ---- PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); // you can set the daemon flag to false if you want the server to block new HTTPServer(new InetSocketAddress(8080), prometheusRegistry.getPrometheusRegistry(), true); ---- If you use the "new" client (`micrometer-registry-prometheus`), another alternative can be `io.prometheus.metrics.exporter.servlet.jakarta.PrometheusMetricsServlet`, which you can find in `io.prometheus:prometheus-metrics-exporter-servlet-jakarta` in case your app is running in a servlet container (such as Tomcat): [source,java] ---- PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); HttpServlet servlet = new PrometheusMetricsServlet(prometheusRegistry.getPrometheusRegistry()); ---- If you use the "legacy" client (`micrometer-registry-prometheus-simpleclient`), another alternative can be `io.prometheus.client.exporter.MetricsServlet`, which you can find in `io.prometheus:simpleclient_servlet` in case your app is running in a servlet container (such as Tomcat): [source,java] ---- PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT); HttpServlet servlet = new MetricsServlet(prometheusRegistry.getPrometheusRegistry()); ---- === Scrape Format By default, the `PrometheusMeterRegistry` `scrape()` method returns the https://prometheus.io/docs/instrumenting/exposition_formats/#text-based-format[Prometheus text format]. The https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md[OpenMetrics] format can also be produced. To specify the format to be returned, you can pass a content type to the `scrape` method. For example, to get the OpenMetrics 1.0.0 format scrape, you could use the Content-Type for it, as follows in case of the "new" client (`micrometer-registry-prometheus`): [source,java] ---- String openMetricsScrape = registry.scrape("application/openmetrics-text"); ---- If you use the "legacy" client (`micrometer-registry-prometheus-simpleclient`), you could use the Prometheus Java client constant for it: [source,java] ---- String openMetricsScrape = registry.scrape(TextFormat.CONTENT_TYPE_OPENMETRICS_100); ---- In Spring Boot applications, the https://docs.spring.io/spring-boot/docs/current/actuator-api/htmlsingle/#prometheus[Prometheus Actuator endpoint] supports scraping in either format, defaulting to the Prometheus text format in the absence of a specific `Accept` header. === The Prometheus Rename Filter In some cases, Micrometer provides instrumentation that overlaps with the commonly used Prometheus simple client modules but has chosen a different naming scheme for consistency and portability. If you wish to use the Prometheus "standard" names, add the following filter: [source,java] ---- prometheusRegistry.config().meterFilter(new PrometheusRenameFilter()); ---- === Prometheus Client Properties If you use the "new" client (`micrometer-registry-prometheus`), you can use some of the properties that the Prometheus Java Client supports, see the https://prometheus.github.io/client_java/config/config/[Prometheus Java Client Config docs]. These properties can be loaded from any source that is supported by the Prometheus Java Client (Properties file, System properties, etc.) or they can be obtained through Micrometer using `PrometheusConfig`: [source,java] ---- PrometheusConfig config = new PrometheusConfig() { @Override public String get(String key) { return null; } @Override public Properties prometheusProperties() { Properties properties = new Properties(); properties.putAll(PrometheusConfig.super.prometheusProperties()); <1> properties.setProperty("io.prometheus.exemplars.sampleIntervalMilliseconds", "1"); <2> return properties; } }; PrometheusMeterRegistry registry = new PrometheusMeterRegistry(config, new PrometheusRegistry(), Clock.SYSTEM); ---- <1> You can reuse the "default" properties defined in `PrometheusConfig`. <2> You can set any property from any property source. Micrometer passes these properties to the Exporters and the Exemplar Sampler of the Prometheus client, so you can use the https://prometheus.github.io/client_java/config/config/#exporter-properties[exporter], and the https://prometheus.github.io/client_java/config/config/#exemplar-properties[exemplar] properties of the Prometheus Client. == Graphing This section serves as a quick start to rendering useful representations in Prometheus for metrics originating in Micrometer. See the https://prometheus.io/docs/querying/basics[Prometheus docs] for a far more complete reference of what is possible in Prometheus. === Grafana Dashboard There are many third-party Grafana dashboards publicly available on https://grafana.com/grafana/dashboards/?search=micrometer[GrafanaHub]. See an example https://grafana.com/grafana/dashboards/4701-jvm-micrometer/[here]. NOTE: The dashboards are maintained by the community in their external GitHub repositories, so if you have an issue, it should be created in their respective GitHub repository. === Counters The query that generates a graph for the random-walk counter is `rate(counter[10s])`. .A Grafana rendered graph of the random walk counter. image::implementations/prometheus-counter.png[Grafana-rendered Prometheus counter] Representing a counter without rate normalization over some time window is rarely useful, as the representation is a function of both the rapidity with which the counter is incremented and the longevity of the service. It is generally most useful to rate-normalize these time series to reason about them. Since Prometheus keeps track of discrete events across all time, it has the advantage of allowing for the selection of an arbitrary time window across which to normalize at query time (for example, `rate(counter[10s])` provides a notion of requests per second over 10 second windows). The rate-normalized graph in the preceding image would return back to a value around 55 as soon as the new instance (say on a production deployment) was in service. .Counter over the same random walk, no rate normalization. image::implementations/prometheus-counter-norate.png[Grafana-rendered Prometheus counter (no rate)] In contrast, without rate normalization, the counter drops back to zero on service restart, and the count increases without bound for the duration of the service's uptime. === Timers The Prometheus `Timer` produces the following time series: * `$\{name}_seconds_count`: Total number of all calls. * `$\{name}_seconds_sum`: Total time of all calls. * `$\{name}_seconds_max`: Maximum recorded value within a configurable time window. Enabling xref:../concepts/histogram-quantiles.adoc[percentile histograms] adds a `$\{name}_seconds_bucket` time series; the count, sum, and max series are unchanged. Again, representing a counter without rate normalization over some time window is rarely useful, as the representation is a function of both the rapidity with which the counter is incremented and the longevity of the service. Using the following Prometheus queries, we can graph the most commonly used statistics about timers: * Average latency: `rate(timer_seconds_sum[10s])/rate(timer_seconds_count[10s])` * Throughput (requests per second): `rate(timer_seconds_count[10s])` .Timer over a simulated service. image::implementations/prometheus-timer.png[Grafana-rendered Prometheus timer] === Long task timers Long task timers measure tasks that are still running when Prometheus scrapes the registry. This differs from a `Timer`, which records completed events. The exposed Prometheus series depend on whether `publishPercentileHistogram()` is enabled: |=== | Default Prometheus output | With percentile histograms enabled | `$\{name}_seconds_count`, `$\{name}_seconds_sum`, `$\{name}_seconds_max` | `$\{name}_seconds_gcount`, `$\{name}_seconds_gsum`, `$\{name}_seconds_bucket`, `$\{name}_seconds_max` |=== The time series with xref:../concepts/histogram-quantiles.adoc[percentile histograms] enabled are defined by the https://github.com/prometheus/OpenMetrics/blob/main/specification/OpenMetrics.md#gaugehistogram[GaugeHistogram] in the OpenMetrics specification. With the `micrometer-registry-prometheus` module, `$\{name}_seconds_max` is exposed as a separate metric family. With the `micrometer-registry-prometheus-simpleclient` module, it is part of the same metric family as the other time series. A `LongTaskTimer` only samples tasks that are running at scrape time, so its values return to zero when no tasks are in progress. If you need bucket-based queries such as `histogram_quantile()` over in-progress task duration, or if alert rules need histogram buckets, enable xref:../concepts/histogram-quantiles.adoc[percentile histograms]. For a serial task, `long_task_timer_seconds_sum` plots the current cumulative duration of in-progress tasks. In Grafana, we can set an alert threshold at some fixed point. .Simulated back-to-back long tasks with a fixed alert threshold. image::implementations/prometheus-long-task-timer.png[Grafana-rendered Prometheus long task timer] == Warning on same name with different set of tag keys Prometheus https://prometheus.io/docs/instrumenting/writing_clientlibs/#labels:[strongly discourages] users from creating meters having the same name with a different set of tag keys. You should register meters with the same name with a consistent set of tag keys. This means that you should avoid the following: [source,java] ---- // Please don't do this registry.counter("test", "first", "1").increment(); registry.counter("test", "second", "2").increment(); ---- Instead, you should prefer doing the following: [source,java] ---- registry.counter("test", "first", "1", "second", "none").increment(); registry.counter("test", "first", "none", "second", "2").increment(); ---- == Exemplars Exemplars are metadata that you can attach to the value of your time series. They can reference data outside of your metrics. A common use case is storing tracing information (`traceId`, `spanId`). Exemplars are not tags/dimensions (labels in Prometheus terminology), they will not increase cardinality since they belong to the values of the time series. In order to setup Exemplars for `PrometheusMeterRegistry`, you will need a component that provides you the tracing information. If you use the "new" client (`micrometer-registry-prometheus`), this component is the `io.prometheus.metrics.tracer.common.SpanContext` while if you use the "legacy" client (`micrometer-registry-prometheus-simpleclient`), it is the `SpanContextSupplier`. Setting them up are somewhat similar, if you use the "new" client (`micrometer-registry-prometheus`): [source,java] ---- PrometheusMeterRegistry registry = new PrometheusMeterRegistry( PrometheusConfig.DEFAULT, new PrometheusRegistry(), Clock.SYSTEM, new MySpanContext() <1> ); registry.counter("test").increment(); System.out.println(registry.scrape("application/openmetrics-text")); ---- <1> You need to implement `SpanContext` (`class MySpanContext implements SpanContext { ... }`) or use an implementation that already exists. But if you use the "legacy" client (`micrometer-registry-prometheus-simpleclient`): [source,java] ---- PrometheusMeterRegistry registry = new PrometheusMeterRegistry( PrometheusConfig.DEFAULT, new CollectorRegistry(), Clock.SYSTEM, new DefaultExemplarSampler(new MySpanContextSupplier()) <1> ); registry.counter("test").increment(); System.out.println(registry.scrape(TextFormat.CONTENT_TYPE_OPENMETRICS_100)); ---- <1> You need to implement `SpanContextSupplier` (`class MySpanContextSupplier implements SpanContextSupplier { ... }`) or use an implementation that already exists. If your configuration is correct, you should get something like this, the `# {span_id="321",trace_id="123"} ...` section is the Exemplar right after the value: [source] ---- # TYPE test counter # HELP test test_total 1.0 # {span_id="321",trace_id="123"} 1.0 1713310767.908 # EOF ---- Exemplars are only supported in the OpenMetrics format (they will not show up in the Prometheus text format). You might need to explicitly ask for the OpenMetrics format, for example: [source,shell] ---- curl --silent -H 'Accept: application/openmetrics-text; version=1.0.0' localhost:8080/prometheus ---- --- ### Modules/ROOT/Pages/Implementations/SignalFx = Micrometer SignalFx :sectnums: :system: signalfx CAUTION: This module has been deprecated in favor of the xref:./otlp.adoc[OTLP Registry] because the https://github.com/signalfx/signalfx-java[SignalFX Java client library] that this module depends on has been deprecated. SignalFx is a dimensional monitoring system SaaS with a full UI that operates on a push model. It has a rich set of alert "`detectors`". include::_install.adoc[] == Configuring The following example configures SignalFx: [source,java] ---- SignalFxConfig signalFxConfig = new SignalFxConfig() { @Override public String accessToken() { return "MYTOKEN"; } @Override public String get(String k) { return null; // accept the rest of the defaults } }; MeterRegistry registry = new SignalFxMeterRegistry(signalFxConfig, Clock.SYSTEM); ---- There are two distinct sources of API keys in SignalFx. `SignalFxConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.signalfx` directly to the `SignalFxConfig`: [source,yml] ---- management.metrics.export.signalfx: access-token: MYTOKEN # The interval at which metrics are sent to Ganglia. See Duration.parse for the expected format. # The default is 1 minute. step: 1m ---- == Graphing This section serves as a quick start to rendering useful representations in SignalFx for metrics originating in Micrometer. See the https://docs.signalfx.com/en/latest/charts/index.html[SignalFx docs] for a far more complete reference of what is possible in SignalFx. === Timers At each publishing interval, the SignalFx `Timer` produces several time series in SignalFx: * `$\{name}.avg`: A mean latency for the publishing interval. * `$\{name}.count`: Throughput per second over the publishing interval. * `$\{name}.totalTime`: Total time per second over the publishing interval (used with `count`) to create aggregable means. * `$\{name}.percentiles`: Micrometer calculated percentiles for the publishing interval. One time series is produced for each percentile, with a tag of `phi` in the range of [0,1]. * `$\{name}.histogram`: One event is produced for each SLO boundary with a tag of 'le', indicating that it represents a cumulative count of events less than or equal to SLO boundaries over the publishing interval. To generate an aggregable view of latency in SignalFx, divide `totalTime` by `count`: image::implementations/signalfx-timer-latency-query.png[SignalFx-aggregable latency query] This is accomplished by adding signals for `$\{name}.totalTime` and `$\{name}.count`, adding a formula that divides them, and hiding the inputs to the formula. .Timer latency. image::implementations/signalfx-timer-latency.png[SignalFx-rendered timer] To generate a throughput chart, use the `$\{name}.count` signal: .Timer throughput. image::implementations/signalfx-timer-throughput.png[SignalFx-rendered timer throughput] To generate a plot of client-side percentiles, use the `$\{name}.percentiles` signal: .Timer Percentiles. image::implementations/signalfx-timer-percentiles.png[SignalFx-rendered percentiles] Note that these percentiles are _not aggregable_. The more dimensions you add to a timer, the less useful these values become. Finally, if you define SLO boundaries with the fluent builder for `Timer`, you can view throughput below certain SLO boundaries by using the `$\{name}.histogram` signal. In this example, we set SLO boundaries at 275 (green), 300 (blue), and 500 (purple) milliseconds for a simulated `Timer` that is recording samples normally distributed around 250 ms. These counts represent the rate/second of samples less than or equal to each SLO boundary. .Timer SLO boundaries. image::implementations/signalfx-timer-sla.png[SignalFx-rendered SLO boundaries] Where the lines converge at various points, it is evident that no sample exceeded the 275 ms SLO boundary. --- ### Modules/ROOT/Pages/Implementations/Stackdriver = Micrometer Stackdriver Monitoring Ray Tsang :sectnums: :system: stackdriver Stackdriver Monitoring is a dimensional time-series SaaS with built-in dashboarding. include::_install.adoc[] == Configuring The following example configures Stackdriver: [source,java] ---- StackdriverConfig stackdriverConfig = new StackdriverConfig() { @Override public String projectId() { return MY_PROJECT_ID; } @Override public String get(String key) { return null; } } MeterRegistry registry = StackdriverMeterRegistry.builder(stackdriverConfig).build(); ---- `StackdriverConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.stackdriver` directly to the `StackdriverConfig`: [source,yml] ---- management.metrics.export.stackdriver: project-id: MY_PROJECT_ID resource-type: global # You will probably want to disable Stackdriver Monitoring publishing in a local development profile. enabled: true # The interval at which metrics are sent to Stackdriver Monitoring. The default is 1 minute. step: 1m ---- For most environments, you need to create and configure credentials to push metrics to Stackdriver Monitoring. In most cases, you should create a service account with Stackdriver Monitoring permissions and configure a `GOOGLE_APPLICATION_CREDENTIALS` environmental variable to the path of the service account key file. The following example shows how to do so: [source] ---- export PROJECT_ID=MY_PROJECT_ID export APP_NAME=MY_APP # Create a service account gcloud iam service-accounts create $APP_NAME # Grant the service account Stackdriver Monitoring writer permission gcloud projects add-iam-policy-binding $PROJECT_ID \ --member serviceAccount:$APP_NAME@$PROJECT_ID.iam.gserviceaccount.com \ --role roles/monitoring.metricWriter # Create a key JSON file gcloud iam service-accounts keys create $HOME/$APP_NAME-key.json \ --iam-account $APP_NAME@$PROJECT_ID.iam.gserviceaccount.com # Configure GOOGLE_APPLICATION_CREDENTIALS env var export GOOGLE_APPLICATION_CREDENTIALS=$HOME/$APP_NAME-key.json ---- When running in managed environments (such as Google App Engine, Google Cloud Run, and Google Cloud Function), you need not configure this environmental variable. In those environments, a service account is automatically associated with the application instance. The underlying Stackdriver Monitoring client library can automatically detect and use those credentials. == Stackdriver Labels Micrometer metrics tags are mapped to https://cloud.google.com/monitoring/api/v3/metrics-details#intro-time-series[Stackdriver metrics labels]. With tags and labels, you can further filter or group by the tag or label. See xref:../concepts/naming.adoc#_tag_naming[Micrometer Concepts] for more information on tags. The following example filters by tags: [source,java] ---- MeterRegistry registry = StackdriverMeterRegistry.builder(stackdriverConfig).build(); registry.config().commonTags("application", "my-application"); ---- You can also configure resource labels with the `StackdriverConfig` method, `resourceLabels`. Depending on the configured `resourceType`, there will be required resource labels. See the documentation on https://cloud.google.com/monitoring/custom-metrics/creating-metrics#which-resource[choosing a monitored resource type]. IMPORTANT: When using Micrometer across multiple applications or instances, it is necessary that Stackdriver labels are unique per application or instance. Otherwise, you will see errors such as `One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured`. If using a resource type other than `global`, the resource labels may already make metrics unique per application instance. If not, a common tag with the hostname or platform-provided instance ID may be a good candidate for achieving this. == Spring Boot Spring Boot provides auto-configuration for Micrometer's `StackdriverMeterRegistry`. For more information, see the https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready-metrics-export-stackdriver[Spring Boot documentation]. You can manually configure or register the `StackdriverMeterRegistry`. In addition to using Spring Boot Actuator, make sure you create the `StackdriverMeterRegistry` bean: [source,java] ---- @Bean StackdriverConfig stackdriverConfig() { return new StackdriverConfig() { @Override public String projectId() { return MY_PROJECT_ID; } @Override public String get(String key) { return null; } } } @Bean StackdriverMeterRegistry meterRegistry(StackdriverConfig stackdriverConfig) { return StackdriverMeterRegistry.builder(stackdriverConfig).build(); } ---- You can also use https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-features.html#production-ready-metrics-common-tags[Spring Boot Actuator Common Tags configuration] to configure common tags: [source] ---- spring.application.name=my-application management.metrics.tags.application=${spring.application.name} ---- --- ### Modules/ROOT/Pages/Implementations/StatsD = Micrometer StatsD :sectnums: :system: statsd StatsD is a UDP-based sidecar-driven metrics collection system. The maintainer of the original StatsD line protocol specification is Etsy. Datadog's DogStatsD and Influx's Telegraf each accept a modified version of the line protocol, having each enriched the original specification with dimensionality in different ways. If you intend to use the Datadog or Telegraf flavors, see the documentation for Micrometer's xref:./datadog.adoc[Datadog] or xref:./influx.adoc[Influx] support. NOTE: Starting a `StatsdMeterRegistry` doesn't block waiting for a connection to the backend to be established, so recorded measurements could be lost until the connection is established. include::_install.adoc[] == Configuring This configuration is used to ship metrics to a StatsD agent that is compatible with the original Etsy protocol. Metrics are shipped immediately over UDP to the agent. [source,java] ---- StatsdConfig config = new StatsdConfig() { @Override public String get(String k) { return null; } @Override public StatsdFlavor flavor() { return StatsdFlavor.Etsy; } }; MeterRegistry registry = new StatsdMeterRegistry(config, Clock.SYSTEM); ---- NOTE: You can also configure Telegraf to accept the dogstatsd format. If you use Telegraf, configuring Micrometer to ship Telegraf-formatted StatsD lines eases the requirements of your Telegraf configuration. `StatsdConfig` is an interface with a set of default methods. If, in the implementation of `get(String k)`, rather than returning `null`, you instead bind it to a property source, you can override the default configuration. For example, Micrometer's Spring Boot support binds properties that are prefixed with `management.metrics.export.statsd` directly to the `StatsdConfig`: [source,yml] ---- management.metrics.export.statsd: flavor: etsy # You will probably want to conditionally disable StatsD publishing in local development. enabled: true # The interval at which metrics are sent to StatsD. The default is 1 minute. step: 1m ---- == Customizing the Metrics Sink By default, Micrometer publishes the StatsD line protocol over UDP, as the vast majority of existing StatsD agents are UDP servers. You can fully customize how the line protocol is shipped by modifying the builder for `StatsdMeterRegistry`: [source,java] ---- Consumer lineLogger = line -> logger.info(line); <1> MeterRegistry registry = StatsdMeterRegistry.builder(StatsdConfig.DEFAULT) <2> .clock(clock) .lineSink(lineLogger) .build(); ---- <1> Define what to do with lines. <2> The flavor configuration option determines the structure of the line for the default line builder. It has no effect if you override the line builder with a customization. === Unix Domain Socket (UDS) Support The StatsD registry supports sending metrics over Unix domain socket datagrams. To enable this transport, configure the `protocol` as `uds_datagram` and set the `host` to the absolute path of the socket file. [NOTE] ==== The `host` value must be the plain absolute path (for example, `/var/run/datadog/dsd.socket`). Do not prefix it with `unix://`. ==== *Java configuration example:* [source,java] ---- StatsdConfig config = new StatsdConfig() { @Override public String get(String k) { return null; // accept defaults } @Override public StatsdProtocol protocol() { return StatsdProtocol.UDS_DATAGRAM; } @Override public String host() { return "/var/run/datadog/dsd.socket"; } }; MeterRegistry registry = new StatsdMeterRegistry(config, Clock.SYSTEM); ---- === Using Apache Kafka for Line Sink You can also use Apache Kafka for line sink, as follows: [source,java] ---- Properties properties = new Properties(); properties.setProperty(BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); properties.setProperty(KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); properties.setProperty(VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); Producer producer = new KafkaProducer<>(properties); StatsdMeterRegistry.builder(statsdConfig) .lineSink((line) -> producer.send(new ProducerRecord<>("my-metrics", line))) .build(); ---- Now Micrometer produces lines for metrics to the `my-metrics` topic and you can consume the lines on the topic. == Customizing the Line Format The built-in Etsy, dogstatsd, and Telegraf flavors cover most known public StatsD agents, but you can completely customize the line format to satisfy closed, proprietary agents. Again, we use the `StatsdMeterRegistry` builder to establish a line builder for each ID. Providing an instance of the builder _per ID_ offers you the opportunity to eagerly cache the serialization of the ID's name and tags to optimize the serialization of a StatsD line based on that ID as samples are recorded. The following listing defines a fictional format: [source,java] ---- Function nameAndUnits = id -> new StatsdLineBuilder() { String name = id.getName() + "/" + (id.getBaseUnit() == null ? "unknown" : id.getBaseUnit()); @Override public String count(long amount, Statistic stat) { return name + ":" + amount + "|c"; } ... // implement gauge, histogram, and timing similarly } MeterRegistry registry = StatsdMeterRegistry.builder(StatsdConfig.DEFAULT) <1> .clock(clock) .lineBuilder(nameAndUnits) .build(); ---- <1> Because you have taken control of line building, the flavor is ignored. --- ### Modules/ROOT/Pages/Implementations/Wavefront = Micrometer Wavefront :sectnums: :system: wavefront CAUTION: This module has been deprecated due to https://support.broadcom.com/web/ecx/support-content-notification/-/external/content/release-announcements/0/25153[Wavefront's End of Life Announcement]. Wavefront is a dimensional monitoring system offered as a SaaS with a full UI, custom query language, and advanced math operations. Wavefront operates on a push model. Metrics can be pushed either through a sidecar process running on the same host (called the Wavefront proxy) or directly to the Wavefront API. include::_install.adoc[] == Configuring This section describes how to configure Wavefront when you send data: * <> * <> [[configuring-directly-to-wavefront]] === Directly to Wavefront The following example configures sending directly to Wavefront: [source,java] ---- WavefrontConfig config = new WavefrontConfig() { @Override public String uri() { return "https://longboard.wavefront.com"; <1> } @Override public String apiToken() { return "MYAPIKEY"; <2> } @Override public String get(String key) { return null; <3> } }; MeterRegistry registry = new WavefrontMeterRegistry(config, Clock.SYSTEM); ---- <1> `longboard` is the name of the co-tenant instance on which most organizations start. Once you reach a sufficient scale, Wavefront may move you to a dedicated host. <2> This is required when pushing directly to Wavefront's API. <3> Accept the rest of the defaults. [[configuring-through-a-wavefront-proxy-sidecar]] === Through a Wavefront Proxy Sidecar The following example configures sending through a Wavefront proxy sidecar: [source,java] ---- MeterRegistry registry = new WavefrontMeterRegistry(WavefrontConfig.DEFAULT_PROXY, Clock.SYSTEM); ---- The default proxy configuration pushes metrics and histogram distributions to a Wavefront proxy sitting on `localhost:2878`. NOTE: If publishing metrics to a Wavefront proxy, the URI must be expressed in the form of `proxy://HOST:PORT`. == Graphing This section serves as a quick start to rendering useful representations in Wavefront for metrics originating in Micrometer. See the https://docs.wavefront.com/query_language_getting_started.html[Wavefront docs] for a far more complete reference of what is possible in Wavefront. === Counters The query that generates a graph for a random-walk counter is `rate(ts(counter))`. .A Wavefront rendered graph of the random walk counter. image::implementations/wavefront-counter-rate.png[Wavefront-rendered counter] Representing a counter without rate normalization over some time window is rarely useful, as the representation is a function of both the rapidity with which the counter is incremented and the longevity of the service. It is generally most useful to rate-normalize these time series to reason about them. Because Wavefront keeps track of cumulative counts across all time, it has the advantage of allowing for the selection of a particular time function at query time (for example, `rate(ts(counter))` to compute the per-second rate of change). === Timers The Wavefront `Timer` produces different time series depending on whether or not `publishPercentileHistogram` is enabled. If `publishPercentileHistogram` is enabled, the Wavefront `Timer` produces histogram distributions that let you query for the latency at any percentile using `hs()` queries. For example, you can visualize latency at the 95th percentile (`percentile(95, hs(timer.m))`) or the 99.9th percentile (`percentile(99.9, hs(timer.m))`). For more information on histogram distributions, see <>, later in this section. If `publishPercentileHistogram` is disabled, the Wavefront `Timer` produces several time series: * `$\{name}.avg`: Mean latency across all calls. * `$\{name}.count`: Total number of all calls. * `$\{name}.sum`: Total time of all calls. * `$\{name}.max`: Max latency over the publishing interval. * `$\{name}.percentiles`: Micrometer-calculated percentiles for the publishing interval. These cannot be aggregated across dimensions. You can use these time series to generate a quick view of latency in Wavefront: .Timer latency. image::implementations/wavefront-timer-latency.png[Wavefront-rendered timer] The preceding chart shows the average latency (`rate(ts(timer.sum))/rate(ts(timer.count))` in green), 95th percentile (`ts(timer.percentile, phi="0.95")` in orange), and max (`ts(timer.max)` in blue). Additionally, `rate(ts(timer.count))` represents a rate/second throughput of events being timed: .Timer throughput. image::implementations/wavefront-timer-throughput.png[Wavefront-rendered timer throughput] [[wavefront-histograms]] === Wavefront Histograms Wavefront's histogram implementation stores an actual distribution of metrics, as opposed to single metrics. This lets you apply any percentile and aggregation function on the distribution at query time without having to specify specific percentiles and metrics to keep during metric collection. Wavefront histogram distributions are collected and reported for any `Timer` or `DistributionSummary` that has `publishPercentileHistogram` enabled. By default, distributions that are reported to Wavefront get aggregated by the minute, providing you with a histogram distribution for each minute. You also have the option of aggregating by hour or day. You can customize this with the following configuration options: * `reportMinuteDistribution`: Boolean specifying whether to aggregate by minute. Enabled by default. Metric name in Wavefront has `.m` suffix. * `reportHourDistribution`: Boolean specifying whether to aggregate by hour. Disabled by default. Metric name in Wavefront has `.h` suffix. * `reportDayDistribution`: Boolean specifying whether to aggregate by day. Disabled by default. Metric name in Wavefront has `.d` suffix. If you are sending to a Wavefront proxy, by default, both metrics and histogram distributions are published to the same port: 2878 in the default proxy configuration. If your proxy is configured to listen for histogram distributions on a different port, you can specify the port to which to publish by using the `distributionPort` configuration option. You can query histogram distributions in Wavefront by using `hs()` queries. For example, `percentile(98, hs($\{name}.m))` returns the 98th percentile for a particular histogram aggregated over each minute. Each histogram metric name has a suffix (`.m`, `.h`, or `.d`), depending on the histogram's aggregation interval. See the https://docs.wavefront.com/proxies_histograms.html[Wavefront Histograms documentation] for more information. --- ### Modules/ROOT/Pages/Observation/Components [[micrometer-observation-components]] = Observation Components In this section we will describe main components related to Micrometer Observation. * <> * <> * <> * <> * <> * <> * <> *Micrometer Observation basic flow* `Observation` through `ObservationRegistry` gets created with a mutable `Observation.Context`. On each Micrometer Observation lifecycle action (e.g. `start()`) a corresponding `ObservationHandler` method is called (e.g. `onStart`) with the mutable `Observation.Context` as argument. // https://arthursonzogni.com/Diagon/#GraphDAG // ObservationRegistry->Observation // Context->Observation // Observation->Handler [source,subs=+attributes] ----- ┌───────────────────┐┌───────┐ │ObservationRegistry││Context│ └┬──────────────────┘└┬──────┘ ┌▽────────────────────▽┐ │Observation │ └┬─────────────────────┘ ┌▽──────┐ │Handler│ └───────┘ ----- *Micrometer Observation detailed flow* // https://arthursonzogni.com/Diagon/#GraphDAG // ObservationRegistry->Observation // Context->Observation // ObservationConvention->Observation // ObservationPredicate->Observation // Observation->Handler // Handler->ObservationFilter [source,subs=+attributes] ----- ┌───────────────────┐┌───────┐┌─────────────────────┐┌────────────────────┐ │ObservationRegistry││Context││ObservationConvention││ObservationPredicate│ └┬──────────────────┘└┬──────┘└┬────────────────────┘└┬───────────────────┘ ┌▽────────────────────▽────────▽──────────────────────▽┐ │Observation │ └┬─────────────────────────────────────────────────────┘ ┌▽──────┐ │Handler│ └┬──────┘ ┌▽────────────────┐ │ObservationFilter│ └─────────────────┘ ----- `Observation` through `ObservationRegistry` gets created with a mutable `Observation.Context`. To allow name and key-value customization, an `ObservationConvention` can be used instead of direct name setting. List of `ObservationPredicate` is run to verify if an `Observation` should be created instead of a no-op version. On each xref:./introduction.adoc[Micrometer Observation lifecycle] action (e.g. `start()`) a corresponding `ObservationHandler` method is called (e.g. `onStart`) with the mutable `Observation.Context` as argument. On `Observation` stop, before calling the `ObservationHandler` `onStop` methods, list of `ObservationFilter` is called to optionally further modify the `Observation.Context`. [[micrometer-observation-context]] == Observation.Context To pass information between the instrumented code and the handler (or between handler methods, such as `onStart` and `onStop`), you can use an `Observation.Context`. An `Observation.Context` is a `Map`-like container that can store values for you while your handler can access the data inside the context. [[micrometer-observation-handler]] == Observation Handler Observation Handler allows adding capabilities to existing instrumentations (i.e. you instrument code once and depending on the Observation Handler setup, different actions, such as create spans, metrics, logs will happen). In other words, if you have instrumented code and want to add metrics around it, it's enough for you to register an Observation Handler in the Observation Registry to add that behaviour. Let's look at the following example of adding a timer behaviour to an existing instrumentation. A popular way to record Observations is storing the start state in a `Timer.Sample` instance and stopping it when the event has ended. Recording such measurements could look like this: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=timer,indent=0] ----- If you want to have more observation options (such as metrics and tracing -- already included in Micrometer -- plus anything else you will plug in), you need to rewrite that code to use the `Observation` API. [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=observation,indent=0] ----- Starting with Micrometer 1.10, you can register "handlers" (`ObservationHandler` instances) that are notified about the xref:./introduction.adoc[lifecycle event] of an observation (for example, you can run custom code when an observation is started or stopped). Using this feature lets you add tracing capabilities to your existing metrics instrumentation (see: `DefaultTracingObservationHandler`). The implementation of these handlers does not need to be tracing related. It is completely up to you how you are going to implement them (for example, you can add logging capabilities). [[micrometer-observation-handler-example]] === ObservationHandler Example Based on this, we can implement a simple handler that lets the users know about its invocations by printing them out to `stdout`: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=simple_handler,indent=0] ----- You need to register the handler to the `ObservationRegistry`: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=register_handler,indent=0] ----- You can use the `observe` method to instrument your codebase: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=instrumenting_code,indent=0] ----- You can also take full control of the scoping mechanism: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=manual_scoping,indent=0] ----- [[micrometer-observation-default-meter-handler]] === DefaultMeterObservationHandler Micrometer provides `DefaultMeterObservationHandler`, an `ObservationHandler` implementation that automatically creates metrics from Observations. When registered on an `ObservationRegistry`, it translates Observation lifecycle events into the following meter types: * **Timer** (``) -- Created on `stop`. Records the duration between `start` and `stop`. Tagged with all low cardinality key values and an `error` tag (set to the exception class name if an error was recorded, or `none` otherwise). * **LongTaskTimer** (`.active`) -- Created on `start`. Tracks the duration of in-progress observations. Stopped when the observation is stopped. Only tagged with low cardinality key values that are available at the time of `start`. * **Counter** (`.`) -- Incremented when an event is signaled via `Observation.event(Event)`. Tagged with all low cardinality key values available at the time of the event. NOTE: Only low cardinality key values are used as meter tags. High cardinality key values are not included in meters because they can cause a cardinality explosion in the metrics backend. WARNING: Since the `LongTaskTimer` is created in the `onStart` method, it can only contain tags that are available by that time. Any low cardinality key values added after calling `start` on the `Observation` will not be included as tags on the `LongTaskTimer`. The following example shows how to register `DefaultMeterObservationHandler` and the metrics it creates: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=default_meter_handler,indent=0] ----- If you do not need the `LongTaskTimer` (for example, to reduce the number of created meters), you can disable it using `IgnoredMeters`: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=default_meter_handler_ignore_ltt,indent=0] ----- [[micrometer-observation-events]] == Signaling Errors and Arbitrary Events When **instrumenting** code, we might want to signal that an error happened or signal that an arbitrary event happened. The observation API lets us do so through its `error` and `event` methods. One use-case for signaling an arbitrary event can be attaching annotations to `Span` for Distributed Tracing, but you can also process them however you want in your own handler, such as emitting log events based on them: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=error_and_event,indent=0] ----- [[micrometer-observation-convention-example]] == Observation.ObservationConvention Example When **instrumenting** code, we want to provide sensible defaults for tags, but also we want to let users easily change those defaults. An `ObservationConvention` interface is a description of what tags and name we should create for an `Observation.Context`: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=tax_example,indent=0] ----- For a more detailed example, see the full usage example of an instrumentation together with overriding the default tags. The following example puts the whole code together: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=observation_convention_example,indent=0] ----- [[micrometer-observation-predicates-filters]] == Observation Predicates and Filters To globally disable observations under given conditions, you can use an `ObservationPredicate`. To mutate the `Observation.Context`, you can use an `ObservationFilter`. To set these, call the `ObservationRegistry#observationConfig()#observationPredicate()` and `ObservationRegistry#observationConfig()#observationFilter()` methods, respectively. The following example uses predicates and filters: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationConfiguringTests.java[tags=predicate_and_filter,indent=0] ----- [[micrometer-observation-annotations]] == Using Annotations With @Observed and @ObservationKeyValue If you have turned on Aspect Oriented Programming (for example, by using `org.aspectj:aspectjweaver`), you can use the `@Observed` annotation to create observations. You can put that annotation either on a method to observe it or on a class to observe all the methods in it. The following example shows an `ObservedService` that has an annotation on a method: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=observed_service,indent=0] ----- The following test asserts whether the proper observation gets created when a proxied `ObservedService` instance gets called: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=observed_aop,indent=0] ----- Also, you can use `@ObservationKeyValue` annotation to add key-values via method parameters. The following example shows an `ObservedServiceWithParameter` that has an annotation on a method: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=observed_service_with_parameter,indent=0] ----- The following test asserts whether the proper observation gets created when a proxied `ObservedServiceWithParameter` instance gets called: [source,java,subs=+attributes] ----- include::{include-java}/observation/ObservationHandlerTests.java[tags=observed_aop_with_parameter,indent=0] ----- ---