kube-state-metrics Documentation

README Copy Section

Documentation

This documentation is intended to be a complete reflection of the current state of the exposed metrics of kube-state-metrics.

Any contribution to improving this documentation or adding sample usages will be appreciated.

Table of Contents Copy Chapter

• Metrics Stages
• Exposed Metrics
• PromQL Queries
• Join Metrics
• CLI arguments

Metrics Stages Copy Chapter

Stages about metrics are grouped into three categories：

Opt-in Metrics Copy Chapter

As of v2.3.0, kube-state-metrics supports additional opt-in metrics via the CLI flag --metric-opt-in-list. See the metric documentation to identify which metrics need to be specified.

Exposed Metrics Copy Chapter

Per group of metrics there is one file for each metrics.
See each file for specific documentation about the exposed metrics:

Default Resources Copy Section

• CertificateSigningRequest Metrics
• ConfigMap Metrics
• CronJob Metrics
• DaemonSet Metrics
• Deployment Metrics
• Endpoint Metrics
• Horizontal Pod Autoscaler Metrics
• Ingress Metrics
• Job Metrics
• Lease Metrics
• LimitRange Metrics
• MutatingAdmissionPolicy Metrics
• MutatingAdmissionPolicyBinding Metrics
• MutatingWebhookConfiguration Metrics
• Namespace Metrics
• NetworkPolicy Metrics
• Node Metrics
• PersistentVolume Metrics
• PersistentVolumeClaim Metrics
• Pod Disruption Budget Metrics
• Pod Metrics
• ReplicaSet Metrics
• ReplicationController Metrics
• ResourceQuota Metrics
• Secret Metrics
• Service Metrics
• StatefulSet Metrics
• StorageClass Metrics
• ValidatingAdmissionPolicy Metrics
• ValidatingAdmissionPolicyBinding Metrics
• ValidatingWebhookConfiguration Metrics
• VolumeAttachment Metrics

Optional Resources Copy Section

• ClusterRole Metrics
• ClusterRoleBinding Metrics
• EndpointSlice Metrics
• IngressClass Metrics
• Role Metrics
• RoleBinding Metrics
• ServiceAccount Metrics

PromQL Queries Copy Chapter

For practical PromQL query examples and alerting rules:

• StatefulSet Queries & Alerts

Join Metrics Copy Chapter

When an additional, not provided by default label is needed, a Prometheus matching operator
can be used to extend single metrics output.

This example adds label_release to the set of default labels of the kube_pod_status_ready metric
and allows you select or group the metrics by Helm release label:

kube_pod_status_ready * on (namespace, pod) group_left(label_release) kube_pod_labels

Another useful example would be to query the memory usage of pods by its phase, such as Running:

sum(kube_pod_container_resource_requests{resource="memory"}) by (namespace, pod, node)
  * on (namespace, pod) group_left() (sum(kube_pod_status_phase{phase="Running"}) by (pod, namespace) == 1)

Metrics from Custom Resources Copy Chapter

[!NOTE]
custom-resource-state is feature-frozen in favor of resource-state-metrics. Once resource-state-metrics is stable, custom-resource-state will be deprecated.

See Custom Resource State Metrics for experimental support for custom resources.

CLI Arguments Copy Chapter

Additionally, options for kube-state-metrics can be passed when executing as a CLI, or in a kubernetes / openshift environment. More information can be found here: CLI Arguments

Protecting /metrics endpoints Copy Chapter

Kube-State-Metrics' metrics can contain sensitive information about the state of the cluster, which you as an operator might want to additionally protect from unauthorized access.
In order to achieve this, you need to enable the --auth-filter flag on kube-state-metrics.
With this, kube-state-metrics will only accept authenticated and authorized requests to the /metrics endpoints.
Kube-state-metrics uses Kubernetes' RBAC mechanisms for this, so this means that every scrape will trigger a request against the API Server for TokenReview and SubjectAccessReview.
The clients scraping the endpoint, need to use a token which can be provided by a ServiceAccount that can be set up the following way:

A ClusterRole providing access like this:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metrics-reader
rules:
- nonResourceURLs:
  - "/metrics"
  verbs:
  - get

and a matching ClusterRoleBinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-reader-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metrics-reader
subjects:
- kind: ServiceAccount
  name: YOUR_SERVICE_ACCOUNT
  namespace: NAMESPACE_OF_THE_SERVICE_ACCOUNT

Your client can then use either this ServiceAccount to gather metrics or you can create a token, that can be used to fetch data like this:

TOKEN=$(kubectl create token YOUR_SERVICE_ACCOUNT -n NAMESPACE_OF_THE_SERVICE_ACCOUNT)
curl -H "Authorization: Bearer $TOKEN" KUBE_STATE_METRICS_URL:8080/metrics

Design/Metrics Best Practices Copy Section

Kube-State-Metrics - Timeseries best practices

Author: Manuel Rüger ([email protected])

Date: October 17th 2024

Introduction Copy Chapter

Kube-State-Metrics' goal is to provide insights into the state of Kubernetes objects by exposing them as metrics.
This document provides guidelines with the goal to create a good user experience when using these metrics.

Please be aware that this document is introduced in a later stage of the project and there might be metrics that do not follow these best practices.
Feel encouraged to report these metrics and provide a pull request to improve them.

General best practices Copy Chapter

We follow Prometheus best practices in terms of naming and labeling.

Best practices for kube-state-metrics Copy Chapter

Avoid pre-computation Copy Section

kube-state-metrics should expose metrics on an individual object level and avoid any sort of pre-computation unless it is required due to for example high cardinality on objects.
We prefer not to add metrics that can be derived from existing raw metrics. For example, we would not want to expose a metric called kube_pod_total as it can be computed with count(kube_pod_info).
This way kube-state-metrics allows the user to have full control on how they want to use the metrics and gives them flexibility to do specific computation.

Static object properties Copy Section

An object usually has a stable set of properties that do not change during its lifecycle in Kubernetes.
This includes properties like name, namespace, uid etc. that have a 1:1 relationship with the object.
It is a good practice to group those together into an _info metric.
If there is a 1:n relationship (e.g. a list of ports), it should be in a separate metric to avoid generating too many metrics.

Dynamic object properties Copy Section

An object can also have a dynamic set of properties, which are usually part of the status field.
These change during the lifecycle of the object.
For example a pod can be in different states like "Pending", "Running" etc.
These should be part of a "State Set" that includes labels that identify the object as well as the dynamic property.

Linked properties Copy Section

If an object contains a substructure that links multiple properties together (e.g. endpoint address and port), those should be reported in the same metric.

Optional properties Copy Section

Some Kubernetes objects have optional fields. In case there is an optional value, the label should still be exposed, ideally as an empty string.

Timestamps Copy Section

Timestamps like creation time or modification time should be exposed as a value. The metric should end with _timestamp_seconds. The date value is represented in UNIX epoch seconds.

Cardinality Copy Section

Some object properties can cause cardinality issues if they can contain a lot of different values or are linked together with multiple properties that also can change a lot.
In this case it is better to limit the number of values that can be exposed within kube-state-metrics by allowing only a few of them and have a default for others.
If for example the Kubernetes object contains a status field that contains an error message that can change a lot, it would be better to have a boolean error="true" label in case there is an error.
If there are some error messages that are worth exposing, those could be exposed and for any other message, a default value could be provided.

Stability Copy Chapter

We follow the stability framework derived from Kubernetes, in which we expose experimental and stable metrics.
Experimental metrics are recently introduced or expose alpha/beta resources in the Kubernetes API.
They can change anytime and should be used with caution.
They can be promoted to a stable metric once the object stabilized in the Kubernetes API or they were part of two consecutive releases and haven't observed any changes in them.

Stable metrics are considered frozen with the exception of new labels being added.
A stable metric or a label on a stable metric can be deprecated in release Major.Minor and the earliest point it will be removed is the release Major.Minor+2.

Design/Metrics Store Performance Optimization Copy Section

Kube-State-Metrics - Performance Optimization Proposal

Author: Max Inden ()

Date: 23. July 2018

Target release: v1.5.0

Glossary Copy Chapter

• kube-state-metrics: “Simple service that listens to the Kubernetes API server
  and generates metrics about the state of the objects”

• Time series: A single line in a /metrics response e.g.
  “metric_name{label="value"} 1”

Problem Statement Copy Chapter

There has been repeated reports of two issues running kube-state-metrics on
production Kubernetes clusters. First kube-state-metrics takes a long time
(“10s - 20s”) to respond on its /metrics endpoint, leading to Prometheus
instances dropping the scrape interval request and marking the given time series
as stale. Second kube-state-metrics uses a lot of memory and thereby being
out-of-memory killed due to low set Kubernetes resource limits.

Goal Copy Chapter

The goal of this proposal can be split into the following sub-goals ordered by
their priority:

1. Decrease response time on /metrics endpoint

2. Decrease overall runtime memory usage

Status Quo Copy Chapter

Instead of requesting the needed information from the Kubernetes API-Server on
demand (on scrape), kube-state-metrics uses the Kubernetes client-go cache tool
to keep a full in memory representation of all Kubernetes objects of a given
cluster. Using the cache speeds up the performance critical path of replying to
a scrape request, and reduces the load on the Kubernetes API-Server by only
sending deltas whenever they occur. Kube-state-metrics does not make use of all
properties and sub-objects of these Kubernetes objects that it stores in its
cache.

On a scrape request by e.g. Prometheus on the /metrics endpoint
kube-state-metrics calculates the configured time series on demand based on the
objects in its cache and converts them to the Prometheus string representation.

Proposal Copy Chapter

Instead of a full representation of all Kubernetes objects with all its
properties in memory via the Kubernetes client-go cache, use a map, addressable
by the Kubernetes object uuid, containing all time series of that object as a
single multi-line string.

var cache = map[uuid][]byte{}

Kube-state-metrics listens on add, update and delete events via Kubernetes
client-go reflectors. On add and update events kube-state-metrics generates all
time series related to the Kubernetes object based on the event’s payload,
concatenates the time series to a single byte slice and sets / replaces the byte
slice in the store at the uuid of the Kubernetes object. One can precompute the
length of a time series byte slice before allocation as the sum of the length of
the metric name, label keys and values as well as the metric value in string
representation. On delete events kube-state-metrics deletes the uuid entry of
the given Kubernetes object in the cache map.

On a scrape request on the /metrics endpoint, kube-state-metrics iterates over
the cache map and concatenates all time series string blobs into a single
string, which is finally passed on as a response.

/* Detailed source-code truncated for AI context efficiency. */

Code to reproduce diagram

Build via text-diagram

object pod_reflector pod_store pod_collector metrics_endpoint

note left of pod_reflector: new pod p1
pod_reflector -> pod_store: Add(p1)
note right of pod_store: generateMetrics(p1)
pod_store -> pod_reflector: nil

note right of metrics_endpoint: GET /metrics
metrics_endpoint -> pod_collector: Collect()
pod_collector -> pod_store: GetAll()
pod_store -> pod_collector: []string{metrics}
pod_collector -> metrics_endpoint: concat(metrics)

FAQ / Follow up improvements Copy Chapter

• If kube-state-metrics only listens on add, update and delete events, how is it
  aware of already existing Kubernetes objects created before kube-state-metrics
  was started? Leveraging Kubernetes client-go, reflectors can initialize all
  existing objects before any add, update or delete events. To ensure no events
  are missed in the long run, periodic resyncs via Kubernetes client-go can be
  triggered. This extra confidence is not a must and should be compared to its
  costs, as Kubernetes client-go already gives decent guarantees on event
  delivery.

• What about metadata (HELP and description) in the /metrics output? As a first
  iteration they would be skipped until we have a better idea on the design.

• How can the cache map be concurrently accessed? The core golang map
  implementation is not thread-safe. As a first iteration a simple mutex should
  be sufficient. Golang's sync.Map might be considered.

• To solve the problem of out of order events send by the Kubernetes API-Server
  to kube-state-metrics, to each blob of time series inside the cache map it can
  keep the Kubernetes resource version. On add and update events, first compare
  the resource version of the event with than the resource version in the cache.
  Only move forward if the former is higher than the latter.

• In case the memory consumption of the time series string blobs is a problem
  the following optimization can be considered: Among the time series strings,
  multiple sub-strings will be heavily duplicated like the metric name. Instead
  of saving unstructured strings inside the cache map, one can structure them,
  using pointers to deduplicate e.g. metric names.

• Kube-state-metrics does not make use of all properties of all Kubernetes
  objects. Instead of unmarshalling unused properties, their json struct tags or
  their Protobuf representation could be removed.

Developer/Cli Arguments Copy Section

Command line arguments

kube-state-metrics can be configured through command line arguments.

Those arguments can be passed during startup when running locally:

kube-state-metrics --telemetry-port=8081 --kubeconfig=<KUBE-CONFIG> --apiserver=<APISERVER> ...

Or configured in the args section of your deployment configuration in a Kubernetes / Openshift context:

spec:
  template:
    spec:
      containers:
        - args:
          - '--telemetry-port=8081'
          - '--kubeconfig=<KUBE-CONFIG>'
          - '--apiserver=<APISERVER>'

Available options Copy Chapter

/* Detailed source-code truncated for AI context efficiency. */

Developer/Guide Copy Section

Developer Guide

This developer guide documentation is intended to assist all contributors in various code contributions.
Any contribution to improving this documentation will be appreciated.

Table of Contents Copy Chapter

• Add New Kubernetes Resource Metric Collector
• Add New Metrics

Add New Kubernetes Resource Metric Collector Copy Section

The following steps are needed to introduce a new resource and its respective resource metrics.

• Reference your new resource(s) to the docs/README.md.
• Reference your new resource(s) in the docs/developer/cli-arguments.md as part of the --resources flag.
• Create a new <name-of-resource>.md in the docs directory to provide documentation on the resource(s) and metrics you implemented. Follow the formatting of all other resources.
• Add the resource(s) you are representing to the jsonnet/kube-state-metrics/kube-state-metrics.libsonnet under the appropriate apiGroup using the verbs: list and watch.
• Run make examples/standard, this should re-generate examples/standard/cluster-role.yaml with the resource(s) added to jsonnet/kube-state-metrics/kube-state-metrics.libsonnet.
• Reference and add build functions for the new resource(s) in internal/store/builder.go.
• Reference the new resource in pkg/options/resource.go.
• Add a sample Kubernetes manifest to be used by tests in the tests/manifests/ directory.
• Lastly, and most importantly, actually implement your new resource(s) and its test binary in internal/store. Follow the formatting and structure of other resources.

Add New Metrics Copy Section

• Make metrics experimental first when introducing them, refer #1910 for more information.

Metrics/Workload/Cronjob Metrics Copy Section

CronJob Metrics

Metrics/Workload/Daemonset Metrics Copy Section

DaemonSet Metrics

Metrics/Workload/Deployment Metrics Copy Section

Deployment Metrics

Metrics/Workload/Horizontalpodautoscaler Metrics Copy Section

Horizontal Pod Autoscaler Metrics

Metrics/Workload/Job Metrics Copy Section

Job Metrics

Metrics/Workload/Pod Metrics Copy Section

Pod Metrics

Note: For pod resource requests and limits, kube-scheduler exposes kube_pod_resource_request and kube_pod_resource_limit on its /metrics/resources endpoint. These are computed from the pod's effective resource requirements and already account for pod-level resources (pod.spec.resources, beta and enabled by default since Kubernetes 1.34), so they are recommended over the container-level metrics below. kube-state-metrics does not expose a separate metric for pod-level resources. These metrics are defined in KEP-1748.

Useful metrics queries Copy Chapter

How to retrieve non-standard Pod state Copy Section

It is not straightforward to get the Pod states for certain cases like "Terminating" and "Unknown" since it is not stored behind a field in the Pod.Status.

So to mimic the logic used by the kubectl command line, you will need to compose multiple metrics.

For example:

• To get the list of pods that are in the Unknown state, you can run the following PromQL query: sum(kube_pod_status_phase{phase="Unknown"}) by (namespace, pod) or (count(kube_pod_deletion_timestamp) by (namespace, pod) * sum(kube_pod_status_reason{reason="NodeLost"}) by(namespace, pod))

• For Pods in Terminating state: count(kube_pod_deletion_timestamp) by (namespace, pod) unless count(kube_pod_status_reason{reason="NodeLost"}) by (namespace, pod)

Here is an example of a Prometheus rule that can be used to alert on a Pod that has been in the Terminating state for more than 5m.

groups:
- name: Pod state
  rules:
  - alert: PodsBlockedInTerminatingState
    expr: count(kube_pod_deletion_timestamp) by (namespace, pod) unless count(kube_pod_status_reason{reason="NodeLost"}) by (namespace, pod)
    for: 5m
    labels:
      severity: page
    annotations:
      summary: Pod {{$labels.namespace}}/{{$labels.pod}} blocked in Terminating state.

Metrics/Workload/Replicaset Metrics Copy Section

ReplicaSet metrics

Metrics/Workload/Replicationcontroller Metrics Copy Section

ReplicationController metrics

Metrics/Workload/Statefulset Metrics Copy Section

Stateful Set Metrics

Metrics/Storage/Configmap Metrics Copy Section

ConfigMap Metrics

Metrics/Storage/Persistentvolume Metrics Copy Section

PersistentVolume Metrics

Useful metrics queries Copy Chapter

How to retrieve non-standard PV state Copy Section

It is not straightforward to get the PV states for certain cases like "Terminating" since it is not stored behind a field in the PersistentVolume.Status.

So to mimic the logic used by the kubectl command line, you will need to compose multiple metrics.

Here is an example of a Prometheus rule that can be used to alert on a PV that has been in the Terminating state for more than 5m.

groups:
- name: PV state
  rules:
  - alert: PVBlockedInTerminatingState
    expr: kube_persistentvolume_deletion_timestamp * on(persistentvolume) group_left() (kube_persistentvolume_status_phase{phase="Bound"} == 1) > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: PV {{$labels.persistentvolume}} blocked in Terminating state.

Metrics/Storage/Persistentvolumeclaim Metrics Copy Section

PersistentVolumeClaim Metrics

Note:

• An empty string will be used if PVC has no storage class.

Useful metrics queries Copy Chapter

How to retrieve non-standard PVC state Copy Section

It is not straightforward to get the PVC states for certain cases like "Terminating" since it is not stored behind a field in the PersistentVolumeClaim.Status.

So to mimic the logic used by the kubectl command line, you will need to compose multiple metrics.

Here is an example of a Prometheus rule that can be used to alert on a PVC that has been in the Terminating state for more than 5m.

groups:
- name: PVC state
  rules:
  - alert: PVCBlockedInTerminatingState
    expr: kube_persistentvolumeclaim_deletion_timestamp * on(namespace, persistentvolumeclaim) group_left() (kube_persistentvolumeclaim_status_phase{phase="Bound"} == 1) > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: PVC {{$labels.namespace}}/{{$labels.persistentvolumeclaim}} blocked in Terminating state.

Metrics/Storage/Secret Metrics Copy Section

Secret Metrics

Metrics/Storage/Storageclass Metrics Copy Section

StorageClass Metrics

Metrics/Storage/Volumeattachment Metrics Copy Section

VolumeAttachment Metrics

Metrics/Service/Endpoint Metrics Copy Section

Endpoint Metrics

Metrics/Service/Endpointslice Metrics Copy Section

Endpoint Metrics

Metrics/Service/Ingress Metrics Copy Section

Ingress Metrics

Metrics/Service/Ingressclass Metrics Copy Section

IngressClass Metrics

Metrics/Service/Service Metrics Copy Section

Service Metrics

Metrics/Policy/Limitrange Metrics Copy Section

LimitRange Metrics

Metrics/Policy/Networkpolicy Metrics Copy Section

Network Policy Metrics

Metrics/Policy/Poddisruptionbudget Metrics Copy Section

PodDisruptionBudget Metrics

Metrics/Policy/Resourcequota Metrics Copy Section

ResourceQuota Metrics

Metrics/Extend/Customresourcestate Metrics Copy Section

Custom Resource State Metrics

[!NOTE]
custom-resource-state is feature-frozen in favor of resource-state-metrics. Once resource-state-metrics is stable, custom-resource-state will be deprecated.

This section describes how to add metrics based on the state of a custom resource without writing a custom resource
registry and running your own build of KSM.

Configuration Copy Chapter

A YAML configuration file described below is required to define your custom resources and the fields to turn into metrics.

Two flags can be used:

• --custom-resource-state-config "inline yaml (see example)" or
• --custom-resource-state-config-file /path/to/config.yaml

When using a --config file, the equivalent YAML key is custom_resource_state_config_file. The pre-v2.17 key custom_resource_config_file is still honored as a deprecated alias and will be removed in a future release; please migrate to custom_resource_state_config_file.

If both flags are provided, the inline configuration will take precedence.
When multiple entries for the same resource exist, kube-state-metrics will exit with an error.
This includes configuration which refers to a different API version.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: kube-state-metrics
        args:
          - --custom-resource-state-config
          # in YAML files, | allows a multi-line string to be passed as a flag value
          # see https://yaml-multiline.info
          -  |
              kind: CustomResourceStateMetrics
              spec:
                resources:
                  - groupVersionKind:
                      group: myteam.io
                      version: "v1"
                      kind: Foo
                    metrics:
                      - name: active_count
                        help: "Count of active Foo"
                        each:
                          type: Gauge
                          ...

It's also possible to configure kube-state-metrics to run in a custom-resource-mode only. In addition to specifying one of --custom-resource-state-config* flags, you could set --custom-resource-state-only to true.
With this configuration only the known custom resources configured in --custom-resource-state-config* will be taken into account by kube-state-metrics.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: kube-state-metrics
        args:
          - --custom-resource-state-config
          # in YAML files, | allows a multi-line string to be passed as a flag value
          # see https://yaml-multiline.info
          -  |
              kind: CustomResourceStateMetrics
              spec:
                resources:
                  - groupVersionKind:
                      group: myteam.io
                      version: "v1"
                      kind: Foo
                    metrics:
                      - name: active_count
                        help: "Count of active Foo"
                        each:
                          type: Gauge
                          ...
          - --custom-resource-state-only=true

NOTE: The customresource_group, customresource_version, and customresource_kind common labels are reserved, and will be overwritten by the values from the groupVersionKind field.

RBAC-enabled Clusters Copy Section

Please be aware that kube-state-metrics needs list and watch permissions granted to customresourcedefinitions.apiextensions.k8s.io as well as to the resources you want to gather metrics from.

Examples Copy Section

The examples in this section will use the following custom resource:

kind: Foo
apiVersion: myteam.io/vl
metadata:
    annotations:
        bar: baz
        qux: quxx
    labels:
        foo: bar
    name: foo
spec:
    version: v1.2.3
    order:
        - id: 1
          value: true
        - id: 3
          value: false
    replicas: 1
    refs:
        - my_other_foo
        - foo_2
        - foo_with_extensions
status:
    phase: Pending
    active:
        type-a: 1
        type-b: 3
    conditions:
        - name: a
          value: 45
        - name: b
          value: 66
    sub:
        type-a:
            active: 1
            ready: 2
        type-b:
            active: 3
            ready: 4
    uptime: 43.21

Single Values

The config:

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        kind: "Foo"
        version: "v1"
      metrics:
        - name: "uptime"
          help: "Foo uptime"
          each:
            type: Gauge
            gauge:
              path: [status, uptime]

Produces the metric:

kube_customresource_uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21

Multiple Metrics/Kitchen Sink

/* Detailed source-code truncated for AI context efficiency. */

Produces the following metrics:

kube_customresource_ready_count{customresource_group="myteam.io", customresource_kind="Foo", 
customresource_version="v1", active="1",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-a",
lorem_bar="baz",lorem_qux="quxx",} 2
kube_customresource_ready_count{customresource_group="myteam.io", customresource_kind="Foo", 
customresource_version="v1", active="3",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-b",
lorem_bar="baz",lorem_qux="quxx",} 4

Non-map Arrays

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        kind: "Foo"
        version: "v1"
      labelsFromPath:
        name: [metadata, name]
      metrics:
        - name: "ref_info"
          help: "Reference to other Foo"
          each:
            type: Info
            info:
              # targeting an array will produce a metric for each element
              # labelsFromPath and value are relative to this path
              path: [spec, refs]

              # if path targets a list of values (e.g. strings or numbers, not objects or maps), individual values can
              # referenced by a label using this syntax
              labelsFromPath:
                ref: []

Produces the following metrics:

kube_customresource_ref_info{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", name="foo",ref="my_other_foo"} 1
kube_customresource_ref_info{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", name="foo",ref="foo_2"} 1
kube_customresource_ref_info{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", name="foo",ref="foo_with_extensions"} 1

Same Metrics with Different Labels

recommendation:
    containerRecommendations:
    - containerName: consumer
      lowerBound:
        cpu: 100m
        memory: 262144k

For example in VPA we have above attributes and we want to have a same metrics for both CPU and Memory, you can use below config:

/* Detailed source-code truncated for AI context efficiency. */

Produces the following metrics:

# HELP kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound Minimum memory resources the container can use before the VerticalPodAutoscaler updater evicts it.
# TYPE kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound gauge
kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound{container="consumer",customresource_group="autoscaling.k8s.io",customresource_kind="VerticalPodAutoscaler",customresource_version="v1",namespace="namespace-example",resource="memory",target_api_version="apps/v1",target_kind="Deployment",target_name="target-name-example",unit="byte",verticalpodautoscaler="vpa-example"} 123456
# HELP kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound Minimum cpu resources the container can use before the VerticalPodAutoscaler updater evicts it.
# TYPE kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound gauge
kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound{container="consumer",customresource_group="autoscaling.k8s.io",customresource_kind="VerticalPodAutoscaler",customresource_version="v1",namespace="namespace-example",resource="cpu",target_api_version="apps/v1",target_kind="Deployment",target_name="target-name-example",unit="core",verticalpodautoscaler="vpa-example"} 0.1

VerticalPodAutoscaler

In v2.9.0 the vericalpodautoscalers resource was removed from the list of default resources. In order to generate metrics for verticalpodautoscalers, you can use the following Custom Resource State config:

/* Detailed source-code truncated for AI context efficiency. */

The above configuration was tested on this VPA configuration, with an added annotation (foo: 123).

All VerticalPodAutoscaler Metrics

As an addition for the above configuration, here's the complete CustomResourceStateMetrics spec to re-enable all of the VPA metrics which are removed from the list of the default resources:

VPA CustomResourceStateMetrics

/* Detailed source-code truncated for AI context efficiency. */

Metric types Copy Section

The configuration supports three kind of metrics from the OpenMetrics specification.

The metric type is specified by the type field and its specific configuration at the types specific struct.

Gauge

Gauges are current measurements, such as bytes of memory currently used or the number of items in a queue. For gauges the absolute value is what is of interest to a user. [0]

Example:

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        kind: "Foo"
        version: "v1"
      metrics:
        - name: "uptime"
          help: "Foo uptime"
          each:
            type: Gauge
            gauge:
              path: [status, uptime]

Produces the metric:

kube_customresource_uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21

Type conversion and special handling

Gauges produce values of type float64 but custom resources can be of all kinds of types.
Kube-state-metrics performs implicit type conversions for a lot of type.
Supported types are:

• (u)int32/64, int, float32 and byte are cast to float64
• nil is generally mapped to 0.0 if NilIsZero is true, otherwise it will throw an error
• for bool true is mapped to 1.0 and false is mapped to 0.0
• for string the following logic applies
  • "true" and "yes" are mapped to 1.0, "false", "no" and "unknown" are mapped to 0.0 (all case-insensitive)
  • RFC3339 times are parsed to float timestamp
  • Quantities like "250m" or "512Gi" are parsed to float using https://github.com/kubernetes/apimachinery/blob/master/pkg/api/resource/quantity.go
  • Percentages ending with a "%" are parsed to float
  • finally the string is parsed to float using https://pkg.go.dev/strconv#ParseFloat which should support all common number formats. If that fails an error is yielded

Example for status conditions on Kubernetes Controllers

kind: CustomResourceStateMetrics
spec:
  resources:
  - groupVersionKind:
      group: myteam.io
      kind: "Foo"
      version: "v1"
    labelsFromPath:
      name:
      - metadata
      - name
      namespace:
      - metadata
      - namespace
    metrics:
    - name: "foo_status"
      help: "status condition "
      each:
        type: Gauge
        gauge:
          path: [status, conditions]
          labelsFromPath:
            type: ["type"]
          valueFrom: ["status"]

This will work for kubernetes controller CRs which expose status conditions according to the kubernetes api (https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#Condition):

status:
  conditions:
    - lastTransitionTime: "2019-10-22T16:29:31Z"
      status: "True"
      type: Ready

kube_customresource_foo_status{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", type="Ready"} 1.0

StateSet

StateSets represent a series of related boolean values, also called a bitset. If ENUMs need to be encoded this MAY be done via StateSet. [1]

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        kind: "Foo"
        version: "v1"
      metrics:
        - name: "status_phase"
          help: "Foo status_phase"
          each:
            type: StateSet
            stateSet:
              labelName: phase
              path: [status, phase]
              list: [Pending, Bar, Baz]

Metrics of type StateSet will generate a metric for each value defined in list for each resource.
The value will be 1, if the value matches the one in list.

If path (or valueFrom, if set) does not resolve, for example because the status field is not populated yet,
no metrics are produced for that resource and no error is logged. Once the field is set, the metrics appear.
A value that resolves to a non-string type is reported as an error.

Produces the metric:

kube_customresource_status_phase{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", phase="Pending"} 1
kube_customresource_status_phase{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", phase="Bar"} 0
kube_customresource_status_phase{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", phase="Baz"} 0

Info

Info metrics are used to expose textual information which SHOULD NOT change during process lifetime. Common examples are an application's version, revision control commit, and the version of a compiler. [2]

Metrics of type Info will always have a value of 1.

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        kind: "Foo"
        version: "v1"
      metrics:
        - name: "version"
          help: "Foo version"
          each:
            type: Info
            info:
              labelsFromPath:
                version: [spec, version]

Produces the metric:

kube_customresource_version{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", version="v1.2.3"} 1

Naming Copy Section

The default metric names are prefixed to avoid collisions with other metrics.
By default, a metric prefix of kube_ concatenated with your custom resource's group+version+kind is used.
You can override this behavior with the metricNamePrefix field.

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind: ...
      metricNamePrefix: myteam_foos
      metrics:
        - name: uptime
          # ...

Produces:

myteam_foos_uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21

To omit namespace and/or subsystem altogether, set them to the empty string:

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind: ...
      metricNamePrefix: ""
      metrics:
        - name: uptime
          # ...

Produces:

uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21

Logging Copy Section

If a metric path is registered but not found on a custom resource, an error will be logged. For some resources,
this may produce a lot of noise. The error log verbosity for a metric or resource can be set with errorLogV on
the resource or metric:

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind: ...
      errorLogV: 0  # 0 = default for errors
      metrics:
        - name: uptime
          errorLogV: 10  # only log at high verbosity

Path Syntax Copy Section

Paths are specified as a list of strings. Each string is a path segment, resolved dynamically against the data of the custom resource.
If any part of a path is missing, the result is nil.

Examples:

# simple path lookup
[spec, replicas]                         # spec.replicas == 1

# indexing an array
[spec, order, "0", value]                # spec.order[0].value = true

# finding an element in a list by key=value  
[status, conditions, "[name=a]", value]  # status.conditions[0].value = 45

# if the value to be matched is a number or boolean, the value is compared as a number or boolean  
[status, conditions, "[value=66]", name]  # status.conditions[1].name = "b"

# For generally matching against a field in an object schema, use the following syntax:
[metadata, "name=foo"] # if v, ok := metadata[name]; ok && v == "foo" { return v; } else { /* ignore */ }

Wildcard matching of version and kind fields Copy Section

The Custom Resource State (CRS hereon) configuration also allows you to monitor all versions and/or kinds that come under a group. It watches
the installed CRDs for this purpose. Taking the aforementioned Foo object as reference, the configuration below allows
you to monitor all objects under all versions and all kinds that come under the myteam.io group.

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: "myteam.io"
        version: "*" # Set to `v1 to monitor all kinds under `myteam.io/v1`. Wildcard matches all installed versions that come under this group.
        kind: "*" # Set to `Foo` to monitor all `Foo` objects under the `myteam.io` group (under all versions). Wildcard matches all installed kinds that come under this group (and version, if specified).
      metrics:
        - name: "myobject_info"
          help: "Foo Bar Baz"
          each:
            type: Info
            info:
              path: [metadata]
              labelsFromPath:
                object: [name]
                namespace: [namespace]

The configuration above produces these metrics.

kube_customresource_myobject_info{customresource_group="myteam.io",customresource_kind="Foo",customresource_version="v1",namespace="ns",object="foo"} 1
kube_customresource_myobject_info{customresource_group="myteam.io",customresource_kind="Bar",customresource_version="v1",namespace="ns",object="bar"} 1

Note

• Only versions that the API server actually serves (served: true in the CRD's spec.versions) are discovered. A version that is still declared in the CRD but no longer served cannot be listed or watched, so the wildcard will not resolve to it, and pinning it explicitly will not generate any metrics. Such skipped versions are logged at verbosity level 1 (-v=1).
• For cases where the GVKs defined in a CRD have multiple versions under a single group for the same kind, as expected, the wildcard value will resolve to all versions, but a query for any specific version will return all resources under all versions, in that versions' representation. This basically means that for two such versions A and B, if a resource exists under B, it will reflect in the metrics generated for A as well, in addition to any resources of itself, and vice-versa. This logic is based on the current listing behavior of the client-go library.
• The introduction of this feature further discourages (and discontinues) the use of native objects in the CRS featureset, since these do not have an explicit CRD associated with them, and conflict with internal stores defined specifically for such native resources. Please consider opening an issue or raising a PR if you'd like to expand on the current metric labelsets for them. Also, any such configuration will be ignored, and no metrics will be generated for the same.