### README # Documentation This documentation is intended to be a complete reflection of the current state of the exposed metrics of kube-state-metrics. Any contribution to improving this documentation or adding sample usages will be appreciated. ## Table of Contents * [Metrics Stages](#metrics-stages) * [Exposed Metrics](#exposed-metrics) * [PromQL Queries](#promql-queries) * [Join Metrics](#join-metrics) * [CLI arguments](#cli-arguments) ## Metrics Stages Stages about metrics are grouped into three categories： | Stage | Description | | ------------ | -------------------------------------------------------------------------------------------------------------------------- | | EXPERIMENTAL | Metrics which normally correspond to the Kubernetes API object alpha status or spec fields and can be changed at any time. | | STABLE | Metrics which should have very few backwards-incompatible changes outside of major version updates. | | DEPRECATED | Metrics which will be removed once the deprecation timeline is met. | ## Opt-in Metrics As of v2.3.0, kube-state-metrics supports additional opt-in metrics via the CLI flag `--metric-opt-in-list`. See the metric documentation to identify which metrics need to be specified. ## Exposed Metrics Per group of metrics there is one file for each metrics. See each file for specific documentation about the exposed metrics: ### Default Resources * [CertificateSigningRequest Metrics](metrics/auth/certificatesigningrequest-metrics.md) * [ConfigMap Metrics](metrics/storage/configmap-metrics.md) * [CronJob Metrics](metrics/workload/cronjob-metrics.md) * [DaemonSet Metrics](metrics/workload/daemonset-metrics.md) * [Deployment Metrics](metrics/workload/deployment-metrics.md) * [Endpoint Metrics](metrics/service/endpoint-metrics.md) * [Horizontal Pod Autoscaler Metrics](metrics/workload/horizontalpodautoscaler-metrics.md) * [Ingress Metrics](metrics/service/ingress-metrics.md) * [Job Metrics](metrics/workload/job-metrics.md) * [Lease Metrics](metrics/cluster/lease-metrics.md) * [LimitRange Metrics](metrics/policy/limitrange-metrics.md) * [MutatingAdmissionPolicy Metrics](metrics/extend/mutatingadmissionpolicy-metrics.md) * [MutatingAdmissionPolicyBinding Metrics](metrics/extend/mutatingadmissionpolicybinding-metrics.md) * [MutatingWebhookConfiguration Metrics](metrics/extend/mutatingwebhookconfiguration-metrics.md) * [Namespace Metrics](metrics/cluster/namespace-metrics.md) * [NetworkPolicy Metrics](metrics/policy/networkpolicy-metrics.md) * [Node Metrics](metrics/cluster/node-metrics.md) * [PersistentVolume Metrics](metrics/storage/persistentvolume-metrics.md) * [PersistentVolumeClaim Metrics](metrics/storage/persistentvolumeclaim-metrics.md) * [Pod Disruption Budget Metrics](metrics/policy/poddisruptionbudget-metrics.md) * [Pod Metrics](metrics/workload/pod-metrics.md) * [ReplicaSet Metrics](metrics/workload/replicaset-metrics.md) * [ReplicationController Metrics](metrics/workload/replicationcontroller-metrics.md) * [ResourceQuota Metrics](metrics/policy/resourcequota-metrics.md) * [Secret Metrics](metrics/storage/secret-metrics.md) * [Service Metrics](metrics/service/service-metrics.md) * [StatefulSet Metrics](metrics/workload/statefulset-metrics.md) * [StorageClass Metrics](metrics/storage/storageclass-metrics.md) * [ValidatingAdmissionPolicy Metrics](metrics/extend/validatingadmissionpolicy-metrics.md) * [ValidatingAdmissionPolicyBinding Metrics](metrics/extend/validatingadmissionpolicybinding-metrics.md) * [ValidatingWebhookConfiguration Metrics](metrics/extend/validatingwebhookconfiguration-metrics.md) * [VolumeAttachment Metrics](metrics/storage/volumeattachment-metrics.md) ### Optional Resources * [ClusterRole Metrics](metrics/cluster/clusterrole-metrics.md) * [ClusterRoleBinding Metrics](metrics/cluster/clusterrolebinding-metrics.md) * [EndpointSlice Metrics](metrics/service/endpointslice-metrics.md) * [IngressClass Metrics](metrics/service/ingressclass-metrics.md) * [Role Metrics](metrics/auth/role-metrics.md) * [RoleBinding Metrics](metrics/auth/rolebinding-metrics.md) * [ServiceAccount Metrics](metrics/auth/serviceaccount-metrics.md) ## PromQL Queries For practical PromQL query examples and alerting rules: * [StatefulSet Queries & Alerts](queries/statefulset-monitoring.md) ## Join Metrics When an additional, not provided by default label is needed, a [Prometheus matching operator](https://prometheus.io/docs/prometheus/latest/querying/operators/#vector-matching) can be used to extend single metrics output. This example adds `label_release` to the set of default labels of the `kube_pod_status_ready` metric and allows you select or group the metrics by Helm release label: ```promql kube_pod_status_ready * on (namespace, pod) group_left(label_release) kube_pod_labels ``` Another useful example would be to query the memory usage of pods by its `phase`, such as `Running`: ```promql sum(kube_pod_container_resource_requests{resource="memory"}) by (namespace, pod, node) * on (namespace, pod) group_left() (sum(kube_pod_status_phase{phase="Running"}) by (pod, namespace) == 1) ``` ## Metrics from Custom Resources > [!NOTE] > `custom-resource-state` is feature-frozen in favor of [resource-state-metrics](https://github.com/kubernetes-sigs/resource-state-metrics). Once `resource-state-metrics` is stable, `custom-resource-state` will be deprecated. See [Custom Resource State Metrics](metrics/extend/customresourcestate-metrics.md) for experimental support for custom resources. ## CLI Arguments Additionally, options for `kube-state-metrics` can be passed when executing as a CLI, or in a kubernetes / openshift environment. More information can be found here: [CLI Arguments](developer/cli-arguments.md) ## Protecting /metrics endpoints Kube-State-Metrics' metrics can contain sensitive information about the state of the cluster, which you as an operator might want to additionally protect from unauthorized access. In order to achieve this, you need to enable the `--auth-filter` flag on kube-state-metrics. With this, kube-state-metrics will only accept authenticated and authorized requests to the /metrics endpoints. Kube-state-metrics uses Kubernetes' RBAC mechanisms for this, so this means that every scrape will trigger a request against the API Server for TokenReview and SubjectAccessReview. The clients scraping the endpoint, need to use a token which can be provided by a ServiceAccount that can be set up the following way: A ClusterRole providing access like this: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: metrics-reader rules: - nonResourceURLs: - "/metrics" verbs: - get ``` and a matching ClusterRoleBinding ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: metrics-reader-rolebinding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: metrics-reader subjects: - kind: ServiceAccount name: YOUR_SERVICE_ACCOUNT namespace: NAMESPACE_OF_THE_SERVICE_ACCOUNT ``` Your client can then use either this ServiceAccount to gather metrics or you can create a token, that can be used to fetch data like this: ```bash TOKEN=$(kubectl create token YOUR_SERVICE_ACCOUNT -n NAMESPACE_OF_THE_SERVICE_ACCOUNT) curl -H "Authorization: Bearer $TOKEN" KUBE_STATE_METRICS_URL:8080/metrics ``` --- ### Design/Metrics Best Practices # Kube-State-Metrics - Timeseries best practices --- Author: Manuel Rüger () Date: October 17th 2024 --- ## Introduction Kube-State-Metrics' goal is to provide insights into the state of Kubernetes objects by exposing them as metrics. This document provides guidelines with the goal to create a good user experience when using these metrics. Please be aware that this document is introduced in a later stage of the project and there might be metrics that do not follow these best practices. Feel encouraged to report these metrics and provide a pull request to improve them. ## General best practices We follow [Prometheus](https://prometheus.io/docs/practices/naming/) best practices in terms of naming and labeling. ## Best practices for kube-state-metrics ### Avoid pre-computation kube-state-metrics should expose metrics on an individual object level and avoid any sort of pre-computation unless it is required due to for example high cardinality on objects. We prefer not to add metrics that can be derived from existing raw metrics. For example, we would not want to expose a metric called `kube_pod_total` as it can be computed with `count(kube_pod_info)`. This way kube-state-metrics allows the user to have full control on how they want to use the metrics and gives them flexibility to do specific computation. ### Static object properties An object usually has a stable set of properties that do not change during its lifecycle in Kubernetes. This includes properties like name, namespace, uid etc. that have a 1:1 relationship with the object. It is a good practice to group those together into an `_info` metric. If there is a 1:n relationship (e.g. a list of ports), it should be in a separate metric to avoid generating too many metrics. ### Dynamic object properties An object can also have a dynamic set of properties, which are usually part of the status field. These change during the lifecycle of the object. For example a pod can be in different states like "Pending", "Running" etc. These should be part of a "State Set" that includes labels that identify the object as well as the dynamic property. ### Linked properties If an object contains a substructure that links multiple properties together (e.g. endpoint address and port), those should be reported in the same metric. ### Optional properties Some Kubernetes objects have optional fields. In case there is an optional value, the label should still be exposed, ideally as an empty string. ### Timestamps Timestamps like creation time or modification time should be exposed as a value. The metric should end with `_timestamp_seconds`. The date value is represented in [UNIX epoch seconds](https://en.wikipedia.org/wiki/Unix_time). ### Cardinality Some object properties can cause cardinality issues if they can contain a lot of different values or are linked together with multiple properties that also can change a lot. In this case it is better to limit the number of values that can be exposed within kube-state-metrics by allowing only a few of them and have a default for others. If for example the Kubernetes object contains a status field that contains an error message that can change a lot, it would be better to have a boolean `error="true"` label in case there is an error. If there are some error messages that are worth exposing, those could be exposed and for any other message, a default value could be provided. ## Stability We follow the stability framework derived from Kubernetes, in which we expose experimental and stable metrics. Experimental metrics are recently introduced or expose alpha/beta resources in the Kubernetes API. They can change anytime and should be used with caution. They can be promoted to a stable metric once the object stabilized in the Kubernetes API or they were part of two consecutive releases and haven't observed any changes in them. Stable metrics are considered frozen with the exception of new labels being added. A stable metric or a label on a stable metric can be deprecated in release Major.Minor and the earliest point it will be removed is the release Major.Minor+2. --- ### Design/Metrics Store Performance Optimization # Kube-State-Metrics - Performance Optimization Proposal --- Author: Max Inden () Date: 23. July 2018 Target release: v1.5.0 --- ## Glossary * kube-state-metrics: “Simple service that listens to the Kubernetes API server and generates metrics about the state of the objects” * Time series: A single line in a /metrics response e.g. “metric_name{label="value"} 1” ## Problem Statement There has been repeated reports of two issues running kube-state-metrics on production Kubernetes clusters. First kube-state-metrics takes a long time (“10s - 20s”) to respond on its /metrics endpoint, leading to Prometheus instances dropping the scrape interval request and marking the given time series as stale. Second kube-state-metrics uses a lot of memory and thereby being out-of-memory killed due to low set Kubernetes resource limits. ## Goal The goal of this proposal can be split into the following sub-goals ordered by their priority: 1. Decrease response time on /metrics endpoint 2. Decrease overall runtime memory usage ## Status Quo Instead of requesting the needed information from the Kubernetes API-Server on demand (on scrape), kube-state-metrics uses the Kubernetes client-go cache tool to keep a full in memory representation of all Kubernetes objects of a given cluster. Using the cache speeds up the performance critical path of replying to a scrape request, and reduces the load on the Kubernetes API-Server by only sending deltas whenever they occur. Kube-state-metrics does not make use of all properties and sub-objects of these Kubernetes objects that it stores in its cache. On a scrape request by e.g. Prometheus on the /metrics endpoint kube-state-metrics calculates the configured time series on demand based on the objects in its cache and converts them to the Prometheus string representation. ## Proposal Instead of a full representation of all Kubernetes objects with all its properties in memory via the Kubernetes client-go cache, use a map, addressable by the Kubernetes object uuid, containing all time series of that object as a single multi-line string. ```go var cache = map[uuid][]byte{} ``` Kube-state-metrics listens on add, update and delete events via Kubernetes client-go reflectors. On add and update events kube-state-metrics generates all time series related to the Kubernetes object based on the event’s payload, concatenates the time series to a single byte slice and sets / replaces the byte slice in the store at the uuid of the Kubernetes object. One can precompute the length of a time series byte slice before allocation as the sum of the length of the metric name, label keys and values as well as the metric value in string representation. On delete events kube-state-metrics deletes the uuid entry of the given Kubernetes object in the cache map. On a scrape request on the /metrics endpoint, kube-state-metrics iterates over the cache map and concatenates all time series string blobs into a single string, which is finally passed on as a response. ``` /* Detailed source-code truncated for AI context efficiency. */ ```
Code to reproduce diagram Build via [text-diagram](http://weidagang.github.io/text-diagram/) ```text object pod_reflector pod_store pod_collector metrics_endpoint note left of pod_reflector: new pod p1 pod_reflector -> pod_store: Add(p1) note right of pod_store: generateMetrics(p1) pod_store -> pod_reflector: nil note right of metrics_endpoint: GET /metrics metrics_endpoint -> pod_collector: Collect() pod_collector -> pod_store: GetAll() pod_store -> pod_collector: []string{metrics} pod_collector -> metrics_endpoint: concat(metrics) ```
## FAQ / Follow up improvements * If kube-state-metrics only listens on add, update and delete events, how is it aware of already existing Kubernetes objects created before kube-state-metrics was started? Leveraging Kubernetes client-go, reflectors can initialize all existing objects before any add, update or delete events. To ensure no events are missed in the long run, periodic resyncs via Kubernetes client-go can be triggered. This extra confidence is not a must and should be compared to its costs, as Kubernetes client-go already gives decent guarantees on event delivery. * What about metadata (HELP and description) in the /metrics output? As a first iteration they would be skipped until we have a better idea on the design. * How can the cache map be concurrently accessed? The core golang map implementation is not thread-safe. As a first iteration a simple mutex should be sufficient. Golang's sync.Map might be considered. * To solve the problem of out of order events send by the Kubernetes API-Server to kube-state-metrics, to each blob of time series inside the cache map it can keep the Kubernetes resource version. On add and update events, first compare the resource version of the event with than the resource version in the cache. Only move forward if the former is higher than the latter. * In case the memory consumption of the time series string blobs is a problem the following optimization can be considered: Among the time series strings, multiple sub-strings will be heavily duplicated like the metric name. Instead of saving unstructured strings inside the cache map, one can structure them, using pointers to deduplicate e.g. metric names. * Kube-state-metrics does not make use of all properties of all Kubernetes objects. Instead of unmarshalling unused properties, their json struct tags or their Protobuf representation could be removed. --- ### Developer/Cli Arguments # Command line arguments kube-state-metrics can be configured through command line arguments. Those arguments can be passed during startup when running locally: `kube-state-metrics --telemetry-port=8081 --kubeconfig= --apiserver= ...` Or configured in the `args` section of your deployment configuration in a Kubernetes / Openshift context: ```yaml spec: template: spec: containers: - args: - '--telemetry-port=8081' - '--kubeconfig=' - '--apiserver=' ``` ## Available options ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Developer/Guide # Developer Guide This developer guide documentation is intended to assist all contributors in various code contributions. Any contribution to improving this documentation will be appreciated. ## Table of Contents * [Add New Kubernetes Resource Metric Collector](#add-new-kubernetes-resource-metric-collector) * [Add New Metrics](#add-new-metrics) ### Add New Kubernetes Resource Metric Collector The following steps are needed to introduce a new resource and its respective resource metrics. * Reference your new resource(s) to the [docs/README.md](./../README.md#exposed-metrics). * Reference your new resource(s) in the [docs/developer/cli-arguments.md](./cli-arguments.md#available-options) as part of the `--resources` flag. * Create a new `.md` in the [docs](./../docs) directory to provide documentation on the resource(s) and metrics you implemented. Follow the formatting of all other resources. * Add the resource(s) you are representing to the [jsonnet/kube-state-metrics/kube-state-metrics.libsonnet](./../../jsonnet/kube-state-metrics/kube-state-metrics.libsonnet) under the appropriate `apiGroup` using the `verbs`: `list` and `watch`. * Run `make examples/standard`, this should re-generate [examples/standard/cluster-role.yaml](./../../examples/standard/cluster-role.yaml) with the resource(s) added to [jsonnet/kube-state-metrics/kube-state-metrics.libsonnet](./../../jsonnet/kube-state-metrics/kube-state-metrics.libsonnet). * Reference and add build functions for the new resource(s) in [internal/store/builder.go](./../../internal/store/builder.go). * Reference the new resource in [pkg/options/resource.go](./../../pkg/options/resource.go). * Add a sample Kubernetes manifest to be used by tests in the [tests/manifests/](./../../tests/manifests) directory. * Lastly, and most importantly, actually implement your new resource(s) and its test binary in [internal/store](./../../internal/store). Follow the formatting and structure of other resources. ### Add New Metrics * Make metrics experimental first when introducing them, refer [#1910](https://github.com/kubernetes/kube-state-metrics/pull/1910) for more information. | Metric stability level | | |------------------------|--------------------| | EXPERIMENTAL | basemetrics.ALPHA | | STABLE | basemetrics.STABLE | --- ### Metrics/Workload/Cronjob Metrics # CronJob Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ---------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_cronjob_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace>
`annotation_CRONJOB_ANNOTATION`=<CRONJOB_ANNOTATION> | EXPERIMENTAL | | kube_cronjob_info | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace>
`schedule`=<schedule>
`concurrency_policy`=<concurrency-policy>
`timezone`=<timezone> | STABLE | | kube_cronjob_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace>
`label_CRONJOB_LABEL`=<CRONJOB_LABEL> | STABLE | | kube_cronjob_created | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | STABLE | | kube_cronjob_next_schedule_time | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | STABLE | | kube_cronjob_schedule_invalid | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | EXPERIMENTAL | | kube_cronjob_status_active | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | STABLE | | kube_cronjob_status_last_schedule_time | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | STABLE | | kube_cronjob_status_last_successful_time | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | STABLE | | kube_cronjob_spec_suspend | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | STABLE | | kube_cronjob_spec_starting_deadline_seconds | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | STABLE | | kube_cronjob_metadata_resource_version | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | STABLE | | kube_cronjob_spec_successful_job_history_limit | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | EXPERIMENTAL | | kube_cronjob_spec_failed_job_history_limit | Gauge | | `cronjob`=<cronjob-name>
`namespace`=<cronjob-namespace> | EXPERIMENTAL | --- ### Metrics/Workload/Daemonset Metrics # DaemonSet Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ---------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_daemonset_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace>
`annotation_DAEMONSET_ANNOTATION`=<DAEMONSET_ANNOTATION> | EXPERIMENTAL | | kube_daemonset_created | Gauge | | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | STABLE | | kube_daemonset_status_current_number_scheduled | Gauge | | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | STABLE | | kube_daemonset_status_desired_number_scheduled | Gauge | | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | STABLE | | kube_daemonset_status_number_available | Gauge | | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | STABLE | | kube_daemonset_status_number_misscheduled | Gauge | | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | STABLE | | kube_daemonset_status_number_ready | Gauge | | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | STABLE | | kube_daemonset_status_number_unavailable | Gauge | | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | STABLE | | kube_daemonset_status_observed_generation | Gauge | | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | STABLE | | kube_daemonset_status_updated_number_scheduled | Gauge | | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | STABLE | | kube_daemonset_metadata_generation | Gauge | | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | STABLE | | kube_daemonset_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace>
`label_DAEMONSET_LABEL`=<DAEMONSET_LABEL> | STABLE | | kube_daemonset_deletion_timestamp | Gauge | Unix deletion timestamp | `daemonset`=<daemonset-name>
`namespace`=<daemonset-namespace> | EXPERIMENTAL | --- ### Metrics/Workload/Deployment Metrics # Deployment Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ----------------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_deployment_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `deployment`=<deployment-name>
`namespace`=<deployment-namespace>
`annotation_DEPLOYMENT_ANNOTATION`=<DEPLOYMENT_ANNOTATION> | EXPERIMENTAL | | kube_deployment_status_replicas | Gauge | The number of replicas per deployment. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_status_replicas_ready | Gauge | The number of ready replicas per deployment. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_status_replicas_available | Gauge | The number of available replicas per deployment. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_status_replicas_unavailable | Gauge | The number of unavailable replicas per deployment. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_status_replicas_updated | Gauge | The number of updated replicas per deployment. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_status_terminating_replicas | Gauge | The number of terminating replicas per deployment. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | ALPHA | | kube_deployment_status_observed_generation | Gauge | The generation observed by the deployment controller. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_status_condition | Gauge | The current status conditions of a deployment. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace>
`reason`=<deployment-transition-reason>
`condition`=<deployment-condition>
`status`=<true\|false\|unknown> | STABLE | | kube_deployment_spec_replicas | Gauge | Number of desired pods for a deployment. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_spec_paused | Gauge | Whether the deployment is paused and will not be processed by the deployment controller. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_spec_strategy_rollingupdate_max_unavailable | Gauge | Maximum number of unavailable replicas during a rolling update of a deployment. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_spec_strategy_rollingupdate_max_surge | Gauge | Maximum number of replicas that can be scheduled above the desired number of replicas during a rolling update of a deployment. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_metadata_generation | Gauge | Sequence number representing a specific generation of the desired state. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `deployment`=<deployment-name>
`namespace`=<deployment-namespace>
`label_DEPLOYMENT_LABEL`=<DEPLOYMENT_LABEL> | STABLE | | kube_deployment_created | Gauge | Unix creation timestamp | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | STABLE | | kube_deployment_deletion_timestamp | Gauge | Unix deletion timestamp | `deployment`=<deployment-name>
`namespace`=<deployment-namespace> | EXPIREMENTAL | | kube_deployment_owner | Gauge | Information about the Deployment's owner. | `deployment`=<deployment-name>
`namespace`=<deployment-namespace>
`owner_kind`=<owner-kind>
`owner_name`=<owner-name> | ALPHA | --- ### Metrics/Workload/Horizontalpodautoscaler Metrics # Horizontal Pod Autoscaler Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ---------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_horizontalpodautoscaler_info | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace>
`scaletargetref_api_version`=<hpa-target-api-version>
`scaletargetref_kind`=<hpa-target-kind>
`scaletargetref_name`=<hpa-target-name> | EXPERIMENTAL | | kube_horizontalpodautoscaler_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | EXPERIMENTAL | | kube_horizontalpodautoscaler_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | STABLE | | kube_horizontalpodautoscaler_metadata_generation | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | STABLE | | kube_horizontalpodautoscaler_spec_max_replicas | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | STABLE | | kube_horizontalpodautoscaler_spec_min_replicas | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | STABLE | | kube_horizontalpodautoscaler_spec_target_metric | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace>
`metric_name`=<metric-name>
`metric_target_type`=<value\|utilization\|average>
`container`=<container-name> | EXPERIMENTAL | | kube_horizontalpodautoscaler_spec_behavior_scale_down_tolerance | Gauge | The tolerance on the ratio between the current and desired metric value below which no scale down occurs. | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | EXPERIMENTAL | | kube_horizontalpodautoscaler_spec_behavior_scale_up_tolerance | Gauge | The tolerance on the ratio between the current and desired metric value below which no scale up occurs. | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | EXPERIMENTAL | | kube_horizontalpodautoscaler_status_target_metric | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace>
`metric_name`=<metric-name>
`metric_target_type`=<value\|utilization\|average>
`container`=<container-name> | STABLE | | kube_horizontalpodautoscaler_status_condition | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace>
`condition`=<hpa-condition>
`status`=<true\|false\|unknown> | STABLE | | kube_horizontalpodautoscaler_status_current_replicas | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | STABLE | | kube_horizontalpodautoscaler_status_desired_replicas | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | STABLE | | kube_horizontalpodautoscaler_created | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | EXPERIMENTAL | | kube_horizontalpodautoscaler_deletion_timestamp | Gauge | | `horizontalpodautoscaler`=<hpa-name>
`namespace`=<hpa-namespace> | EXPERIMENTAL | --- ### Metrics/Workload/Job Metrics # Job Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_job_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `job_name`=<job-name>
`namespace`=<job-namespace>
`annotation_JOB_ANNOTATION`=<JOB_ANNOTATION> | EXPERIMENTAL | | kube_job_info | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace> | STABLE | | kube_job_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `job_name`=<job-name>
`namespace`=<job-namespace>
`label_JOB_LABEL`=<JOB_LABEL> | STABLE | | kube_job_owner | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace>
`owner_kind`=<owner kind>
`owner_name`=<owner name>
`owner_is_controller`=<whether owner is controller> | STABLE | | kube_job_spec_parallelism | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace> | STABLE | | kube_job_spec_completions | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace> | STABLE | | kube_job_spec_active_deadline_seconds | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace> | STABLE | | kube_job_status_active | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace> | STABLE | | kube_job_status_succeeded | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace> | STABLE | | kube_job_status_failed | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace>
`reason`=<failure reason> | STABLE | | kube_job_status_start_time | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace> | STABLE | | kube_job_status_completion_time | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace> | STABLE | | kube_job_complete | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace>
`condition`=<true\|false\|unknown> | STABLE | | kube_job_failed | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace>
`condition`=<true\|false\|unknown> | STABLE | | kube_job_created | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace> | STABLE | | kube_job_status_suspended | Gauge | | `job_name`=<job-name>
`namespace`=<job-namespace> | EXPERIMENTAL | | kube_job_status_ready | Gauge | The number of ready pods that belong to this Job. | `job_name`=<job-name>
`namespace`=<job-namespace> | EXPERIMENTAL | --- ### Metrics/Workload/Pod Metrics # Pod Metrics > Note: For pod resource requests and limits, kube-scheduler exposes `kube_pod_resource_request` and `kube_pod_resource_limit` on its `/metrics/resources` endpoint. These are computed from the pod's effective resource requirements and already account for pod-level resources (`pod.spec.resources`, beta and enabled by default since Kubernetes 1.34), so they are recommended over the container-level metrics below. kube-state-metrics does not expose a separate metric for pod-level resources. These metrics are defined in [KEP-1748](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/1748-pod-resource-metrics). | Metric name | Metric type | Description | Unit (where applicable) | Labels/tags | Status | Opt-in | | ----------------------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | ------ | | kube_pod_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`annotation_POD_ANNOTATION`=<POD_ANNOTATION>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_info | Gauge | Information about pod | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`host_ip`=<host-ip>
`pod_ip`=<pod-ip>
`node`=<node-name>
`created_by_kind`=<created_by_kind>
`created_by_name`=<created_by_name>
`uid`=<pod-uid>
`priority_class`=<priority_class>
`host_network`=<host_network> | STABLE | - | | kube_pod_ips | Gauge | Pod IP addresses | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`ip`=<pod-ip-address>
`ip_family`=<4 OR 6>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_start_time | Gauge | Start time in unix timestamp for a pod | seconds | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_completion_time | Gauge | Completion time in unix timestamp for a pod | seconds | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_owner | Gauge | Information about the Pod's owner | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`owner_kind`=<owner kind>
`owner_name`=<owner name>
`owner_is_controller`=<whether owner is controller>
`uid`=<pod-uid> | STABLE | - | | kube_pod_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`label_POD_LABEL`=<POD_LABEL>
`uid`=<pod-uid> | STABLE | - | | kube_pod_nodeselectors | Gauge | Describes the Pod nodeSelectors | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`nodeselector_NODE_SELECTOR`=<NODE_SELECTOR>
`uid`=<pod-uid> | EXPERIMENTAL | Opt-in | | kube_pod_status_phase | Gauge | The pods current phase | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`phase`=<Pending\|Running\|Succeeded\|Failed\|Unknown>
`uid`=<pod-uid> | STABLE | - | | kube_pod_status_qos_class | Gauge | The pods current qosClass | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`qos_class`=<BestEffort\|Burstable\|Guaranteed>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_status_ready | Gauge | Describes whether the pod is ready to serve requests | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`condition`=<true\|false\|unknown>
`uid`=<pod-uid> | STABLE | - | | kube_pod_status_scheduled | Gauge | Describes the status of the scheduling process for the pod | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`condition`=<true\|false\|unknown>
`uid`=<pod-uid> | STABLE | - | | kube_pod_container_info | Gauge | Information about a container in a pod | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`image`=<image-name>
`image_id`=<image-id>
`image_spec`=<image-spec>
`container_id`=<containerid>
`uid`=<pod-uid> | STABLE | - | | kube_pod_container_status_waiting | Gauge | Describes whether the container is currently in waiting state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_container_status_waiting_reason | Gauge | Describes the reason the container is currently in waiting state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`reason`=<container-waiting-reason>
`uid`=<pod-uid> | STABLE | - | | kube_pod_container_status_running | Gauge | Describes whether the container is currently in running state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_container_state_started | Gauge | Start time in unix timestamp for a pod container | seconds | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_container_status_terminated | Gauge | Describes whether the container is currently in terminated state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_container_status_terminated_reason | Gauge | Describes the reason the container is currently in terminated state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`reason`=<container-terminated-reason>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_container_status_last_terminated_reason | Gauge | Describes the last reason the container was in terminated state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`reason`=<last-terminated-reason>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_container_status_last_terminated_exitcode | Gauge | Describes the exit code for the last container in terminated state. | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_container_status_last_terminated_timestamp | Gauge | Last terminated time for a pod container in unix timestamp. | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_container_status_ready | Gauge | Describes whether the containers readiness check succeeded | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_status_initialized_time | Gauge | Time when the pod is initialized. | seconds | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_status_ready_time | Gauge | Time when pod passed readiness probes. | seconds | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_status_container_ready_time | Gauge | Time when the container of the pod entered Ready state. | seconds | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_container_status_restarts_total | Counter | The number of container restarts per container | | `container`=<container-name>
`namespace`=<pod-namespace>
`pod`=<pod-name>
`uid`=<pod-uid> | STABLE | - | | kube_pod_container_resource_requests | Gauge | The number of requested request resource by a container. It is recommended to use the `kube_pod_resource_request` metric exposed by kube-scheduler instead, as it is more precise. | `cpu`=<core>
`memory`=<bytes> | `resource`=<resource-name>
`unit`=<resource-unit>
`container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`node`=< node-name>
`uid`=<pod-uid> | STABLE | - | | kube_pod_container_resource_limits | Gauge | The number of requested limit resource by a container. It is recommended to use the `kube_pod_resource_limit` metric exposed by kube-scheduler instead, as it is more precise. | `cpu`=<core>
`memory`=<bytes> | `resource`=<resource-name>
`unit`=<resource-unit>
`container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`node`=< node-name>
`uid`=<pod-uid> | STABLE | - | | kube_pod_overhead_cpu_cores | Gauge | The pod overhead in regards to cpu cores associated with running a pod | core | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_overhead_memory_bytes | Gauge | The pod overhead in regards to memory associated with running a pod | bytes | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_runtimeclass_name_info | Gauge | The runtimeclass associated with the pod | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_created | Gauge | Unix creation timestamp | seconds | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_deletion_timestamp | Gauge | Unix deletion timestamp | seconds | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_restart_policy | Gauge | Describes the restart policy in use by this pod | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`type`=<Always\|Never\|OnFailure>
`uid`=<pod-uid> | STABLE | - | | kube_pod_init_container_info | Gauge | Information about an init container in a pod | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`image`=<image-name>
`image_id`=<image-id>
`image_spec`=<image-spec>
`container_id`=<containerid>
`uid`=<pod-uid>
`restart_policy`=<restart-policy> | STABLE | - | | kube_pod_init_container_state_started | Gauge | Start time in unix timestamp for a pod init container | seconds | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_init_container_status_waiting | Gauge | Describes whether the init container is currently in waiting state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_init_container_status_waiting_reason | Gauge | Describes the reason the init container is currently in waiting state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`reason`=<container-waiting-reason>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_init_container_status_running | Gauge | Describes whether the init container is currently in running state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_init_container_status_terminated | Gauge | Describes whether the init container is currently in terminated state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_init_container_status_terminated_reason | Gauge | Describes the reason the init container is currently in terminated state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`reason`=<container-terminated-reason>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_init_container_status_last_terminated_reason | Gauge | Describes the last reason the init container was in terminated state | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`reason`=<last-terminated-reason>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_init_container_status_last_terminated_exitcode | Gauge | Describes the exit code for the last init container in terminated state. | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_init_container_status_last_terminated_timestamp | Gauge | Last terminated time for a pod init container in unix timestamp | seconds | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_init_container_status_ready | Gauge | Describes whether the init containers readiness check succeeded | | `container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_init_container_status_restarts_total | Counter | The number of restarts for the init container | integer | `container`=<container-name>
`namespace`=<pod-namespace>
`pod`=<pod-name>
`uid`=<pod-uid> | STABLE | - | | kube_pod_init_container_resource_limits | Gauge | The number of CPU cores requested limit by an init container | `cpu`=<core>
`memory`=<bytes> | `resource`=<resource-name>
`unit`=<resource-unit>
`container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`node`=< node-name>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_init_container_resource_requests | Gauge | The number of CPU cores requested by an init container | `cpu`=<core>
`memory`=<bytes> | `resource`=<resource-name>
`unit`=<resource-unit>
`container`=<container-name>
`pod`=<pod-name>
`namespace`=<pod-namespace>
`node`=< node-name>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_spec_volumes_persistentvolumeclaims_info | Gauge | Information about persistentvolumeclaim and ephemeral volumes in a pod. For `ephemeral="true"`, the `persistentvolumeclaim` label is the generated claim name `-`; non-ephemeral volumes retain their declared claim name. | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`volume`=<volume-name>
`persistentvolumeclaim`=<persistentvolumeclaim-claimname>
`ephemeral`=<true\|false>
`uid`=<pod-uid> | STABLE | - | | kube_pod_spec_volumes_persistentvolumeclaims_readonly | Gauge | Describes whether a persistentvolumeclaim is mounted read only. Ephemeral volumes always report 0 since the ephemeral volume source does not support a read-only flag. For `ephemeral="true"`, the `persistentvolumeclaim` label is the generated claim name `-`; non-ephemeral volumes retain their declared claim name. | bool | `pod`=<pod-name>
`namespace`=<pod-namespace>
`volume`=<volume-name>
`persistentvolumeclaim`=<persistentvolumeclaim-claimname>
`ephemeral`=<true\|false>
`uid`=<pod-uid> | STABLE | - | | kube_pod_status_reason | Gauge | The pod status reasons. Emitted only for the reason that is actually set; a missing series does not mean the reason is false. An unrecognized `pod.status.reason` is reported as `Other`; conditions and container-terminated reasons outside this list are not reported. | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`reason`=<Evicted\|NodeAffinity\|NodeLost\|PreemptionByScheduler\|SchedulingGated\|Shutdown\|TerminationByKubelet\|UnexpectedAdmissionError\|Other>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_status_disruption_reason | Gauge | The pod disruption condition reason. Emitted only when a `DisruptionTarget` condition is present; a missing series does not mean the reason is false. An unrecognized reason is reported as `Other`. `kube_pod_status_reason` remains available separately and existing queries against it should not be replaced with this metric. | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`reason`=<PreemptionByScheduler\|DeletionByTaintManager\|EvictionByEvictionAPI\|DeletionByPodGC\|TerminationByKubelet\|Other>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_status_scheduled_time | Gauge | Unix timestamp when pod moved into scheduled status | seconds | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_status_unschedulable | Gauge | Describes the unschedulable status for the pod | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | STABLE | - | | kube_pod_status_unscheduled_time | Gauge | Unix timestamp when pod moved into unscheduled status | seconds | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid> | EXPERIMENTAL | - | | kube_pod_tolerations | Gauge | Information about the pod tolerations | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid>
`key`=<toleration-key>
`operator`=<toleration-operator>
`value`=<toleration-value>
`effect`=<toleration-effect> `toleration_seconds`=<toleration-seconds> | EXPERIMENTAL | - | | kube_pod_service_account | Gauge | The service account for a pod | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid>
`service_account`=<service_account> | EXPERIMENTAL | - | | kube_pod_scheduler | Gauge | The scheduler for a pod | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid>
`name`=<scheduler-name> | EXPERIMENTAL | - | | kube_pod_resourceclaim_info | Gauge | Information about a DRA ResourceClaim referenced by a pod, one series per pod.spec.resourceClaims entry | | `pod`=<pod-name>
`namespace`=<pod-namespace>
`uid`=<pod-uid>
`claim_name`=<pod-local-claim-name>
`resourceclaim_name`=<resolved-resourceclaim-name>
`resourceclaim_template_name`=<resourceclaimtemplate-name> | EXPERIMENTAL | - | ## Useful metrics queries ### How to retrieve non-standard Pod state It is not straightforward to get the Pod states for certain cases like "Terminating" and "Unknown" since it is not stored behind a field in the `Pod.Status`. So to mimic the [logic](https://github.com/kubernetes/kubernetes/blob/v1.17.3/pkg/printers/internalversion/printers.go#L624) used by the `kubectl` command line, you will need to compose multiple metrics. For example: * To get the list of pods that are in the `Unknown` state, you can run the following PromQL query: `sum(kube_pod_status_phase{phase="Unknown"}) by (namespace, pod) or (count(kube_pod_deletion_timestamp) by (namespace, pod) * sum(kube_pod_status_reason{reason="NodeLost"}) by(namespace, pod))` * For Pods in `Terminating` state: `count(kube_pod_deletion_timestamp) by (namespace, pod) unless count(kube_pod_status_reason{reason="NodeLost"}) by (namespace, pod)` Here is an example of a Prometheus rule that can be used to alert on a Pod that has been in the `Terminating` state for more than `5m`. ```yaml groups: - name: Pod state rules: - alert: PodsBlockedInTerminatingState expr: count(kube_pod_deletion_timestamp) by (namespace, pod) unless count(kube_pod_status_reason{reason="NodeLost"}) by (namespace, pod) for: 5m labels: severity: page annotations: summary: Pod {{$labels.namespace}}/{{$labels.pod}} blocked in Terminating state. ``` --- ### Metrics/Workload/Replicaset Metrics # ReplicaSet metrics | Metric name | Metric type | Description | Labels/tags | Status | | --------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_replicaset_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace>
`annotation_REPLICASET_ANNOTATION`=<REPLICASET_ANNOTATION> | EXPERIMENTAL | | kube_replicaset_status_replicas | Gauge | | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace> | STABLE | | kube_replicaset_status_fully_labeled_replicas | Gauge | | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace> | STABLE | | kube_replicaset_status_ready_replicas | Gauge | | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace> | STABLE | | kube_replicaset_status_terminating_replicas | Gauge | The number of terminating replicas per ReplicaSet. | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace> | ALPHA | | kube_replicaset_status_observed_generation | Gauge | | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace> | STABLE | | kube_replicaset_spec_replicas | Gauge | | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace> | STABLE | | kube_replicaset_metadata_generation | Gauge | | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace> | STABLE | | kube_replicaset_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace>
`label_REPLICASET_LABEL`=<REPLICASET_LABEL> | STABLE | | kube_replicaset_created | Gauge | | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace> | STABLE | | kube_replicaset_owner | Gauge | | `replicaset`=<replicaset-name>
`namespace`=<replicaset-namespace>
`owner_kind`=<owner kind>
`owner_name`=<owner name>
`owner_is_controller`=<whether owner is controller> | STABLE | --- ### Metrics/Workload/Replicationcontroller Metrics # ReplicationController metrics | Metric name | Metric type | Description | Labels/tags | Status | | -------------------------------------------------------- | ----------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | | kube_replicationcontroller_status_replicas | Gauge | | `replicationcontroller`=<replicationcontroller-name>
`namespace`=<replicationcontroller-namespace> | STABLE | | kube_replicationcontroller_status_fully_labeled_replicas | Gauge | | `replicationcontroller`=<replicationcontroller-name>
`namespace`=<replicationcontroller-namespace> | STABLE | | kube_replicationcontroller_status_ready_replicas | Gauge | | `replicationcontroller`=<replicationcontroller-name>
`namespace`=<replicationcontroller-namespace> | STABLE | | kube_replicationcontroller_status_available_replicas | Gauge | | `replicationcontroller`=<replicationcontroller-name>
`namespace`=<replicationcontroller-namespace> | STABLE | | kube_replicationcontroller_status_observed_generation | Gauge | | `replicationcontroller`=<replicationcontroller-name>
`namespace`=<replicationcontroller-namespace> | STABLE | | kube_replicationcontroller_spec_replicas | Gauge | | `replicationcontroller`=<replicationcontroller-name>
`namespace`=<replicationcontroller-namespace> | STABLE | | kube_replicationcontroller_metadata_generation | Gauge | | `replicationcontroller`=<replicationcontroller-name>
`namespace`=<replicationcontroller-namespace> | STABLE | | kube_replicationcontroller_created | Gauge | | `replicationcontroller`=<replicationcontroller-name>
`namespace`=<replicationcontroller-namespace> | STABLE | | kube_replicationcontroller_owner | Gauge | | `replicationcontroller`=<replicationcontroller-name>
`namespace`=<replicationcontroller-namespace>
`owner_kind`=<owner kind>
`owner_name`=<owner name>
`owner_is_controller`=<whether owner is controller> | EXPERIMENTAL | --- ### Metrics/Workload/Statefulset Metrics # Stateful Set Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ------------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_statefulset_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace>
`annotation_STATEFULSET_ANNOTATION`=<STATEFULSET_ANNOTATION> | STABLE | | kube_statefulset_status_replicas | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | | kube_statefulset_status_replicas_current | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | | kube_statefulset_status_replicas_ready | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | | kube_statefulset_status_replicas_available | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | | kube_statefulset_status_replicas_updated | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | | kube_statefulset_status_observed_generation | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | | kube_statefulset_replicas | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | | kube_statefulset_ordinals_start | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | | kube_statefulset_metadata_generation | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | | kube_statefulset_persistentvolumeclaim_retention_policy | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace>
`when_deleted`=<statefulset-when-deleted-pvc-policy>
`when_scaled`=<statefulset-when-scaled-pvc-policy> | STABLE | | kube_statefulset_created | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | | kube_statefulset_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace>
`label_STATEFULSET_LABEL`=<STATEFULSET_LABEL> | STABLE | | kube_statefulset_status_current_revision | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace>
`revision`=<statefulset-current-revision> | STABLE | | kube_statefulset_status_update_revision | Gauge | | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace>
`revision`=<statefulset-update-revision> | STABLE | | kube_statefulset_deletion_timestamp | Gauge | Unix deletion timestamp | `statefulset`=<statefulset-name>
`namespace`=<statefulset-namespace> | STABLE | --- ### Metrics/Storage/Configmap Metrics # ConfigMap Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ---------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_configmap_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `configmap`=<configmap-name>
`namespace`=<configmap-namespace>
`annotation_CONFIGMAP_ANNOTATION`=<CONFIGMAP_ANNOTATION> | EXPERIMENTAL | | kube_configmap_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `configmap`=<configmap-name>
`namespace`=<configmap-namespace>
`label_CONFIGMAP_LABEL`=<CONFIGMAP_LABEL> | STABLE | | kube_configmap_info | Gauge | | `configmap`=<configmap-name>
`namespace`=<configmap-namespace> | STABLE | | kube_configmap_created | Gauge | | `configmap`=<configmap-name>
`namespace`=<configmap-namespace> | STABLE | | kube_configmap_metadata_resource_version | Gauge | | `configmap`=<configmap-name>
`namespace`=<configmap-namespace> | EXPERIMENTAL | --- ### Metrics/Storage/Persistentvolume Metrics # PersistentVolume Metrics | Metric name | Metric type | Description | Unit (where applicable) | Labels/tags | Status | |------------------------------------------|-------------|---------------------------------------------------------------------------------------------------------------------------|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------| | kube_persistentvolume_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | | `persistentvolume`=<persistentvolume-name>
`annotation_PERSISTENTVOLUME_ANNOTATION`=<PERSISTENTVOLUME_ANNOTATION> | EXPERIMENTAL | | kube_persistentvolume_capacity_bytes | Gauge | | | `persistentvolume`=<pv-name> | STABLE | | kube_persistentvolume_status_phase | Gauge | | | `persistentvolume`=<pv-name>
`phase`=<Bound\|Failed\|Pending\|Available\|Released> | STABLE | | kube_persistentvolume_claim_ref | Gauge | | | `persistentvolume`=<pv-name>
`claim_namespace`=``;
`name`=``; | STABLE | | kube_persistentvolume_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | | `persistentvolume`=<persistentvolume-name>
`label_PERSISTENTVOLUME_LABEL`=<PERSISTENTVOLUME_LABEL> | STABLE | | kube_persistentvolume_info | Gauge | Information about Persistent Volumes | | `persistentvolume`=<pv-name>
`storageclass`=<storageclass-name>
`gce_persistent_disk_name`=<pd-name>
`host_path`=<path-of-a-host-volume>
`host_path_type`=<host-mount-type>
`ebs_volume_id`=<ebs-volume-id>
`azure_disk_name`=<azure-disk-name>
`fc_wwids`=<fc-wwids-comma-separated>
`fc_lun`=<fc-lun>
`fc_target_wwns`=<fc-target-wwns-comma-separated>
`iscsi_target_portal`=<iscsi-target-portal>
`iscsi_iqn`=<iscsi-iqn>
`iscsi_lun`=<iscsi-lun>
`iscsi_initiator_name`=<iscsi-initiator-name>
`local_path`=<path-of-a-local-volume>
`local_fs`=<local-volume-fs-type>
`nfs_server`=<nfs-server>
`nfs_path`=<nfs-path>
`csi_driver`=<csi-driver>
`csi_volume_handle`=<csi-volume-handle>
`reclaim_policy`=<reclaim-policy> | STABLE | | kube_persistentvolume_created | Gauge | Unix creation timestamp | seconds | `persistentvolume`=<persistentvolume-name>
| EXPERIMENTAL | | kube_persistentvolume_deletion_timestamp | Gauge | Unix deletion timestamp | seconds | `persistentvolume`=<persistentvolume-name>
| EXPERIMENTAL | | kube_persistentvolume_csi_attributes | Gauge | CSI attributes of the Persistent Volume, disabled by default, manage with [--metric-opt-in-list](../../developer/cli-arguments.md)) | | `persistentvolume`=<persistentvolume-name>
`csi_mounter`=<csi-mounter>
`csi_map_options`=<csi-map-options> | EXPERIMENTAL | | kube_persistentvolume_volume_mode | Gauge | Volume Mode information for the PersistentVolume. | | `persistentvolume`=<persistentvolume-name>
`volumemode`=<volumemode> | EXPERIMENTAL | | kube_persistentvolume_access_mode | Gauge | Access modes of the PersistentVolume. | | `persistentvolume`=<persistentvolume-name>
`access_mode`=<ReadWriteOnce \| ReadOnlyMany \| ReadWriteMany \| ReadWriteOncePod> | EXPERIMENTAL | ## Useful metrics queries ### How to retrieve non-standard PV state It is not straightforward to get the PV states for certain cases like "Terminating" since it is not stored behind a field in the `PersistentVolume.Status`. So to mimic the [logic](https://github.com/kubernetes/kubernetes/blob/v1.27.2/pkg/printers/internalversion/printers.go#L1838) used by the `kubectl` command line, you will need to compose multiple metrics. Here is an example of a Prometheus rule that can be used to alert on a PV that has been in the `Terminating` state for more than `5m`. ```yaml groups: - name: PV state rules: - alert: PVBlockedInTerminatingState expr: kube_persistentvolume_deletion_timestamp * on(persistentvolume) group_left() (kube_persistentvolume_status_phase{phase="Bound"} == 1) > 0 for: 5m labels: severity: warning annotations: summary: PV {{$labels.persistentvolume}} blocked in Terminating state. ``` --- ### Metrics/Storage/Persistentvolumeclaim Metrics # PersistentVolumeClaim Metrics | Metric name | Metric type | Description | Unit (where applicable) | Labels/tags | Status | | ---------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ | | kube_persistentvolumeclaim_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | | `persistentvolumeclaim`=<persistentvolumeclaim-name>
`namespace`=<persistentvolumeclaim-namespace>
`annotation_PERSISTENTVOLUMECLAIM_ANNOTATION`=<PERSISTENTVOLUMECLAIM_ANNOATION> | EXPERIMENTAL | | kube_persistentvolumeclaim_access_mode | Gauge | | | `access_mode`=<persistentvolumeclaim-access-mode>
`namespace`=<persistentvolumeclaim-namespace>
`persistentvolumeclaim`=<persistentvolumeclaim-name> | STABLE | | kube_persistentvolumeclaim_info | Gauge | | | `namespace`=<persistentvolumeclaim-namespace>
`persistentvolumeclaim`=<persistentvolumeclaim-name>
`storageclass`=<persistentvolumeclaim-storageclassname>
`volumename`=<volumename>
`volumemode`=<volumemode> | STABLE | | kube_persistentvolumeclaim_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | | `persistentvolumeclaim`=<persistentvolumeclaim-name>
`namespace`=<persistentvolumeclaim-namespace>
`label_PERSISTENTVOLUMECLAIM_LABEL`=<PERSISTENTVOLUMECLAIM_LABEL> | STABLE | | kube_persistentvolumeclaim_resource_requests_storage_bytes | Gauge | | | `namespace`=<persistentvolumeclaim-namespace>
`persistentvolumeclaim`=<persistentvolumeclaim-name> | STABLE | | kube_persistentvolumeclaim_status_condition | Gauge | | | `namespace` =<persistentvolumeclaim-namespace>
`persistentvolumeclaim`=<persistentvolumeclaim-name>
`type`=<persistentvolumeclaim-condition-type>
`status`=<true\false\unknown> | EXPERIMENTAL | | kube_persistentvolumeclaim_status_phase | Gauge | | | `namespace`=<persistentvolumeclaim-namespace>
`persistentvolumeclaim`=<persistentvolumeclaim-name>
`phase`=<Pending\Bound\Lost> | STABLE | | kube_persistentvolumeclaim_created | Gauge | Unix creation timestamp | seconds | `namespace`=<persistentvolumeclaim-namespace>
`persistentvolumeclaim`=<persistentvolumeclaim-name> | EXPERIMENTAL | | kube_persistentvolumeclaim_deletion_timestamp | Gauge | Unix deletion timestamp | seconds | `namespace`=<persistentvolumeclaim-namespace>
`persistentvolumeclaim`=<persistentvolumeclaim-name> | EXPERIMENTAL | Note: * An empty string will be used if PVC has no storage class. ## Useful metrics queries ### How to retrieve non-standard PVC state It is not straightforward to get the PVC states for certain cases like "Terminating" since it is not stored behind a field in the `PersistentVolumeClaim.Status`. So to mimic the [logic](https://github.com/kubernetes/kubernetes/blob/v1.27.2/pkg/printers/internalversion/printers.go#L1883) used by the `kubectl` command line, you will need to compose multiple metrics. Here is an example of a Prometheus rule that can be used to alert on a PVC that has been in the `Terminating` state for more than `5m`. ```yaml groups: - name: PVC state rules: - alert: PVCBlockedInTerminatingState expr: kube_persistentvolumeclaim_deletion_timestamp * on(namespace, persistentvolumeclaim) group_left() (kube_persistentvolumeclaim_status_phase{phase="Bound"} == 1) > 0 for: 5m labels: severity: warning annotations: summary: PVC {{$labels.namespace}}/{{$labels.persistentvolumeclaim}} blocked in Terminating state. ``` --- ### Metrics/Storage/Secret Metrics # Secret Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_secret_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `secret`=<secret-name>
`namespace`=<secret-namespace>
`annotations_SECRET_ANNOTATION`=<SECRET_ANNOTATION> | EXPERIMENTAL | | kube_secret_info | Gauge | | `secret`=<secret-name>
`namespace`=<secret-namespace> | STABLE | | kube_secret_type | Gauge | | `secret`=<secret-name>
`namespace`=<secret-namespace>
`type`=<secret-type> | STABLE | | kube_secret_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `secret`=<secret-name>
`namespace`=<secret-namespace>
`label_SECRET_LABEL`=<SECRET_LABEL> | STABLE | | kube_secret_created | Gauge | | `secret`=<secret-name>
`namespace`=<secret-namespace> | STABLE | | kube_secret_metadata_resource_version | Gauge | | `secret`=<secret-name>
`namespace`=<secret-namespace> | EXPERIMENTAL | | kube_secret_owner | Gauge | | `secret`=<secret-name>
`namespace`=<secret-namespace>
`owner_kind`=<owner kind>
`owner_name`=<owner name>
`owner_is_controller`=<whether owner is controller> | EXPERIMENTAL | --- ### Metrics/Storage/Storageclass Metrics # StorageClass Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ----------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_storageclass_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `storageclass`=<storageclass-name>
`annotation_STORAGECLASS_ANNOTATION`=<STORAGECLASS_ANNOTATION> | EXPERIMENTAL | | kube_storageclass_info | Gauge | | `storageclass`=<storageclass-name>
`provisioner`=<storageclass-provisioner>
`reclaim_policy`=<storageclass-reclaimPolicy>
`volume_binding_mode`=<storageclass-volumeBindingMode> | STABLE | | kube_storageclass_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `storageclass`=<storageclass-name>
`label_STORAGECLASS_LABEL`=<STORAGECLASS_LABEL> | STABLE | | kube_storageclass_created | Gauge | | `storageclass`=<storageclass-name> | STABLE | --- ### Metrics/Storage/Volumeattachment Metrics # VolumeAttachment Metrics | Metric name | Metric type | Description | Labels/tags | Status | | -------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_volumeattachment_info | Gauge | | `volumeattachment`=<volumeattachment-name>
`attacher`=<attacher-name>
`node`=<node-name> | EXPERIMENTAL | | kube_volumeattachment_created | Gauge | | `volumeattachment`=<volumeattachment-name> | EXPERIMENTAL | | kube_volumeattachment_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `volumeattachment`=<volumeattachment-name>
`label_VOLUMEATTACHMENT_LABEL`=<VOLUMEATTACHMENT_LABEL> | EXPERIMENTAL | | kube_volumeattachment_spec_source_persistentvolume | Gauge | | `volumeattachment`=<volumeattachment-name>
`volumename`=<persistentvolume-name> | EXPERIMENTAL | | kube_volumeattachment_status_attached | Gauge | | `volumeattachment`=<volumeattachment-name> | EXPERIMENTAL | | kube_volumeattachment_status_attachment_metadata | Gauge | | `volumeattachment`=<volumeattachment-name>
`metadata_METADATA_KEY`=<METADATA_VALUE> | EXPERIMENTAL | --- ### Metrics/Service/Endpoint Metrics # Endpoint Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_endpoint_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `endpoint`=<endpoint-name>
`namespace`=<endpoint-namespace>
`annotation_ENDPOINT_ANNOTATION`=<ENDPOINT_ANNOTATION> | EXPERIMENTAL | | kube_endpoint_info | Gauge | | `endpoint`=<endpoint-name>
`namespace`=<endpoint-namespace> | STABLE | | kube_endpoint_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `endpoint`=<endpoint-name>
`namespace`=<endpoint-namespace>
`label_ENDPOINT_LABEL`=<ENDPOINT_LABEL> | STABLE | | kube_endpoint_created | Gauge | | `endpoint`=<endpoint-name>
`namespace`=<endpoint-namespace> | STABLE | | kube_endpoint_ports | Gauge | | `endpoint`=<endpoint-name>
`namespace`=<endpoint-namespace>
`port_name`=<endpoint-port-name>
`port_protocol`=<endpoint-port-protocol>
`port_number`=<endpoint-port-number> | STABLE (Deprecated from 2.14.0) | | kube_endpoint_address | Gauge | | `endpoint`=<endpoint-name>
`namespace`=<endpoint-namespace>
`ip`=<endpoint-ip>
`port_name`=<endpoint-port-name>
`port_protocol`=<endpoint-port-protocol>
`port_number`=<endpoint-port-number>`ready`=<true if available, false if unavailalbe> | STABLE | --- ### Metrics/Service/Endpointslice Metrics # Endpoint Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_endpointslice_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `endpointslice`=<endpointslice-name>
`namespace`=<endpointslice-namespace>
`annotation_ENDPOINTSLICE_ANNOTATION`=<ENDPOINTSLICE_ANNOTATION> | EXPERIMENTAL | | kube_endpointslice_info | Gauge | | `endpointslice`=<endpointslice-name>
`namespace`=<endpointslice-namespace>
`addresstype`=<endpointslice-addresstype> | EXPERIMENTAL | | kube_endpointslice_ports | Gauge | | `endpointslice`=<endpointslice-name>
`namespace`=<endpointslice-namespace>
`port_name`=<endpointslice-port-name>
`port_protocol`=<endpointslice-port-protocol>
`port_number`=<endpointslice-port-number> | EXPERIMENTAL | | kube_endpointslice_endpoints | Gauge | | `endpointslice`=<endpointslice-name>
`namespace`=<endpointslice-namespace>
`address`=<endpointslice-address>
`ready`=<endpointslice-ready>
`serving`=<endpointslice-serving>
`terminating`=<endpointslice-terminating>
`hostname`=<endpointslice-hostname>
`targetref_kind`=<endpointslice-targetref-kind>
`targetref_name`=<endpointslice-targetref-name>
`targetref_namespace`=<endpointslice-targetref-namespace>
`nodename`=<endpointslice-nodename>
`endpoint_zone`=<endpointslice-zone> | EXPERIMENTAL | | kube_endpointslice_endpoints_hints | Gauge | Each line is a hint applied to an endpoint-slice | `endpointslice`=<endpointslice-name>
`namespace`=<endpointslice-namespace>
`address`=<endpointslice-address[0]>
`for_zone`=<endpointslice-hint> | EXPERIMENTAL | | kube_endpointslice_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `endpointslice`=<endpointslice-name>
`namespace`=<endpointslice-namespace>
`label_ENDPOINTSLICE_LABEL`=<ENDPOINTSLICE_LABEL> | EXPERIMENTAL | | kube_endpointslice_created | Gauge | | `endpointslice`=<endpointslice-name>
`namespace`=<endpointslice-namespace> | EXPERIMENTAL | --- ### Metrics/Service/Ingress Metrics # Ingress Metrics | Metric name | Metric type | Description | Labels/tags | Status | | -------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ------------ | | kube_ingress_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `ingress`=<ingress-name>
`namespace`=<ingress-namespace>
`annotation_INGRESS_ANNOTATION`=<ANNOTATION_LABEL> | EXPERIMENTAL | | kube_ingress_info | Gauge | | `ingress`=<ingress-name>
`namespace`=<ingress-namespace>
`ingressclass`=<ingress-class> or `_default` if not set | STABLE | | kube_ingress_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `ingress`=<ingress-name>
`namespace`=<ingress-namespace>
`label_INGRESS_LABEL`=<INGRESS_LABEL> | STABLE | | kube_ingress_created | Gauge | | `ingress`=<ingress-name>
`namespace`=<ingress-namespace> | STABLE | | kube_ingress_metadata_resource_version | Gauge | | `ingress`=<ingress-name>
`namespace`=<ingress-namespace> | EXPERIMENTAL | | kube_ingress_path | Gauge | | `ingress`=<ingress-name>
`namespace`=<ingress-namespace>
`host`=<ingress-host>
`path`=<ingress-path>
`path_type`=<ingress-path type>
If path served by Service Backend
`service_name`=<service name for the path>
`service_port`=<service port for the path>
If path served by Resource Backend
`resource_api_group`=<resource backend api group>
`resource_kind`=<resource backend kind>
`resource_name`=<resource backend name> | STABLE | | kube_ingress_tls | Gauge | | `ingress`=<ingress-name>
`namespace`=<ingress-namespace>
`tls_host`=<tls hostname>
`secret`=<tls secret name> | STABLE | --- ### Metrics/Service/Ingressclass Metrics # IngressClass Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ----------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------ | | kube_ingressclass_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `ingressclass`=<ingressclass-name>
`annotation_INGRESSCLASS_ANNOTATION`=<INGRESSCLASS_ANNOTATION> | EXPERIMENTAL | | kube_ingressclass_info | Gauge | | `ingressclass`=<ingressclass-name>
`controller`=<ingress-controller-name>
| EXPERIMENTAL | | kube_ingressclass_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `ingressclass`=<ingressclass-name>
`label_INGRESSCLASS_LABEL`=<INGRESSCLASS_LABEL> | EXPERIMENTAL | | kube_ingressclass_created | Gauge | | `ingressclass`=<ingressclass-name> | EXPERIMENTAL | --- ### Metrics/Service/Service Metrics # Service Metrics | Metric name | Metric type | Description | Unit (where applicable) | Labels/tags | Status | | ----------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_service_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | | `service`=<service-name>
`namespace`=<service-namespace>
`uid`=<service-uid>
`annotation_SERVICE_ANNOTATION`=<SERVICE_ANNOTATION> | EXPERIMENTAL | | kube_service_info | Gauge | Information about service | | `service`=<service-name>
`namespace`=<service-namespace>
`uid`=<service-uid>
`cluster_ip`=<service cluster ip>
`external_name`=<service external name>
`external_traffic_policy`=<service external traffic policy>
`load_balancer_ip`=<service load balancer ip> | STABLE | | kube_service_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | | `service`=<service-name>
`namespace`=<service-namespace>
`uid`=<service-uid>
`label_SERVICE_LABEL`=<SERVICE_LABEL> | STABLE | | kube_service_created | Gauge | Unix creation timestamp | seconds | `service`=<service-name>
`namespace`=<service-namespace>
`uid`=<service-uid> | STABLE | | kube_service_spec_type | Gauge | Type about service | | `service`=<service-name>
`namespace`=<service-namespace>
`uid`=<service-uid>
`type`=<ClusterIP\|NodePort\|LoadBalancer\|ExternalName> | STABLE | | kube_service_spec_external_ip | Gauge | Service external ips. One series for each ip | | `service`=<service-name>
`namespace`=<service-namespace>
`uid`=<service-uid>
`external_ip`=<external-ip> | STABLE | | kube_service_status_load_balancer_ingress | Gauge | Service load balancer ingress status | | `service`=<service-name>
`namespace`=<service-namespace>
`uid`=<service-uid>
`ip`=<load-balancer-ingress-ip>
`hostname`=<load-balancer-ingress-hostname> | STABLE | | kube_service_deletion_timestamp | Gauge | Unix deletion timestamp | seconds | `service`=<service-name>
`namespace`=<service-namespace>
`uid`=<service-uid> | EXPERIMENTAL | --- ### Metrics/Policy/Limitrange Metrics # LimitRange Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ----------------------- | ----------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ | | kube_limitrange | Gauge | | `limitrange`=<limitrange-name>
`namespace`=<namespace>
`resource`=<ResourceName>
`type`=<Pod\|Container\|PersistentVolumeClaim>
`constraint`=<constraint> | STABLE | | kube_limitrange_created | Gauge | | `limitrange`=<limitrange-name>
`namespace`=<namespace> | STABLE | --- ### Metrics/Policy/Networkpolicy Metrics # Network Policy Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ------------ | | kube_networkpolicy_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `namespace`=<namespace name> `networkpolicy`=<networkpolicy name> | EXPERIMENTAL | | kube_networkpolicy_created | Gauge | | `namespace`=<namespace name> `networkpolicy`=<networkpolicy name> | EXPERIMENTAL | | kube_networkpolicy_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `namespace`=<namespace name> `networkpolicy`=<networkpolicy name> | EXPERIMENTAL | | kube_networkpolicy_spec_egress_rules | Gauge | | `namespace`=<namespace name> `networkpolicy`=<networkpolicy name> | EXPERIMENTAL | | kube_networkpolicy_spec_ingress_rules | Gauge | | `namespace`=<namespace name> `networkpolicy`=<networkpolicy name> | EXPERIMENTAL | --- ### Metrics/Policy/Poddisruptionbudget Metrics # PodDisruptionBudget Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ------------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_poddisruptionbudget_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `poddisruptionbudget`=<poddisruptionbudget-name>
`namespace`=<poddisruptionbudget-namespace>
`annotation_PODDISRUPTIONBUDGET_ANNOTATION`=<PODDISRUPTIONBUDGET_ANNOATION> | EXPERIMENTAL | | kube_poddisruptionbudget_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `poddisruptionbudget`=<poddisruptionbudget-name>
`namespace`=<poddisruptionbudget-namespace>
`label_PODDISRUPTIONBUDGET_LABEL`=<PODDISRUPTIONBUDGET_ANNOATION> | EXPERIMENTAL | | kube_poddisruptionbudget_created | Gauge | | `poddisruptionbudget`=<pdb-name>
`namespace`=<pdb-namespace> | STABLE | | kube_poddisruptionbudget_status_current_healthy | Gauge | | `poddisruptionbudget`=<pdb-name>
`namespace`=<pdb-namespace> | STABLE | | kube_poddisruptionbudget_status_desired_healthy | Gauge | | `poddisruptionbudget`=<pdb-name>
`namespace`=<pdb-namespace> | STABLE | | kube_poddisruptionbudget_status_pod_disruptions_allowed | Gauge | | `poddisruptionbudget`=<pdb-name>
`namespace`=<pdb-namespace> | STABLE | | kube_poddisruptionbudget_status_expected_pods | Gauge | | `poddisruptionbudget`=<pdb-name>
`namespace`=<pdb-namespace> | STABLE | | kube_poddisruptionbudget_status_observed_generation | Gauge | | `poddisruptionbudget`=<pdb-name>
`namespace`=<pdb-namespace> | STABLE | | kube_poddisruptionbudget_deletion_timestamp | Gauge | | `poddisruptionbudget`=<pdb-name>
`namespace`=<pdb-namespace> | EXPERIMENTAL | --- ### Metrics/Policy/Resourcequota Metrics # ResourceQuota Metrics | Metric name | Metric type | Description | Labels/tags | Status | | ------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | | kube_resourcequota | Gauge | | `resourcequota`=<quota-name>
`namespace`=<namespace>
`resource`=<ResourceName>
`type`=<quota-type> | STABLE | | kube_resourcequota_created | Gauge | | `resourcequota`=<quota-name>
`namespace`=<namespace> | STABLE | | kube_resourcequota_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via [--metric-annotations-allowlist](../../developer/cli-arguments.md) | `resourcequota`=<quota-name>
`namespace`=<namespace>
`annotation_RESOURCE_QUOTA_ANNOTATION`=<RESOURCE_QUOTA_ANNOTATION> | EXPERIMENTAL | | kube_resourcequota_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via [--metric-labels-allowlist](../../developer/cli-arguments.md) | `resourcequota`=<quota-name>
`namespace`=<namespace>
`label_RESOURCE_QUOTA_LABEL`=<RESOURCE_QUOTA_LABEL> | EXPERIMENTAL | --- ### Metrics/Extend/Customresourcestate Metrics # Custom Resource State Metrics > [!NOTE] > `custom-resource-state` is feature-frozen in favor of [resource-state-metrics](https://github.com/kubernetes-sigs/resource-state-metrics). Once `resource-state-metrics` is stable, `custom-resource-state` will be deprecated. This section describes how to add metrics based on the state of a custom resource without writing a custom resource registry and running your own build of KSM. ## Configuration A YAML configuration file described below is required to define your custom resources and the fields to turn into metrics. Two flags can be used: * `--custom-resource-state-config "inline yaml (see example)"` or * `--custom-resource-state-config-file /path/to/config.yaml` When using a `--config` file, the equivalent YAML key is `custom_resource_state_config_file`. The pre-v2.17 key `custom_resource_config_file` is still honored as a deprecated alias and will be removed in a future release; please migrate to `custom_resource_state_config_file`. If both flags are provided, the inline configuration will take precedence. When multiple entries for the same resource exist, kube-state-metrics will exit with an error. This includes configuration which refers to a different API version. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: kube-state-metrics namespace: kube-system spec: template: spec: containers: - name: kube-state-metrics args: - --custom-resource-state-config # in YAML files, | allows a multi-line string to be passed as a flag value # see https://yaml-multiline.info - | kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: group: myteam.io version: "v1" kind: Foo metrics: - name: active_count help: "Count of active Foo" each: type: Gauge ... ``` It's also possible to configure kube-state-metrics to run in a `custom-resource-mode` only. In addition to specifying one of `--custom-resource-state-config*` flags, you could set `--custom-resource-state-only` to `true`. With this configuration only the known custom resources configured in `--custom-resource-state-config*` will be taken into account by kube-state-metrics. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: kube-state-metrics namespace: kube-system spec: template: spec: containers: - name: kube-state-metrics args: - --custom-resource-state-config # in YAML files, | allows a multi-line string to be passed as a flag value # see https://yaml-multiline.info - | kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: group: myteam.io version: "v1" kind: Foo metrics: - name: active_count help: "Count of active Foo" each: type: Gauge ... - --custom-resource-state-only=true ``` NOTE: The `customresource_group`, `customresource_version`, and `customresource_kind` common labels are reserved, and will be overwritten by the values from the `groupVersionKind` field. ### RBAC-enabled Clusters Please be aware that kube-state-metrics needs list and watch permissions granted to `customresourcedefinitions.apiextensions.k8s.io` as well as to the resources you want to gather metrics from. ### Examples The examples in this section will use the following custom resource: ```yaml kind: Foo apiVersion: myteam.io/vl metadata: annotations: bar: baz qux: quxx labels: foo: bar name: foo spec: version: v1.2.3 order: - id: 1 value: true - id: 3 value: false replicas: 1 refs: - my_other_foo - foo_2 - foo_with_extensions status: phase: Pending active: type-a: 1 type-b: 3 conditions: - name: a value: 45 - name: b value: 66 sub: type-a: active: 1 ready: 2 type-b: active: 3 ready: 4 uptime: 43.21 ``` #### Single Values The config: ```yaml kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: group: myteam.io kind: "Foo" version: "v1" metrics: - name: "uptime" help: "Foo uptime" each: type: Gauge gauge: path: [status, uptime] ``` Produces the metric: ```prometheus kube_customresource_uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21 ``` #### Multiple Metrics/Kitchen Sink ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Produces the following metrics: ```prometheus kube_customresource_ready_count{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", active="1",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-a", lorem_bar="baz",lorem_qux="quxx",} 2 kube_customresource_ready_count{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", active="3",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-b", lorem_bar="baz",lorem_qux="quxx",} 4 ``` #### Non-map Arrays ```yaml kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: group: myteam.io kind: "Foo" version: "v1" labelsFromPath: name: [metadata, name] metrics: - name: "ref_info" help: "Reference to other Foo" each: type: Info info: # targeting an array will produce a metric for each element # labelsFromPath and value are relative to this path path: [spec, refs] # if path targets a list of values (e.g. strings or numbers, not objects or maps), individual values can # referenced by a label using this syntax labelsFromPath: ref: [] ``` Produces the following metrics: ```prometheus kube_customresource_ref_info{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", name="foo",ref="my_other_foo"} 1 kube_customresource_ref_info{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", name="foo",ref="foo_2"} 1 kube_customresource_ref_info{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", name="foo",ref="foo_with_extensions"} 1 ``` #### Same Metrics with Different Labels ```yaml recommendation: containerRecommendations: - containerName: consumer lowerBound: cpu: 100m memory: 262144k ``` For example in VPA we have above attributes and we want to have a same metrics for both CPU and Memory, you can use below config: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Produces the following metrics: ```prometheus # HELP kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound Minimum memory resources the container can use before the VerticalPodAutoscaler updater evicts it. # TYPE kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound gauge kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound{container="consumer",customresource_group="autoscaling.k8s.io",customresource_kind="VerticalPodAutoscaler",customresource_version="v1",namespace="namespace-example",resource="memory",target_api_version="apps/v1",target_kind="Deployment",target_name="target-name-example",unit="byte",verticalpodautoscaler="vpa-example"} 123456 # HELP kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound Minimum cpu resources the container can use before the VerticalPodAutoscaler updater evicts it. # TYPE kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound gauge kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound{container="consumer",customresource_group="autoscaling.k8s.io",customresource_kind="VerticalPodAutoscaler",customresource_version="v1",namespace="namespace-example",resource="cpu",target_api_version="apps/v1",target_kind="Deployment",target_name="target-name-example",unit="core",verticalpodautoscaler="vpa-example"} 0.1 ``` #### VerticalPodAutoscaler In v2.9.0 the `vericalpodautoscalers` resource was removed from the list of default resources. In order to generate metrics for `verticalpodautoscalers`, you can use the following Custom Resource State config: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` The above configuration was tested on [this](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/examples/hamster.yaml) VPA configuration, with an added annotation (`foo: 123`). #### All VerticalPodAutoscaler Metrics As an addition for the above configuration, here's the complete `CustomResourceStateMetrics` spec to re-enable all of the VPA metrics which are removed from the list of the default resources:
VPA CustomResourceStateMetrics ``` /* Detailed source-code truncated for AI context efficiency. */ ```
### Metric types The configuration supports three kind of metrics from the [OpenMetrics specification](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md). The metric type is specified by the `type` field and its specific configuration at the types specific struct. #### Gauge > Gauges are current measurements, such as bytes of memory currently used or the number of items in a queue. For gauges the absolute value is what is of interest to a user. [[0]](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#gauge) Example: ```yaml kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: group: myteam.io kind: "Foo" version: "v1" metrics: - name: "uptime" help: "Foo uptime" each: type: Gauge gauge: path: [status, uptime] ``` Produces the metric: ```prometheus kube_customresource_uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21 ``` ##### Type conversion and special handling Gauges produce values of type float64 but custom resources can be of all kinds of types. Kube-state-metrics performs implicit type conversions for a lot of type. Supported types are: * (u)int32/64, int, float32 and byte are cast to float64 * `nil` is generally mapped to `0.0` if NilIsZero is `true`, otherwise it will throw an error * for bool `true` is mapped to `1.0` and `false` is mapped to `0.0` * for string the following logic applies * `"true"` and `"yes"` are mapped to `1.0`, `"false"`, `"no"` and `"unknown"` are mapped to `0.0` (all case-insensitive) * RFC3339 times are parsed to float timestamp * Quantities like "250m" or "512Gi" are parsed to float using * Percentages ending with a "%" are parsed to float * finally the string is parsed to float using which should support all common number formats. If that fails an error is yielded ##### Example for status conditions on Kubernetes Controllers ```yaml kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: group: myteam.io kind: "Foo" version: "v1" labelsFromPath: name: - metadata - name namespace: - metadata - namespace metrics: - name: "foo_status" help: "status condition " each: type: Gauge gauge: path: [status, conditions] labelsFromPath: type: ["type"] valueFrom: ["status"] ``` This will work for kubernetes controller CRs which expose status conditions according to the kubernetes api (): ```yaml status: conditions: - lastTransitionTime: "2019-10-22T16:29:31Z" status: "True" type: Ready ``` kube_customresource_foo_status{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", type="Ready"} 1.0 #### StateSet > StateSets represent a series of related boolean values, also called a bitset. If ENUMs need to be encoded this MAY be done via StateSet. [[1]](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#stateset) ```yaml kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: group: myteam.io kind: "Foo" version: "v1" metrics: - name: "status_phase" help: "Foo status_phase" each: type: StateSet stateSet: labelName: phase path: [status, phase] list: [Pending, Bar, Baz] ``` Metrics of type `StateSet` will generate a metric for each value defined in `list` for each resource. The value will be 1, if the value matches the one in list. If `path` (or `valueFrom`, if set) does not resolve, for example because the status field is not populated yet, no metrics are produced for that resource and no error is logged. Once the field is set, the metrics appear. A value that resolves to a non-string type is reported as an error. Produces the metric: ```prometheus kube_customresource_status_phase{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", phase="Pending"} 1 kube_customresource_status_phase{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", phase="Bar"} 0 kube_customresource_status_phase{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", phase="Baz"} 0 ``` #### Info > Info metrics are used to expose textual information which SHOULD NOT change during process lifetime. Common examples are an application's version, revision control commit, and the version of a compiler. [[2]](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#info) Metrics of type `Info` will always have a value of 1. ```yaml kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: group: myteam.io kind: "Foo" version: "v1" metrics: - name: "version" help: "Foo version" each: type: Info info: labelsFromPath: version: [spec, version] ``` Produces the metric: ```prometheus kube_customresource_version{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", version="v1.2.3"} 1 ``` ### Naming The default metric names are prefixed to avoid collisions with other metrics. By default, a metric prefix of `kube_` concatenated with your custom resource's group+version+kind is used. You can override this behavior with the `metricNamePrefix` field. ```yaml kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: ... metricNamePrefix: myteam_foos metrics: - name: uptime # ... ``` Produces: ```prometheus myteam_foos_uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21 ``` To omit namespace and/or subsystem altogether, set them to the empty string: ```yaml kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: ... metricNamePrefix: "" metrics: - name: uptime # ... ``` Produces: ```prometheus uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21 ``` ### Logging If a metric path is registered but not found on a custom resource, an error will be logged. For some resources, this may produce a lot of noise. The error log [verbosity][vlog] for a metric or resource can be set with `errorLogV` on the resource or metric: ```yaml kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: ... errorLogV: 0 # 0 = default for errors metrics: - name: uptime errorLogV: 10 # only log at high verbosity ``` [vlog]: https://github.com/go-logr/logr#why-v-levels ### Path Syntax Paths are specified as a list of strings. Each string is a path segment, resolved dynamically against the data of the custom resource. If any part of a path is missing, the result is nil. Examples: ```yaml # simple path lookup [spec, replicas] # spec.replicas == 1 # indexing an array [spec, order, "0", value] # spec.order[0].value = true # finding an element in a list by key=value [status, conditions, "[name=a]", value] # status.conditions[0].value = 45 # if the value to be matched is a number or boolean, the value is compared as a number or boolean [status, conditions, "[value=66]", name] # status.conditions[1].name = "b" # For generally matching against a field in an object schema, use the following syntax: [metadata, "name=foo"] # if v, ok := metadata[name]; ok && v == "foo" { return v; } else { /* ignore */ } ``` ### Wildcard matching of version and kind fields The Custom Resource State (CRS hereon) configuration also allows you to monitor all versions and/or kinds that come under a group. It watches the installed CRDs for this purpose. Taking the aforementioned `Foo` object as reference, the configuration below allows you to monitor all objects under all versions *and* all kinds that come under the `myteam.io` group. ```yaml kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: group: "myteam.io" version: "*" # Set to `v1 to monitor all kinds under `myteam.io/v1`. Wildcard matches all installed versions that come under this group. kind: "*" # Set to `Foo` to monitor all `Foo` objects under the `myteam.io` group (under all versions). Wildcard matches all installed kinds that come under this group (and version, if specified). metrics: - name: "myobject_info" help: "Foo Bar Baz" each: type: Info info: path: [metadata] labelsFromPath: object: [name] namespace: [namespace] ``` The configuration above produces these metrics. ```yaml kube_customresource_myobject_info{customresource_group="myteam.io",customresource_kind="Foo",customresource_version="v1",namespace="ns",object="foo"} 1 kube_customresource_myobject_info{customresource_group="myteam.io",customresource_kind="Bar",customresource_version="v1",namespace="ns",object="bar"} 1 ``` #### Note * Only versions that the API server actually serves (`served: true` in the CRD's `spec.versions`) are discovered. A version that is still declared in the CRD but no longer served cannot be listed or watched, so the wildcard will not resolve to it, and pinning it explicitly will not generate any metrics. Such skipped versions are logged at verbosity level 1 (`-v=1`). * For cases where the GVKs defined in a CRD have multiple versions under a single group for the same kind, as expected, the wildcard value will resolve to *all* versions, but a query for any specific version will return all resources under all versions, in that versions' representation. This basically means that for two such versions `A` and `B`, if a resource exists under `B`, it will reflect in the metrics generated for `A` as well, in addition to any resources of itself, and vice-versa. This logic is based on the [current `list`ing behavior](https://github.com/kubernetes/client-go/issues/1251#issuecomment-1544083071) of the client-go library. * The introduction of this feature further discourages (and discontinues) the use of native objects in the CRS featureset, since these do not have an explicit CRD associated with them, and conflict with internal stores defined specifically for such native resources. Please consider opening an issue or raising a PR if you'd like to expand on the current metric labelsets for them. Also, any such configuration will be ignored, and no metrics will be generated for the same. ---