README

Documentation

This documentation is intended to be a complete reflection of the current state of the exposed metrics of kube-state-metrics.

Any contribution to improving this documentation or adding sample usages will be appreciated.

* Metrics Stages
* Exposed Metrics
* PromQL Queries
* Join Metrics
* CLI arguments

Metrics Stages

Stages about metrics are grouped into three categories：

| Stage | Description |
| ------------ | -------------------------------------------------------------------------------------------------------------------------- |
| EXPERIMENTAL | Metrics which normally correspond to the Kubernetes API object alpha status or spec fields and can be changed at any time. |
| STABLE | Metrics which should have very few backwards-incompatible changes outside of major version updates. |
| DEPRECATED | Metrics which will be removed once the deprecation timeline is met. |

Opt-in Metrics

As of v2.3.0, kube-state-metrics supports additional opt-in metrics via the CLI flag --metric-opt-in-list. See the metric documentation to identify which metrics need to be specified.

Exposed Metrics

Per group of metrics there is one file for each metrics.
See each file for specific documentation about the exposed metrics:

Default Resources

* CertificateSigningRequest Metrics
* ConfigMap Metrics
* CronJob Metrics
* DaemonSet Metrics
* Deployment Metrics
* Endpoint Metrics
* Horizontal Pod Autoscaler Metrics
* Ingress Metrics
* Job Metrics
* Lease Metrics
* LimitRange Metrics
* MutatingAdmissionPolicy Metrics
* MutatingAdmissionPolicyBinding Metrics
* MutatingWebhookConfiguration Metrics
* Namespace Metrics
* NetworkPolicy Metrics
* Node Metrics
* PersistentVolume Metrics
* PersistentVolumeClaim Metrics
* Pod Disruption Budget Metrics
* Pod Metrics
* ReplicaSet Metrics
* ReplicationController Metrics
* ResourceQuota Metrics
* Secret Metrics
* Service Metrics
* StatefulSet Metrics
* StorageClass Metrics
* ValidatingAdmissionPolicy Metrics
* ValidatingAdmissionPolicyBinding Metrics
* ValidatingWebhookConfiguration Metrics
* VolumeAttachment Metrics

Optional Resources

* ClusterRole Metrics
* ClusterRoleBinding Metrics
* EndpointSlice Metrics
* IngressClass Metrics
* Role Metrics
* RoleBinding Metrics
* ServiceAccount Metrics

PromQL Queries

For practical PromQL query examples and alerting rules:

* StatefulSet Queries & Alerts

Join Metrics

When an additional, not provided by default label is needed, a Prometheus matching operator
can be used to extend single metrics output.

This example adds label_release to the set of default labels of the kube_pod_status_ready metric
and allows you select or group the metrics by Helm release label:

promql

kube_pod_status_ready * on (namespace, pod) group_left(label_release) kube_pod_labels

Another useful example would be to query the memory usage of pods by its phase, such as Running:

promql

sum(kube_pod_container_resource_requests{resource="memory"}) by (namespace, pod, node)
  * on (namespace, pod) group_left() (sum(kube_pod_status_phase{phase="Running"}) by (pod, namespace) == 1)

Metrics from Custom Resources

NOTE

custom-resource-state is feature-frozen in favor of resource-state-metrics. Once resource-state-metrics is stable, custom-resource-state will be deprecated.

See Custom Resource State Metrics for experimental support for custom resources.

CLI Arguments

Additionally, options for kube-state-metrics can be passed when executing as a CLI, or in a kubernetes / openshift environment. More information can be found here: CLI Arguments

Protecting /metrics endpoints

Kube-State-Metrics' metrics can contain sensitive information about the state of the cluster, which you as an operator might want to additionally protect from unauthorized access.
In order to achieve this, you need to enable the --auth-filter flag on kube-state-metrics.
With this, kube-state-metrics will only accept authenticated and authorized requests to the /metrics endpoints.
Kube-state-metrics uses Kubernetes' RBAC mechanisms for this, so this means that every scrape will trigger a request against the API Server for TokenReview and SubjectAccessReview.
The clients scraping the endpoint, need to use a token which can be provided by a ServiceAccount that can be set up the following way:

A ClusterRole providing access like this:

yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metrics-reader
rules:
- nonResourceURLs:
  - "/metrics"
  verbs:
  - get

and a matching ClusterRoleBinding

yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-reader-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metrics-reader
subjects:
- kind: ServiceAccount
  name: YOUR_SERVICE_ACCOUNT
  namespace: NAMESPACE_OF_THE_SERVICE_ACCOUNT

Your client can then use either this ServiceAccount to gather metrics or you can create a token, that can be used to fetch data like this:

bash

TOKEN=$(kubectl create token YOUR_SERVICE_ACCOUNT -n NAMESPACE_OF_THE_SERVICE_ACCOUNT)
curl -H "Authorization: Bearer $TOKEN" KUBE_STATE_METRICS_URL:8080/metrics

---

Design/Metrics Best Practices

Kube-State-Metrics - Timeseries best practices

---

Author: Manuel Rüger (<[email protected]>)

Date: October 17th 2024

---

Introduction

Kube-State-Metrics' goal is to provide insights into the state of Kubernetes objects by exposing them as metrics.
This document provides guidelines with the goal to create a good user experience when using these metrics.

Please be aware that this document is introduced in a later stage of the project and there might be metrics that do not follow these best practices.
Feel encouraged to report these metrics and provide a pull request to improve them.

General best practices

We follow Prometheus best practices in terms of naming and labeling.

Best practices for kube-state-metrics

Avoid pre-computation

kube-state-metrics should expose metrics on an individual object level and avoid any sort of pre-computation unless it is required due to for example high cardinality on objects.
We prefer not to add metrics that can be derived from existing raw metrics. For example, we would not want to expose a metric called kube_pod_total as it can be computed with count(kube_pod_info).
This way kube-state-metrics allows the user to have full control on how they want to use the metrics and gives them flexibility to do specific computation.

Static object properties

An object usually has a stable set of properties that do not change during its lifecycle in Kubernetes.
This includes properties like name, namespace, uid etc. that have a 1:1 relationship with the object.
It is a good practice to group those together into an _info metric.
If there is a 1:n relationship (e.g. a list of ports), it should be in a separate metric to avoid generating too many metrics.

Dynamic object properties

An object can also have a dynamic set of properties, which are usually part of the status field.
These change during the lifecycle of the object.
For example a pod can be in different states like "Pending", "Running" etc.
These should be part of a "State Set" that includes labels that identify the object as well as the dynamic property.

Linked properties

If an object contains a substructure that links multiple properties together (e.g. endpoint address and port), those should be reported in the same metric.

Optional properties

Some Kubernetes objects have optional fields. In case there is an optional value, the label should still be exposed, ideally as an empty string.

Timestamps

Timestamps like creation time or modification time should be exposed as a value. The metric should end with _timestamp_seconds. The date value is represented in UNIX epoch seconds.

Cardinality

Some object properties can cause cardinality issues if they can contain a lot of different values or are linked together with multiple properties that also can change a lot.
In this case it is better to limit the number of values that can be exposed within kube-state-metrics by allowing only a few of them and have a default for others.
If for example the Kubernetes object contains a status field that contains an error message that can change a lot, it would be better to have a boolean error="true" label in case there is an error.
If there are some error messages that are worth exposing, those could be exposed and for any other message, a default value could be provided.

Stability

We follow the stability framework derived from Kubernetes, in which we expose experimental and stable metrics.
Experimental metrics are recently introduced or expose alpha/beta resources in the Kubernetes API.
They can change anytime and should be used with caution.
They can be promoted to a stable metric once the object stabilized in the Kubernetes API or they were part of two consecutive releases and haven't observed any changes in them.

Stable metrics are considered frozen with the exception of new labels being added.
A stable metric or a label on a stable metric can be deprecated in release Major.Minor and the earliest point it will be removed is the release Major.Minor+2.

---

Design/Metrics Store Performance Optimization

Kube-State-Metrics - Performance Optimization Proposal

---

Author: Max Inden (<[email protected]>)

Date: 23. July 2018

Target release: v1.5.0

---

Glossary

* kube-state-metrics: “Simple service that listens to the Kubernetes API server
and generates metrics about the state of the objects”

* Time series: A single line in a /metrics response e.g.
“metric_name{label="value"} 1”

Problem Statement

There has been repeated reports of two issues running kube-state-metrics on
production Kubernetes clusters. First kube-state-metrics takes a long time
(“10s - 20s”) to respond on its /metrics endpoint, leading to Prometheus
instances dropping the scrape interval request and marking the given time series
as stale. Second kube-state-metrics uses a lot of memory and thereby being
out-of-memory killed due to low set Kubernetes resource limits.

Goal

The goal of this proposal can be split into the following sub-goals ordered by
their priority:

1. Decrease response time on /metrics endpoint

2. Decrease overall runtime memory usage

Status Quo

Instead of requesting the needed information from the Kubernetes API-Server on
demand (on scrape), kube-state-metrics uses the Kubernetes client-go cache tool
to keep a full in memory representation of all Kubernetes objects of a given
cluster. Using the cache speeds up the performance critical path of replying to
a scrape request, and reduces the load on the Kubernetes API-Server by only
sending deltas whenever they occur. Kube-state-metrics does not make use of all
properties and sub-objects of these Kubernetes objects that it stores in its
cache.

On a scrape request by e.g. Prometheus on the /metrics endpoint
kube-state-metrics calculates the configured time series on demand based on the
objects in its cache and converts them to the Prometheus string representation.

Proposal

Instead of a full representation of all Kubernetes objects with all its
properties in memory via the Kubernetes client-go cache, use a map, addressable
by the Kubernetes object uuid, containing all time series of that object as a
single multi-line string.

var cache = map[uuid][]byte{}

Kube-state-metrics listens on add, update and delete events via Kubernetes
client-go reflectors. On add and update events kube-state-metrics generates all
time series related to the Kubernetes object based on the event’s payload,
concatenates the time series to a single byte slice and sets / replaces the byte
slice in the store at the uuid of the Kubernetes object. One can precompute the
length of a time series byte slice before allocation as the sum of the length of
the metric name, label keys and values as well as the metric value in string
representation. On delete events kube-state-metrics deletes the uuid entry of
the given Kubernetes object in the cache map.

On a scrape request on the /metrics endpoint, kube-state-metrics iterates over
the cache map and concatenates all time series string blobs into a single
string, which is finally passed on as a response.

text

/ Detailed source-code truncated for AI context efficiency. /

<details>
<summary>Code to reproduce diagram</summary>

Build via text-diagram

text

object pod_reflector pod_store pod_collector metrics_endpoint
note left of pod_reflector: new pod p1
pod_reflector -> pod_store: Add(p1)
note right of pod_store: generateMetrics(p1)
pod_store -> pod_reflector: nilnote right of metrics_endpoint: GET /metrics
metrics_endpoint -> pod_collector: Collect()
pod_collector -> pod_store: GetAll()
pod_store -> pod_collector: []string{metrics}
pod_collector -> metrics_endpoint: concat(metrics)

</details>

FAQ / Follow up improvements

* If kube-state-metrics only listens on add, update and delete events, how is it
aware of already existing Kubernetes objects created before kube-state-metrics
was started? Leveraging Kubernetes client-go, reflectors can initialize all
existing objects before any add, update or delete events. To ensure no events
are missed in the long run, periodic resyncs via Kubernetes client-go can be
triggered. This extra confidence is not a must and should be compared to its
costs, as Kubernetes client-go already gives decent guarantees on event
delivery.

* What about metadata (HELP and description) in the /metrics output? As a first
iteration they would be skipped until we have a better idea on the design.

* How can the cache map be concurrently accessed? The core golang map
implementation is not thread-safe. As a first iteration a simple mutex should
be sufficient. Golang's sync.Map might be considered.

* To solve the problem of out of order events send by the Kubernetes API-Server
to kube-state-metrics, to each blob of time series inside the cache map it can
keep the Kubernetes resource version. On add and update events, first compare
the resource version of the event with than the resource version in the cache.
Only move forward if the former is higher than the latter.

* In case the memory consumption of the time series string blobs is a problem
the following optimization can be considered: Among the time series strings,
multiple sub-strings will be heavily duplicated like the metric name. Instead
of saving unstructured strings inside the cache map, one can structure them,
using pointers to deduplicate e.g. metric names.

* Kube-state-metrics does not make use of all properties of all Kubernetes
objects. Instead of unmarshalling unused properties, their json struct tags or
their Protobuf representation could be removed.

---

Developer/Cli Arguments

Command line arguments

kube-state-metrics can be configured through command line arguments.

Those arguments can be passed during startup when running locally:

kube-state-metrics --telemetry-port=8081 --kubeconfig=<KUBE-CONFIG> --apiserver=<APISERVER> ...

Or configured in the args section of your deployment configuration in a Kubernetes / Openshift context:

yaml

spec:
  template:
    spec:
      containers:
        - args:
          - '--telemetry-port=8081'
          - '--kubeconfig=<KUBE-CONFIG>'
          - '--apiserver=<APISERVER>'

Available options

text

/ Detailed source-code truncated for AI context efficiency. /

---

Developer/Guide

Developer Guide

This developer guide documentation is intended to assist all contributors in various code contributions.
Any contribution to improving this documentation will be appreciated.

* Add New Kubernetes Resource Metric Collector
* Add New Metrics

Add New Kubernetes Resource Metric Collector

The following steps are needed to introduce a new resource and its respective resource metrics.

* Reference your new resource(s) to the docs/README.md.
* Reference your new resource(s) in the docs/developer/cli-arguments.md as part of the --resources flag.
* Create a new <name-of-resource>.md in the docs directory to provide documentation on the resource(s) and metrics you implemented. Follow the formatting of all other resources.
* Add the resource(s) you are representing to the jsonnet/kube-state-metrics/kube-state-metrics.libsonnet under the appropriate apiGroup using the verbs: list and watch.
* Run make examples/standard, this should re-generate examples/standard/cluster-role.yaml with the resource(s) added to jsonnet/kube-state-metrics/kube-state-metrics.libsonnet.
* Reference and add build functions for the new resource(s) in internal/store/builder.go.
* Reference the new resource in pkg/options/resource.go.
* Add a sample Kubernetes manifest to be used by tests in the tests/manifests/ directory.
* Lastly, and most importantly, actually implement your new resource(s) and its test binary in internal/store. Follow the formatting and structure of other resources.

Add New Metrics

* Make metrics experimental first when introducing them, refer #1910 for more information.

| Metric stability level | |
|------------------------|--------------------|
| EXPERIMENTAL | basemetrics.ALPHA |
| STABLE | basemetrics.STABLE |

---

Metrics/Workload/Cronjob Metrics

CronJob Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ---------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_cronjob_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | cronjob=<cronjob-name> namespace=<cronjob-namespace> annotation_CRONJOB_ANNOTATION=<CRONJOB_ANNOTATION> | EXPERIMENTAL |
| kube_cronjob_info | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> schedule=<schedule> concurrency_policy=<concurrency-policy> timezone=<timezone> | STABLE |
| kube_cronjob_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | cronjob=<cronjob-name> namespace=<cronjob-namespace> label_CRONJOB_LABEL=<CRONJOB_LABEL> | STABLE |
| kube_cronjob_created | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | STABLE |
| kube_cronjob_next_schedule_time | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | STABLE |
| kube_cronjob_schedule_invalid | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | EXPERIMENTAL |
| kube_cronjob_status_active | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | STABLE |
| kube_cronjob_status_last_schedule_time | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | STABLE |
| kube_cronjob_status_last_successful_time | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | STABLE |
| kube_cronjob_spec_suspend | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | STABLE |
| kube_cronjob_spec_starting_deadline_seconds | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | STABLE |
| kube_cronjob_metadata_resource_version | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | STABLE |
| kube_cronjob_spec_successful_job_history_limit | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | EXPERIMENTAL |
| kube_cronjob_spec_failed_job_history_limit | Gauge | | cronjob=<cronjob-name> namespace=<cronjob-namespace> | EXPERIMENTAL |

---

Metrics/Workload/Daemonset Metrics

DaemonSet Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ---------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_daemonset_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | daemonset=<daemonset-name> namespace=<daemonset-namespace> annotation_DAEMONSET_ANNOTATION=<DAEMONSET_ANNOTATION> | EXPERIMENTAL |
| kube_daemonset_created | Gauge | | daemonset=<daemonset-name> namespace=<daemonset-namespace> | STABLE |
| kube_daemonset_status_current_number_scheduled | Gauge | | daemonset=<daemonset-name> namespace=<daemonset-namespace> | STABLE |
| kube_daemonset_status_desired_number_scheduled | Gauge | | daemonset=<daemonset-name> namespace=<daemonset-namespace> | STABLE |
| kube_daemonset_status_number_available | Gauge | | daemonset=<daemonset-name> namespace=<daemonset-namespace> | STABLE |
| kube_daemonset_status_number_misscheduled | Gauge | | daemonset=<daemonset-name> namespace=<daemonset-namespace> | STABLE |
| kube_daemonset_status_number_ready | Gauge | | daemonset=<daemonset-name> namespace=<daemonset-namespace> | STABLE |
| kube_daemonset_status_number_unavailable | Gauge | | daemonset=<daemonset-name> namespace=<daemonset-namespace> | STABLE |
| kube_daemonset_status_observed_generation | Gauge | | daemonset=<daemonset-name> namespace=<daemonset-namespace> | STABLE |
| kube_daemonset_status_updated_number_scheduled | Gauge | | daemonset=<daemonset-name> namespace=<daemonset-namespace> | STABLE |
| kube_daemonset_metadata_generation | Gauge | | daemonset=<daemonset-name> namespace=<daemonset-namespace> | STABLE |
| kube_daemonset_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | daemonset=<daemonset-name> namespace=<daemonset-namespace> label_DAEMONSET_LABEL=<DAEMONSET_LABEL> | STABLE |
| kube_daemonset_deletion_timestamp | Gauge | Unix deletion timestamp | daemonset=<daemonset-name> namespace=<daemonset-namespace> | EXPERIMENTAL |

---

Metrics/Workload/Deployment Metrics

Deployment Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ----------------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_deployment_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | deployment=<deployment-name> namespace=<deployment-namespace> annotation_DEPLOYMENT_ANNOTATION=<DEPLOYMENT_ANNOTATION> | EXPERIMENTAL |
| kube_deployment_status_replicas | Gauge | The number of replicas per deployment. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_status_replicas_ready | Gauge | The number of ready replicas per deployment. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_status_replicas_available | Gauge | The number of available replicas per deployment. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_status_replicas_unavailable | Gauge | The number of unavailable replicas per deployment. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_status_replicas_updated | Gauge | The number of updated replicas per deployment. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_status_terminating_replicas | Gauge | The number of terminating replicas per deployment. | deployment=<deployment-name> namespace=<deployment-namespace> | ALPHA |
| kube_deployment_status_observed_generation | Gauge | The generation observed by the deployment controller. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_status_condition | Gauge | The current status conditions of a deployment. | deployment=<deployment-name> namespace=<deployment-namespace> reason=<deployment-transition-reason> condition=<deployment-condition> status=<true\|false\|unknown> | STABLE |
| kube_deployment_spec_replicas | Gauge | Number of desired pods for a deployment. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_spec_paused | Gauge | Whether the deployment is paused and will not be processed by the deployment controller. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_spec_strategy_rollingupdate_max_unavailable | Gauge | Maximum number of unavailable replicas during a rolling update of a deployment. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_spec_strategy_rollingupdate_max_surge | Gauge | Maximum number of replicas that can be scheduled above the desired number of replicas during a rolling update of a deployment. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_metadata_generation | Gauge | Sequence number representing a specific generation of the desired state. | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | deployment=<deployment-name> namespace=<deployment-namespace> label_DEPLOYMENT_LABEL=<DEPLOYMENT_LABEL> | STABLE |
| kube_deployment_created | Gauge | Unix creation timestamp | deployment=<deployment-name> namespace=<deployment-namespace> | STABLE |
| kube_deployment_deletion_timestamp | Gauge | Unix deletion timestamp | deployment=<deployment-name> namespace=<deployment-namespace> | EXPIREMENTAL |
| kube_deployment_owner | Gauge | Information about the Deployment's owner. | deployment=<deployment-name> namespace=<deployment-namespace> owner_kind=<owner-kind> owner_name=<owner-name> | ALPHA |

---

Metrics/Workload/Horizontalpodautoscaler Metrics

Horizontal Pod Autoscaler Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ---------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_horizontalpodautoscaler_info | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> scaletargetref_api_version=<hpa-target-api-version> scaletargetref_kind=<hpa-target-kind> scaletargetref_name=<hpa-target-name> | EXPERIMENTAL |
| kube_horizontalpodautoscaler_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | EXPERIMENTAL |
| kube_horizontalpodautoscaler_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | STABLE |
| kube_horizontalpodautoscaler_metadata_generation | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | STABLE |
| kube_horizontalpodautoscaler_spec_max_replicas | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | STABLE |
| kube_horizontalpodautoscaler_spec_min_replicas | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | STABLE |
| kube_horizontalpodautoscaler_spec_target_metric | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> metric_name=<metric-name> metric_target_type=<value\|utilization\|average> container=<container-name> | EXPERIMENTAL |
| kube_horizontalpodautoscaler_spec_behavior_scale_down_tolerance | Gauge | The tolerance on the ratio between the current and desired metric value below which no scale down occurs. | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | EXPERIMENTAL |
| kube_horizontalpodautoscaler_spec_behavior_scale_up_tolerance | Gauge | The tolerance on the ratio between the current and desired metric value below which no scale up occurs. | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | EXPERIMENTAL |
| kube_horizontalpodautoscaler_status_target_metric | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> metric_name=<metric-name> metric_target_type=<value\|utilization\|average> container=<container-name> | STABLE |
| kube_horizontalpodautoscaler_status_condition | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> condition=<hpa-condition> status=<true\|false\|unknown> | STABLE |
| kube_horizontalpodautoscaler_status_current_replicas | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | STABLE |
| kube_horizontalpodautoscaler_status_desired_replicas | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | STABLE |
| kube_horizontalpodautoscaler_created | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | EXPERIMENTAL |
| kube_horizontalpodautoscaler_deletion_timestamp | Gauge | | horizontalpodautoscaler=<hpa-name> namespace=<hpa-namespace> | EXPERIMENTAL |

---

Metrics/Workload/Job Metrics

Job Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_job_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | job_name=<job-name> namespace=<job-namespace> annotation_JOB_ANNOTATION=<JOB_ANNOTATION> | EXPERIMENTAL |
| kube_job_info | Gauge | | job_name=<job-name> namespace=<job-namespace> | STABLE |
| kube_job_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | job_name=<job-name> namespace=<job-namespace> label_JOB_LABEL=<JOB_LABEL> | STABLE |
| kube_job_owner | Gauge | | job_name=<job-name> namespace=<job-namespace> owner_kind=<owner kind> owner_name=<owner name> owner_is_controller=<whether owner is controller> | STABLE |
| kube_job_spec_parallelism | Gauge | | job_name=<job-name> namespace=<job-namespace> | STABLE |
| kube_job_spec_completions | Gauge | | job_name=<job-name> namespace=<job-namespace> | STABLE |
| kube_job_spec_active_deadline_seconds | Gauge | | job_name=<job-name> namespace=<job-namespace> | STABLE |
| kube_job_status_active | Gauge | | job_name=<job-name> namespace=<job-namespace> | STABLE |
| kube_job_status_succeeded | Gauge | | job_name=<job-name> namespace=<job-namespace> | STABLE |
| kube_job_status_failed | Gauge | | job_name=<job-name> namespace=<job-namespace> reason=<failure reason> | STABLE |
| kube_job_status_start_time | Gauge | | job_name=<job-name> namespace=<job-namespace> | STABLE |
| kube_job_status_completion_time | Gauge | | job_name=<job-name> namespace=<job-namespace> | STABLE |
| kube_job_complete | Gauge | | job_name=<job-name> namespace=<job-namespace> condition=<true\|false\|unknown> | STABLE |
| kube_job_failed | Gauge | | job_name=<job-name> namespace=<job-namespace> condition=<true\|false\|unknown> | STABLE |
| kube_job_created | Gauge | | job_name=<job-name> namespace=<job-namespace> | STABLE |
| kube_job_status_suspended | Gauge | | job_name=<job-name> namespace=<job-namespace> | EXPERIMENTAL |
| kube_job_status_ready | Gauge | The number of ready pods that belong to this Job. | job_name=<job-name> namespace=<job-namespace> | EXPERIMENTAL |

---

Metrics/Workload/Pod Metrics

Pod Metrics

Note: For pod resource requests and limits, kube-scheduler exposes kube_pod_resource_request and kube_pod_resource_limit on its /metrics/resources endpoint. These are computed from the pod's effective resource requirements and already account for pod-level resources (pod.spec.resources, beta and enabled by default since Kubernetes 1.34), so they are recommended over the container-level metrics below. kube-state-metrics does not expose a separate metric for pod-level resources. These metrics are defined in KEP-1748.

| Metric name | Metric type | Description | Unit (where applicable) | Labels/tags | Status | Opt-in |
| ----------------------------------------------------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | ------ |
| kube_pod_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | | pod=<pod-name> namespace=<pod-namespace> annotation_POD_ANNOTATION=<POD_ANNOTATION> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_info | Gauge | Information about pod | | pod=<pod-name> namespace=<pod-namespace> host_ip=<host-ip> pod_ip=<pod-ip> node=<node-name> created_by_kind=<created_by_kind> created_by_name=<created_by_name> uid=<pod-uid> priority_class=<priority_class> host_network=<host_network> | STABLE | - |
| kube_pod_ips | Gauge | Pod IP addresses | | pod=<pod-name> namespace=<pod-namespace> ip=<pod-ip-address> ip_family=<4 OR 6> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_start_time | Gauge | Start time in unix timestamp for a pod | seconds | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_completion_time | Gauge | Completion time in unix timestamp for a pod | seconds | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_owner | Gauge | Information about the Pod's owner | | pod=<pod-name> namespace=<pod-namespace> owner_kind=<owner kind> owner_name=<owner name> owner_is_controller=<whether owner is controller> uid=<pod-uid> | STABLE | - |
| kube_pod_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | | pod=<pod-name> namespace=<pod-namespace> label_POD_LABEL=<POD_LABEL> uid=<pod-uid> | STABLE | - |
| kube_pod_nodeselectors | Gauge | Describes the Pod nodeSelectors | | pod=<pod-name> namespace=<pod-namespace> nodeselector_NODE_SELECTOR=<NODE_SELECTOR> uid=<pod-uid> | EXPERIMENTAL | Opt-in |
| kube_pod_status_phase | Gauge | The pods current phase | | pod=<pod-name> namespace=<pod-namespace> phase=<Pending\|Running\|Succeeded\|Failed\|Unknown> uid=<pod-uid> | STABLE | - |
| kube_pod_status_qos_class | Gauge | The pods current qosClass | | pod=<pod-name> namespace=<pod-namespace> qos_class=<BestEffort\|Burstable\|Guaranteed> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_status_ready | Gauge | Describes whether the pod is ready to serve requests | | pod=<pod-name> namespace=<pod-namespace> condition=<true\|false\|unknown> uid=<pod-uid> | STABLE | - |
| kube_pod_status_scheduled | Gauge | Describes the status of the scheduling process for the pod | | pod=<pod-name> namespace=<pod-namespace> condition=<true\|false\|unknown> uid=<pod-uid> | STABLE | - |
| kube_pod_container_info | Gauge | Information about a container in a pod | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> image=<image-name> image_id=<image-id> image_spec=<image-spec> container_id=<containerid> uid=<pod-uid> | STABLE | - |
| kube_pod_container_status_waiting | Gauge | Describes whether the container is currently in waiting state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_container_status_waiting_reason | Gauge | Describes the reason the container is currently in waiting state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> reason=<container-waiting-reason> uid=<pod-uid> | STABLE | - |
| kube_pod_container_status_running | Gauge | Describes whether the container is currently in running state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_container_state_started | Gauge | Start time in unix timestamp for a pod container | seconds | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_container_status_terminated | Gauge | Describes whether the container is currently in terminated state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_container_status_terminated_reason | Gauge | Describes the reason the container is currently in terminated state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> reason=<container-terminated-reason> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_container_status_last_terminated_reason | Gauge | Describes the last reason the container was in terminated state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> reason=<last-terminated-reason> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_container_status_last_terminated_exitcode | Gauge | Describes the exit code for the last container in terminated state. | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_container_status_last_terminated_timestamp | Gauge | Last terminated time for a pod container in unix timestamp. | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_container_status_ready | Gauge | Describes whether the containers readiness check succeeded | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_status_initialized_time | Gauge | Time when the pod is initialized. | seconds | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_status_ready_time | Gauge | Time when pod passed readiness probes. | seconds | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_status_container_ready_time | Gauge | Time when the container of the pod entered Ready state. | seconds | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_container_status_restarts_total | Counter | The number of container restarts per container | | container=<container-name> namespace=<pod-namespace> pod=<pod-name> uid=<pod-uid> | STABLE | - |
| kube_pod_container_resource_requests | Gauge | The number of requested request resource by a container. It is recommended to use the kube_pod_resource_request metric exposed by kube-scheduler instead, as it is more precise. | cpu=<core> memory=<bytes> | resource=<resource-name> unit=<resource-unit> container=<container-name> pod=<pod-name> namespace=<pod-namespace> node=< node-name> uid=<pod-uid> | STABLE | - |
| kube_pod_container_resource_limits | Gauge | The number of requested limit resource by a container. It is recommended to use the kube_pod_resource_limit metric exposed by kube-scheduler instead, as it is more precise. | cpu=<core> memory=<bytes> | resource=<resource-name> unit=<resource-unit> container=<container-name> pod=<pod-name> namespace=<pod-namespace> node=< node-name> uid=<pod-uid> | STABLE | - |
| kube_pod_overhead_cpu_cores | Gauge | The pod overhead in regards to cpu cores associated with running a pod | core | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_overhead_memory_bytes | Gauge | The pod overhead in regards to memory associated with running a pod | bytes | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_runtimeclass_name_info | Gauge | The runtimeclass associated with the pod | | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_created | Gauge | Unix creation timestamp | seconds | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_deletion_timestamp | Gauge | Unix deletion timestamp | seconds | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_restart_policy | Gauge | Describes the restart policy in use by this pod | | pod=<pod-name> namespace=<pod-namespace> type=<Always\|Never\|OnFailure> uid=<pod-uid> | STABLE | - |
| kube_pod_init_container_info | Gauge | Information about an init container in a pod | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> image=<image-name> image_id=<image-id> image_spec=<image-spec> container_id=<containerid> uid=<pod-uid> restart_policy=<restart-policy> | STABLE | - |
| kube_pod_init_container_state_started | Gauge | Start time in unix timestamp for a pod init container | seconds | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_init_container_status_waiting | Gauge | Describes whether the init container is currently in waiting state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_init_container_status_waiting_reason | Gauge | Describes the reason the init container is currently in waiting state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> reason=<container-waiting-reason> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_init_container_status_running | Gauge | Describes whether the init container is currently in running state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_init_container_status_terminated | Gauge | Describes whether the init container is currently in terminated state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_init_container_status_terminated_reason | Gauge | Describes the reason the init container is currently in terminated state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> reason=<container-terminated-reason> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_init_container_status_last_terminated_reason | Gauge | Describes the last reason the init container was in terminated state | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> reason=<last-terminated-reason> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_init_container_status_last_terminated_exitcode | Gauge | Describes the exit code for the last init container in terminated state. | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_init_container_status_last_terminated_timestamp | Gauge | Last terminated time for a pod init container in unix timestamp | seconds | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_init_container_status_ready | Gauge | Describes whether the init containers readiness check succeeded | | container=<container-name> pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_init_container_status_restarts_total | Counter | The number of restarts for the init container | integer | container=<container-name> namespace=<pod-namespace> pod=<pod-name> uid=<pod-uid> | STABLE | - |
| kube_pod_init_container_resource_limits | Gauge | The number of CPU cores requested limit by an init container | cpu=<core> memory=<bytes> | resource=<resource-name> unit=<resource-unit> container=<container-name> pod=<pod-name> namespace=<pod-namespace> node=< node-name> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_init_container_resource_requests | Gauge | The number of CPU cores requested by an init container | cpu=<core> memory=<bytes> | resource=<resource-name> unit=<resource-unit> container=<container-name> pod=<pod-name> namespace=<pod-namespace> node=< node-name> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_spec_volumes_persistentvolumeclaims_info | Gauge | Information about persistentvolumeclaim and ephemeral volumes in a pod. For ephemeral="true", the persistentvolumeclaim label is the generated claim name <pod-name>-<volume-name>; non-ephemeral volumes retain their declared claim name. | | pod=<pod-name> namespace=<pod-namespace> volume=<volume-name> persistentvolumeclaim=<persistentvolumeclaim-claimname> ephemeral=<true\|false> uid=<pod-uid> | STABLE | - |
| kube_pod_spec_volumes_persistentvolumeclaims_readonly | Gauge | Describes whether a persistentvolumeclaim is mounted read only. Ephemeral volumes always report 0 since the ephemeral volume source does not support a read-only flag. For ephemeral="true", the persistentvolumeclaim label is the generated claim name <pod-name>-<volume-name>; non-ephemeral volumes retain their declared claim name. | bool | pod=<pod-name> namespace=<pod-namespace> volume=<volume-name> persistentvolumeclaim=<persistentvolumeclaim-claimname> ephemeral=<true\|false> uid=<pod-uid> | STABLE | - |
| kube_pod_status_reason | Gauge | The pod status reasons. Emitted only for the reason that is actually set; a missing series does not mean the reason is false. An unrecognized pod.status.reason is reported as Other; conditions and container-terminated reasons outside this list are not reported. | | pod=<pod-name> namespace=<pod-namespace> reason=<Evicted\|NodeAffinity\|NodeLost\|PreemptionByScheduler\|SchedulingGated\|Shutdown\|TerminationByKubelet\|UnexpectedAdmissionError\|Other> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_status_disruption_reason | Gauge | The pod disruption condition reason. Emitted only when a DisruptionTarget condition is present; a missing series does not mean the reason is false. An unrecognized reason is reported as Other. kube_pod_status_reason remains available separately and existing queries against it should not be replaced with this metric. | | pod=<pod-name> namespace=<pod-namespace> reason=<PreemptionByScheduler\|DeletionByTaintManager\|EvictionByEvictionAPI\|DeletionByPodGC\|TerminationByKubelet\|Other> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_status_scheduled_time | Gauge | Unix timestamp when pod moved into scheduled status | seconds | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_status_unschedulable | Gauge | Describes the unschedulable status for the pod | | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | STABLE | - |
| kube_pod_status_unscheduled_time | Gauge | Unix timestamp when pod moved into unscheduled status | seconds | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> | EXPERIMENTAL | - |
| kube_pod_tolerations | Gauge | Information about the pod tolerations | | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> key=<toleration-key> operator=<toleration-operator> value=<toleration-value> effect=<toleration-effect> toleration_seconds=<toleration-seconds> | EXPERIMENTAL | - |
| kube_pod_service_account | Gauge | The service account for a pod | | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> service_account=<service_account> | EXPERIMENTAL | - |
| kube_pod_scheduler | Gauge | The scheduler for a pod | | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> name=<scheduler-name> | EXPERIMENTAL | - |
| kube_pod_resourceclaim_info | Gauge | Information about a DRA ResourceClaim referenced by a pod, one series per pod.spec.resourceClaims entry | | pod=<pod-name> namespace=<pod-namespace> uid=<pod-uid> claim_name=<pod-local-claim-name> resourceclaim_name=<resolved-resourceclaim-name> resourceclaim_template_name=<resourceclaimtemplate-name> | EXPERIMENTAL | - |

Useful metrics queries

How to retrieve non-standard Pod state

It is not straightforward to get the Pod states for certain cases like "Terminating" and "Unknown" since it is not stored behind a field in the Pod.Status.

So to mimic the logic used by the kubectl command line, you will need to compose multiple metrics.

For example:

To get the list of pods that are in the Unknown state, you can run the following PromQL query: sum(kube_pod_status_phase{phase="Unknown"}) by (namespace, pod) or (count(kube_pod_deletion_timestamp) by (namespace, pod) sum(kube_pod_status_reason{reason="NodeLost"}) by(namespace, pod))

* For Pods in Terminating state: count(kube_pod_deletion_timestamp) by (namespace, pod) unless count(kube_pod_status_reason{reason="NodeLost"}) by (namespace, pod)

Here is an example of a Prometheus rule that can be used to alert on a Pod that has been in the Terminating state for more than 5m.

yaml

groups:
- name: Pod state
  rules:
  - alert: PodsBlockedInTerminatingState
    expr: count(kube_pod_deletion_timestamp) by (namespace, pod) unless count(kube_pod_status_reason{reason="NodeLost"}) by (namespace, pod)
    for: 5m
    labels:
      severity: page
    annotations:
      summary: Pod {{$labels.namespace}}/{{$labels.pod}} blocked in Terminating state.

---

Metrics/Workload/Replicaset Metrics

ReplicaSet metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| --------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_replicaset_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | replicaset=<replicaset-name> namespace=<replicaset-namespace> annotation_REPLICASET_ANNOTATION=<REPLICASET_ANNOTATION> | EXPERIMENTAL |
| kube_replicaset_status_replicas | Gauge | | replicaset=<replicaset-name> namespace=<replicaset-namespace> | STABLE |
| kube_replicaset_status_fully_labeled_replicas | Gauge | | replicaset=<replicaset-name> namespace=<replicaset-namespace> | STABLE |
| kube_replicaset_status_ready_replicas | Gauge | | replicaset=<replicaset-name> namespace=<replicaset-namespace> | STABLE |
| kube_replicaset_status_terminating_replicas | Gauge | The number of terminating replicas per ReplicaSet. | replicaset=<replicaset-name> namespace=<replicaset-namespace> | ALPHA |
| kube_replicaset_status_observed_generation | Gauge | | replicaset=<replicaset-name> namespace=<replicaset-namespace> | STABLE |
| kube_replicaset_spec_replicas | Gauge | | replicaset=<replicaset-name> namespace=<replicaset-namespace> | STABLE |
| kube_replicaset_metadata_generation | Gauge | | replicaset=<replicaset-name> namespace=<replicaset-namespace> | STABLE |
| kube_replicaset_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | replicaset=<replicaset-name> namespace=<replicaset-namespace> label_REPLICASET_LABEL=<REPLICASET_LABEL> | STABLE |
| kube_replicaset_created | Gauge | | replicaset=<replicaset-name> namespace=<replicaset-namespace> | STABLE |
| kube_replicaset_owner | Gauge | | replicaset=<replicaset-name> namespace=<replicaset-namespace> owner_kind=<owner kind> owner_name=<owner name> owner_is_controller=<whether owner is controller> | STABLE |

---

Metrics/Workload/Replicationcontroller Metrics

ReplicationController metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| -------------------------------------------------------- | ----------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ |
| kube_replicationcontroller_status_replicas | Gauge | | replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace> | STABLE |
| kube_replicationcontroller_status_fully_labeled_replicas | Gauge | | replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace> | STABLE |
| kube_replicationcontroller_status_ready_replicas | Gauge | | replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace> | STABLE |
| kube_replicationcontroller_status_available_replicas | Gauge | | replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace> | STABLE |
| kube_replicationcontroller_status_observed_generation | Gauge | | replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace> | STABLE |
| kube_replicationcontroller_spec_replicas | Gauge | | replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace> | STABLE |
| kube_replicationcontroller_metadata_generation | Gauge | | replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace> | STABLE |
| kube_replicationcontroller_created | Gauge | | replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace> | STABLE |
| kube_replicationcontroller_owner | Gauge | | replicationcontroller=<replicationcontroller-name> namespace=<replicationcontroller-namespace> owner_kind=<owner kind> owner_name=<owner name> owner_is_controller=<whether owner is controller> | EXPERIMENTAL |

---

Metrics/Workload/Statefulset Metrics

Stateful Set Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ------------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_statefulset_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | statefulset=<statefulset-name> namespace=<statefulset-namespace> annotation_STATEFULSET_ANNOTATION=<STATEFULSET_ANNOTATION> | STABLE |
| kube_statefulset_status_replicas | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |
| kube_statefulset_status_replicas_current | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |
| kube_statefulset_status_replicas_ready | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |
| kube_statefulset_status_replicas_available | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |
| kube_statefulset_status_replicas_updated | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |
| kube_statefulset_status_observed_generation | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |
| kube_statefulset_replicas | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |
| kube_statefulset_ordinals_start | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |
| kube_statefulset_metadata_generation | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |
| kube_statefulset_persistentvolumeclaim_retention_policy | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> when_deleted=<statefulset-when-deleted-pvc-policy> when_scaled=<statefulset-when-scaled-pvc-policy> | STABLE |
| kube_statefulset_created | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |
| kube_statefulset_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | statefulset=<statefulset-name> namespace=<statefulset-namespace> label_STATEFULSET_LABEL=<STATEFULSET_LABEL> | STABLE |
| kube_statefulset_status_current_revision | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> revision=<statefulset-current-revision> | STABLE |
| kube_statefulset_status_update_revision | Gauge | | statefulset=<statefulset-name> namespace=<statefulset-namespace> revision=<statefulset-update-revision> | STABLE |
| kube_statefulset_deletion_timestamp | Gauge | Unix deletion timestamp | statefulset=<statefulset-name> namespace=<statefulset-namespace> | STABLE |

---

Metrics/Storage/Configmap Metrics

ConfigMap Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ---------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_configmap_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | configmap=<configmap-name> namespace=<configmap-namespace> annotation_CONFIGMAP_ANNOTATION=<CONFIGMAP_ANNOTATION> | EXPERIMENTAL |
| kube_configmap_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | configmap=<configmap-name> namespace=<configmap-namespace> label_CONFIGMAP_LABEL=<CONFIGMAP_LABEL> | STABLE |
| kube_configmap_info | Gauge | | configmap=<configmap-name> namespace=<configmap-namespace> | STABLE |
| kube_configmap_created | Gauge | | configmap=<configmap-name> namespace=<configmap-namespace> | STABLE |
| kube_configmap_metadata_resource_version | Gauge | | configmap=<configmap-name> namespace=<configmap-namespace> | EXPERIMENTAL |

---

Metrics/Storage/Persistentvolume Metrics

PersistentVolume Metrics

| Metric name | Metric type | Description | Unit (where applicable) | Labels/tags | Status |
|------------------------------------------|-------------|---------------------------------------------------------------------------------------------------------------------------|-------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
| kube_persistentvolume_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | | persistentvolume=<persistentvolume-name> annotation_PERSISTENTVOLUME_ANNOTATION=<PERSISTENTVOLUME_ANNOTATION> | EXPERIMENTAL |
| kube_persistentvolume_capacity_bytes | Gauge | | | persistentvolume=<pv-name> | STABLE |
| kube_persistentvolume_status_phase | Gauge | | | persistentvolume=<pv-name> phase=<Bound\|Failed\|Pending\|Available\|Released> | STABLE |
| kube_persistentvolume_claim_ref | Gauge | | | persistentvolume=<pv-name> claim_namespace=<namespace>; name=<name>; | STABLE |
| kube_persistentvolume_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | | persistentvolume=<persistentvolume-name> label_PERSISTENTVOLUME_LABEL=<PERSISTENTVOLUME_LABEL> | STABLE |
| kube_persistentvolume_info | Gauge | Information about Persistent Volumes | | persistentvolume=<pv-name> storageclass=<storageclass-name> gce_persistent_disk_name=<pd-name> host_path=<path-of-a-host-volume> host_path_type=<host-mount-type> ebs_volume_id=<ebs-volume-id> azure_disk_name=<azure-disk-name> fc_wwids=<fc-wwids-comma-separated> fc_lun=<fc-lun> fc_target_wwns=<fc-target-wwns-comma-separated> iscsi_target_portal=<iscsi-target-portal> iscsi_iqn=<iscsi-iqn> iscsi_lun=<iscsi-lun> iscsi_initiator_name=<iscsi-initiator-name> local_path=<path-of-a-local-volume> local_fs=<local-volume-fs-type> nfs_server=<nfs-server> nfs_path=<nfs-path> csi_driver=<csi-driver> csi_volume_handle=<csi-volume-handle> reclaim_policy=<reclaim-policy> | STABLE |
| kube_persistentvolume_created | Gauge | Unix creation timestamp | seconds | persistentvolume=<persistentvolume-name> | EXPERIMENTAL |
| kube_persistentvolume_deletion_timestamp | Gauge | Unix deletion timestamp | seconds | persistentvolume=<persistentvolume-name> | EXPERIMENTAL |
| kube_persistentvolume_csi_attributes | Gauge | CSI attributes of the Persistent Volume, disabled by default, manage with --metric-opt-in-list) | | persistentvolume=<persistentvolume-name> csi_mounter=<csi-mounter> csi_map_options=<csi-map-options> | EXPERIMENTAL |
| kube_persistentvolume_volume_mode | Gauge | Volume Mode information for the PersistentVolume. | | persistentvolume=<persistentvolume-name> volumemode=<volumemode> | EXPERIMENTAL |
| kube_persistentvolume_access_mode | Gauge | Access modes of the PersistentVolume. | | persistentvolume=<persistentvolume-name> access_mode=<ReadWriteOnce \| ReadOnlyMany \| ReadWriteMany \| ReadWriteOncePod> | EXPERIMENTAL |

Useful metrics queries

How to retrieve non-standard PV state

It is not straightforward to get the PV states for certain cases like "Terminating" since it is not stored behind a field in the PersistentVolume.Status.

So to mimic the logic used by the kubectl command line, you will need to compose multiple metrics.

Here is an example of a Prometheus rule that can be used to alert on a PV that has been in the Terminating state for more than 5m.

yaml

groups:
- name: PV state
  rules:
  - alert: PVBlockedInTerminatingState
    expr: kube_persistentvolume_deletion_timestamp * on(persistentvolume) group_left() (kube_persistentvolume_status_phase{phase="Bound"} == 1) > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: PV {{$labels.persistentvolume}} blocked in Terminating state.

---

Metrics/Storage/Persistentvolumeclaim Metrics

PersistentVolumeClaim Metrics

| Metric name | Metric type | Description | Unit (where applicable) | Labels/tags | Status |
| ---------------------------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------ |
| kube_persistentvolumeclaim_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | | persistentvolumeclaim=<persistentvolumeclaim-name> namespace=<persistentvolumeclaim-namespace> annotation_PERSISTENTVOLUMECLAIM_ANNOTATION=<PERSISTENTVOLUMECLAIM_ANNOATION> | EXPERIMENTAL |
| kube_persistentvolumeclaim_access_mode | Gauge | | | access_mode=<persistentvolumeclaim-access-mode> namespace=<persistentvolumeclaim-namespace> persistentvolumeclaim=<persistentvolumeclaim-name> | STABLE |
| kube_persistentvolumeclaim_info | Gauge | | | namespace=<persistentvolumeclaim-namespace> persistentvolumeclaim=<persistentvolumeclaim-name> storageclass=<persistentvolumeclaim-storageclassname> volumename=<volumename> volumemode=<volumemode> | STABLE |
| kube_persistentvolumeclaim_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | | persistentvolumeclaim=<persistentvolumeclaim-name> namespace=<persistentvolumeclaim-namespace> label_PERSISTENTVOLUMECLAIM_LABEL=<PERSISTENTVOLUMECLAIM_LABEL> | STABLE |
| kube_persistentvolumeclaim_resource_requests_storage_bytes | Gauge | | | namespace=<persistentvolumeclaim-namespace> persistentvolumeclaim=<persistentvolumeclaim-name> | STABLE |
| kube_persistentvolumeclaim_status_condition | Gauge | | | namespace =<persistentvolumeclaim-namespace> persistentvolumeclaim=<persistentvolumeclaim-name> type=<persistentvolumeclaim-condition-type> status=<true\false\unknown> | EXPERIMENTAL |
| kube_persistentvolumeclaim_status_phase | Gauge | | | namespace=<persistentvolumeclaim-namespace> persistentvolumeclaim=<persistentvolumeclaim-name> phase=<Pending\Bound\Lost> | STABLE |
| kube_persistentvolumeclaim_created | Gauge | Unix creation timestamp | seconds | namespace=<persistentvolumeclaim-namespace> persistentvolumeclaim=<persistentvolumeclaim-name> | EXPERIMENTAL |
| kube_persistentvolumeclaim_deletion_timestamp | Gauge | Unix deletion timestamp | seconds | namespace=<persistentvolumeclaim-namespace> persistentvolumeclaim=<persistentvolumeclaim-name> | EXPERIMENTAL |

Note:

* An empty string will be used if PVC has no storage class.

Useful metrics queries

How to retrieve non-standard PVC state

It is not straightforward to get the PVC states for certain cases like "Terminating" since it is not stored behind a field in the PersistentVolumeClaim.Status.

So to mimic the logic used by the kubectl command line, you will need to compose multiple metrics.

Here is an example of a Prometheus rule that can be used to alert on a PVC that has been in the Terminating state for more than 5m.

yaml

groups:
- name: PVC state
  rules:
  - alert: PVCBlockedInTerminatingState
    expr: kube_persistentvolumeclaim_deletion_timestamp * on(namespace, persistentvolumeclaim) group_left() (kube_persistentvolumeclaim_status_phase{phase="Bound"} == 1) > 0
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: PVC {{$labels.namespace}}/{{$labels.persistentvolumeclaim}} blocked in Terminating state.

---

Metrics/Storage/Secret Metrics

Secret Metrics

---

Metrics/Storage/Storageclass Metrics

StorageClass Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ----------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_storageclass_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | storageclass=<storageclass-name> annotation_STORAGECLASS_ANNOTATION=<STORAGECLASS_ANNOTATION> | EXPERIMENTAL |
| kube_storageclass_info | Gauge | | storageclass=<storageclass-name> provisioner=<storageclass-provisioner> reclaim_policy=<storageclass-reclaimPolicy> volume_binding_mode=<storageclass-volumeBindingMode> | STABLE |
| kube_storageclass_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | storageclass=<storageclass-name> label_STORAGECLASS_LABEL=<STORAGECLASS_LABEL> | STABLE |
| kube_storageclass_created | Gauge | | storageclass=<storageclass-name> | STABLE |

---

Metrics/Storage/Volumeattachment Metrics

VolumeAttachment Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| -------------------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_volumeattachment_info | Gauge | | volumeattachment=<volumeattachment-name> attacher=<attacher-name> node=<node-name> | EXPERIMENTAL |
| kube_volumeattachment_created | Gauge | | volumeattachment=<volumeattachment-name> | EXPERIMENTAL |
| kube_volumeattachment_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | volumeattachment=<volumeattachment-name> label_VOLUMEATTACHMENT_LABEL=<VOLUMEATTACHMENT_LABEL> | EXPERIMENTAL |
| kube_volumeattachment_spec_source_persistentvolume | Gauge | | volumeattachment=<volumeattachment-name> volumename=<persistentvolume-name> | EXPERIMENTAL |
| kube_volumeattachment_status_attached | Gauge | | volumeattachment=<volumeattachment-name> | EXPERIMENTAL |
| kube_volumeattachment_status_attachment_metadata | Gauge | | volumeattachment=<volumeattachment-name> metadata_METADATA_KEY=<METADATA_VALUE> | EXPERIMENTAL |

---

Metrics/Service/Endpoint Metrics

Endpoint Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_endpoint_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | endpoint=<endpoint-name> namespace=<endpoint-namespace> annotation_ENDPOINT_ANNOTATION=<ENDPOINT_ANNOTATION> | EXPERIMENTAL |
| kube_endpoint_info | Gauge | | endpoint=<endpoint-name> namespace=<endpoint-namespace> | STABLE |
| kube_endpoint_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | endpoint=<endpoint-name> namespace=<endpoint-namespace> label_ENDPOINT_LABEL=<ENDPOINT_LABEL> | STABLE |
| kube_endpoint_created | Gauge | | endpoint=<endpoint-name> namespace=<endpoint-namespace> | STABLE |
| kube_endpoint_ports | Gauge | | endpoint=<endpoint-name> namespace=<endpoint-namespace> port_name=<endpoint-port-name> port_protocol=<endpoint-port-protocol> port_number=<endpoint-port-number> | STABLE (Deprecated from 2.14.0) |
| kube_endpoint_address | Gauge | | endpoint=<endpoint-name> namespace=<endpoint-namespace> ip=<endpoint-ip> port_name=<endpoint-port-name> port_protocol=<endpoint-port-protocol> port_number=<endpoint-port-number>ready=<true if available, false if unavailalbe> | STABLE |

---

Metrics/Service/Endpointslice Metrics

Endpoint Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ------------------------------ | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_endpointslice_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | endpointslice=<endpointslice-name> namespace=<endpointslice-namespace> annotation_ENDPOINTSLICE_ANNOTATION=<ENDPOINTSLICE_ANNOTATION> | EXPERIMENTAL |
| kube_endpointslice_info | Gauge | | endpointslice=<endpointslice-name> namespace=<endpointslice-namespace> addresstype=<endpointslice-addresstype> | EXPERIMENTAL |
| kube_endpointslice_ports | Gauge | | endpointslice=<endpointslice-name> namespace=<endpointslice-namespace> port_name=<endpointslice-port-name> port_protocol=<endpointslice-port-protocol> port_number=<endpointslice-port-number> | EXPERIMENTAL |
| kube_endpointslice_endpoints | Gauge | | endpointslice=<endpointslice-name> namespace=<endpointslice-namespace> address=<endpointslice-address> ready=<endpointslice-ready> serving=<endpointslice-serving> terminating=<endpointslice-terminating> hostname=<endpointslice-hostname> targetref_kind=<endpointslice-targetref-kind> targetref_name=<endpointslice-targetref-name> targetref_namespace=<endpointslice-targetref-namespace> nodename=<endpointslice-nodename> endpoint_zone=<endpointslice-zone> | EXPERIMENTAL |
| kube_endpointslice_endpoints_hints | Gauge | Each line is a hint applied to an endpoint-slice | endpointslice=<endpointslice-name> namespace=<endpointslice-namespace> address=<endpointslice-address[0]> for_zone=<endpointslice-hint> | EXPERIMENTAL |
| kube_endpointslice_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | endpointslice=<endpointslice-name> namespace=<endpointslice-namespace> label_ENDPOINTSLICE_LABEL=<ENDPOINTSLICE_LABEL> | EXPERIMENTAL |
| kube_endpointslice_created | Gauge | | endpointslice=<endpointslice-name> namespace=<endpointslice-namespace> | EXPERIMENTAL |

---

Metrics/Service/Ingress Metrics

Ingress Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| -------------------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- |------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ------------ |
| kube_ingress_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | ingress=<ingress-name> namespace=<ingress-namespace> annotation_INGRESS_ANNOTATION=<ANNOTATION_LABEL> | EXPERIMENTAL |
| kube_ingress_info | Gauge | | ingress=<ingress-name> namespace=<ingress-namespace> ingressclass=<ingress-class> or _default if not set | STABLE |
| kube_ingress_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | ingress=<ingress-name> namespace=<ingress-namespace> label_INGRESS_LABEL=<INGRESS_LABEL> | STABLE |
| kube_ingress_created | Gauge | | ingress=<ingress-name> namespace=<ingress-namespace> | STABLE |
| kube_ingress_metadata_resource_version | Gauge | | ingress=<ingress-name> namespace=<ingress-namespace> | EXPERIMENTAL |
| kube_ingress_path | Gauge | | ingress=<ingress-name> namespace=<ingress-namespace> host=<ingress-host> path=<ingress-path> path_type=<ingress-path type> If path served by Service Backend service_name=<service name for the path> service_port=<service port for the path> If path served by Resource Backend resource_api_group=<resource backend api group> resource_kind=<resource backend kind> resource_name=<resource backend name> | STABLE |
| kube_ingress_tls | Gauge | | ingress=<ingress-name> namespace=<ingress-namespace> tls_host=<tls hostname> secret=<tls secret name> | STABLE |

---

Metrics/Service/Ingressclass Metrics

IngressClass Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ----------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ | ------------ |
| kube_ingressclass_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | ingressclass=<ingressclass-name> annotation_INGRESSCLASS_ANNOTATION=<INGRESSCLASS_ANNOTATION> | EXPERIMENTAL |
| kube_ingressclass_info | Gauge | | ingressclass=<ingressclass-name> controller=<ingress-controller-name> | EXPERIMENTAL |
| kube_ingressclass_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | ingressclass=<ingressclass-name> label_INGRESSCLASS_LABEL=<INGRESSCLASS_LABEL> | EXPERIMENTAL |
| kube_ingressclass_created | Gauge | | ingressclass=<ingressclass-name> | EXPERIMENTAL |

---

Metrics/Service/Service Metrics

Service Metrics

| Metric name | Metric type | Description | Unit (where applicable) | Labels/tags | Status |
| ----------------------------------------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
| kube_service_annotations | Gauge | Kubernetes annotations converted to Prometheus labels controlled via --metric-annotations-allowlist | | service=<service-name> namespace=<service-namespace> uid=<service-uid> annotation_SERVICE_ANNOTATION=<SERVICE_ANNOTATION> | EXPERIMENTAL |
| kube_service_info | Gauge | Information about service | | service=<service-name> namespace=<service-namespace> uid=<service-uid> cluster_ip=<service cluster ip> external_name=<service external name> external_traffic_policy=<service external traffic policy> load_balancer_ip=<service load balancer ip> | STABLE |
| kube_service_labels | Gauge | Kubernetes labels converted to Prometheus labels controlled via --metric-labels-allowlist | | service=<service-name> namespace=<service-namespace> uid=<service-uid> label_SERVICE_LABEL=<SERVICE_LABEL> | STABLE |
| kube_service_created | Gauge | Unix creation timestamp | seconds | service=<service-name> namespace=<service-namespace> uid=<service-uid> | STABLE |
| kube_service_spec_type | Gauge | Type about service | | service=<service-name> namespace=<service-namespace> uid=<service-uid> type=<ClusterIP\|NodePort\|LoadBalancer\|ExternalName> | STABLE |
| kube_service_spec_external_ip | Gauge | Service external ips. One series for each ip | | service=<service-name> namespace=<service-namespace> uid=<service-uid> external_ip=<external-ip> | STABLE |
| kube_service_status_load_balancer_ingress | Gauge | Service load balancer ingress status | | service=<service-name> namespace=<service-namespace> uid=<service-uid> ip=<load-balancer-ingress-ip> hostname=<load-balancer-ingress-hostname> | STABLE |
| kube_service_deletion_timestamp | Gauge | Unix deletion timestamp | seconds | service=<service-name> namespace=<service-namespace> uid=<service-uid> | EXPERIMENTAL |

---

Metrics/Policy/Limitrange Metrics

LimitRange Metrics

| Metric name | Metric type | Description | Labels/tags | Status |
| ----------------------- | ----------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------ |
| kube_limitrange | Gauge | | limitrange=<limitrange-name> namespace=<namespace> resource=<ResourceName> type=<Pod\|Container\|PersistentVolumeClaim> constraint=<constraint> | STABLE |
| kube_limitrange_created | Gauge | | limitrange=<limitrange-name> namespace=<namespace> | STABLE |

---

Metrics/Policy/Networkpolicy Metrics

Network Policy Metrics

---

Metrics/Policy/Poddisruptionbudget Metrics

PodDisruptionBudget Metrics

---

Metrics/Policy/Resourcequota Metrics

ResourceQuota Metrics

---

Metrics/Extend/Customresourcestate Metrics

Custom Resource State Metrics

NOTE

custom-resource-state is feature-frozen in favor of resource-state-metrics. Once resource-state-metrics is stable, custom-resource-state will be deprecated.

This section describes how to add metrics based on the state of a custom resource without writing a custom resource
registry and running your own build of KSM.

Configuration

A YAML configuration file described below is required to define your custom resources and the fields to turn into metrics.

Two flags can be used:

* --custom-resource-state-config "inline yaml (see example)" or
* --custom-resource-state-config-file /path/to/config.yaml

When using a --config file, the equivalent YAML key is custom_resource_state_config_file. The pre-v2.17 key custom_resource_config_file is still honored as a deprecated alias and will be removed in a future release; please migrate to custom_resource_state_config_file.

If both flags are provided, the inline configuration will take precedence.
When multiple entries for the same resource exist, kube-state-metrics will exit with an error.
This includes configuration which refers to a different API version.

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: kube-state-metrics
        args:
          - --custom-resource-state-config
          # in YAML files, | allows a multi-line string to be passed as a flag value
          # see https://yaml-multiline.info
          -  |
              kind: CustomResourceStateMetrics
              spec:
                resources:
                  - groupVersionKind:
                      group: myteam.io
                      version: "v1"
                      kind: Foo
                    metrics:
                      - name: active_count
                        help: "Count of active Foo"
                        each:
                          type: Gauge
                          ...

It's also possible to configure kube-state-metrics to run in a custom-resource-mode only. In addition to specifying one of --custom-resource-state-config* flags, you could set --custom-resource-state-only to true.
With this configuration only the known custom resources configured in --custom-resource-state-config* will be taken into account by kube-state-metrics.

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: kube-system
spec:
  template:
    spec:
      containers:
      - name: kube-state-metrics
        args:
          - --custom-resource-state-config
          # in YAML files, | allows a multi-line string to be passed as a flag value
          # see https://yaml-multiline.info
          -  |
              kind: CustomResourceStateMetrics
              spec:
                resources:
                  - groupVersionKind:
                      group: myteam.io
                      version: "v1"
                      kind: Foo
                    metrics:
                      - name: active_count
                        help: "Count of active Foo"
                        each:
                          type: Gauge
                          ...
          - --custom-resource-state-only=true

NOTE: The customresource_group, customresource_version, and customresource_kind common labels are reserved, and will be overwritten by the values from the groupVersionKind field.

RBAC-enabled Clusters

Please be aware that kube-state-metrics needs list and watch permissions granted to customresourcedefinitions.apiextensions.k8s.io as well as to the resources you want to gather metrics from.

Examples

The examples in this section will use the following custom resource:

yaml

kind: Foo
apiVersion: myteam.io/vl
metadata:
    annotations:
        bar: baz
        qux: quxx
    labels:
        foo: bar
    name: foo
spec:
    version: v1.2.3
    order:
        - id: 1
          value: true
        - id: 3
          value: false
    replicas: 1
    refs:
        - my_other_foo
        - foo_2
        - foo_with_extensions
status:
    phase: Pending
    active:
        type-a: 1
        type-b: 3
    conditions:
        - name: a
          value: 45
        - name: b
          value: 66
    sub:
        type-a:
            active: 1
            ready: 2
        type-b:
            active: 3
            ready: 4
    uptime: 43.21

#### Single Values

The config:

yaml

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        kind: "Foo"
        version: "v1"
      metrics:
        - name: "uptime"
          help: "Foo uptime"
          each:
            type: Gauge
            gauge:
              path: [status, uptime]

Produces the metric:

prometheus

kube_customresource_uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21

#### Multiple Metrics/Kitchen Sink

text

/ Detailed source-code truncated for AI context efficiency. /

Produces the following metrics:

prometheus

kube_customresource_ready_count{customresource_group="myteam.io", customresource_kind="Foo", 
customresource_version="v1", active="1",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-a",
lorem_bar="baz",lorem_qux="quxx",} 2
kube_customresource_ready_count{customresource_group="myteam.io", customresource_kind="Foo", 
customresource_version="v1", active="3",custom_metric="yes",foo="bar",name="foo",bar="baz",qux="quxx",type="type-b",
lorem_bar="baz",lorem_qux="quxx",} 4

#### Non-map Arrays

yaml

kind: CustomResourceStateMetrics spec: resources: - groupVersionKind: group: myteam.io kind: "Foo" version: "v1" labelsFromPath: name: [metadata, name] metrics: - name: "ref_info" help: "Reference to other Foo" each: type: Info info: # targeting an array will produce a metric for each element # labelsFromPath and value are relative to this path path: [spec, refs]

# if path targets a list of values (e.g. strings or numbers, not objects or maps), individual values can # referenced by a label using this syntax labelsFromPath: ref: []

Produces the following metrics:

prometheus

kube_customresource_ref_info{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", name="foo",ref="my_other_foo"} 1
kube_customresource_ref_info{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", name="foo",ref="foo_2"} 1
kube_customresource_ref_info{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", name="foo",ref="foo_with_extensions"} 1

#### Same Metrics with Different Labels

yaml

recommendation:
    containerRecommendations:
    - containerName: consumer
      lowerBound:
        cpu: 100m
        memory: 262144k

For example in VPA we have above attributes and we want to have a same metrics for both CPU and Memory, you can use below config:

text

/ Detailed source-code truncated for AI context efficiency. /

Produces the following metrics:

prometheus

HELP kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound Minimum memory resources the container can use before the VerticalPodAutoscaler updater evicts it.

TYPE kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound gauge

kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound{container="consumer",customresource_group="autoscaling.k8s.io",customresource_kind="VerticalPodAutoscaler",customresource_version="v1",namespace="namespace-example",resource="memory",target_api_version="apps/v1",target_kind="Deployment",target_name="target-name-example",unit="byte",verticalpodautoscaler="vpa-example"} 123456
HELP kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound Minimum cpu resources the container can use before the VerticalPodAutoscaler updater evicts it.

TYPE kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound gauge

kube_customresource_verticalpodautoscaler_status_recommendation_containerrecommendations_lowerbound{container="consumer",customresource_group="autoscaling.k8s.io",customresource_kind="VerticalPodAutoscaler",customresource_version="v1",namespace="namespace-example",resource="cpu",target_api_version="apps/v1",target_kind="Deployment",target_name="target-name-example",unit="core",verticalpodautoscaler="vpa-example"} 0.1

#### VerticalPodAutoscaler

In v2.9.0 the vericalpodautoscalers resource was removed from the list of default resources. In order to generate metrics for verticalpodautoscalers, you can use the following Custom Resource State config:

text

/ Detailed source-code truncated for AI context efficiency. /

The above configuration was tested on this VPA configuration, with an added annotation (foo: 123).

#### All VerticalPodAutoscaler Metrics

As an addition for the above configuration, here's the complete CustomResourceStateMetrics spec to re-enable all of the VPA metrics which are removed from the list of the default resources:

<summary>VPA CustomResourceStateMetrics</summary>

text

/ Detailed source-code truncated for AI context efficiency. /

</details>

Metric types

The configuration supports three kind of metrics from the OpenMetrics specification.

The metric type is specified by the type field and its specific configuration at the types specific struct.

#### Gauge

Gauges are current measurements, such as bytes of memory currently used or the number of items in a queue. For gauges the absolute value is what is of interest to a user. [[0]](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#gauge)

Example:

yaml

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        kind: "Foo"
        version: "v1"
      metrics:
        - name: "uptime"
          help: "Foo uptime"
          each:
            type: Gauge
            gauge:
              path: [status, uptime]

Produces the metric:

prometheus

kube_customresource_uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21

##### Type conversion and special handling

Gauges produce values of type float64 but custom resources can be of all kinds of types.
Kube-state-metrics performs implicit type conversions for a lot of type.
Supported types are:

* (u)int32/64, int, float32 and byte are cast to float64
* nil is generally mapped to 0.0 if NilIsZero is true, otherwise it will throw an error
* for bool true is mapped to 1.0 and false is mapped to 0.0
* for string the following logic applies
* "true" and "yes" are mapped to 1.0, "false", "no" and "unknown" are mapped to 0.0 (all case-insensitive)
* RFC3339 times are parsed to float timestamp
* Quantities like "250m" or "512Gi" are parsed to float using <https://github.com/kubernetes/apimachinery/blob/master/pkg/api/resource/quantity.go>
* Percentages ending with a "%" are parsed to float
* finally the string is parsed to float using <https://pkg.go.dev/strconv#ParseFloat> which should support all common number formats. If that fails an error is yielded

##### Example for status conditions on Kubernetes Controllers

yaml

kind: CustomResourceStateMetrics
spec:
  resources:
  - groupVersionKind:
      group: myteam.io
      kind: "Foo"
      version: "v1"
    labelsFromPath:
      name:
      - metadata
      - name
      namespace:
      - metadata
      - namespace
    metrics:
    - name: "foo_status"
      help: "status condition "
      each:
        type: Gauge
        gauge:
          path: [status, conditions]
          labelsFromPath:
            type: ["type"]
          valueFrom: ["status"]

This will work for kubernetes controller CRs which expose status conditions according to the kubernetes api (<https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#Condition>):

yaml

status:
  conditions:
    - lastTransitionTime: "2019-10-22T16:29:31Z"
      status: "True"
      type: Ready

kube_customresource_foo_status{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", type="Ready"} 1.0

#### StateSet

StateSets represent a series of related boolean values, also called a bitset. If ENUMs need to be encoded this MAY be done via StateSet. [[1]](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#stateset)

yaml

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        kind: "Foo"
        version: "v1"
      metrics:
        - name: "status_phase"
          help: "Foo status_phase"
          each:
            type: StateSet
            stateSet:
              labelName: phase
              path: [status, phase]
              list: [Pending, Bar, Baz]

Metrics of type StateSet will generate a metric for each value defined in list for each resource.
The value will be 1, if the value matches the one in list.

If path (or valueFrom, if set) does not resolve, for example because the status field is not populated yet,
no metrics are produced for that resource and no error is logged. Once the field is set, the metrics appear.
A value that resolves to a non-string type is reported as an error.

Produces the metric:

prometheus

kube_customresource_status_phase{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", phase="Pending"} 1
kube_customresource_status_phase{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", phase="Bar"} 0
kube_customresource_status_phase{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", phase="Baz"} 0

#### Info

Info metrics are used to expose textual information which SHOULD NOT change during process lifetime. Common examples are an application's version, revision control commit, and the version of a compiler. [[2]](https://github.com/prometheus/OpenMetrics/blob/v1.0.0/specification/OpenMetrics.md#info)

Metrics of type Info will always have a value of 1.

yaml

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: myteam.io
        kind: "Foo"
        version: "v1"
      metrics:
        - name: "version"
          help: "Foo version"
          each:
            type: Info
            info:
              labelsFromPath:
                version: [spec, version]

Produces the metric:

prometheus

kube_customresource_version{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1", version="v1.2.3"} 1

Naming

The default metric names are prefixed to avoid collisions with other metrics.
By default, a metric prefix of kube_ concatenated with your custom resource's group+version+kind is used.
You can override this behavior with the metricNamePrefix field.

yaml

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind: ...
      metricNamePrefix: myteam_foos
      metrics:
        - name: uptime
          # ...

Produces:

prometheus

myteam_foos_uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21

To omit namespace and/or subsystem altogether, set them to the empty string:

yaml

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind: ...
      metricNamePrefix: ""
      metrics:
        - name: uptime
          # ...

Produces:

prometheus

uptime{customresource_group="myteam.io", customresource_kind="Foo", customresource_version="v1"} 43.21

Logging

If a metric path is registered but not found on a custom resource, an error will be logged. For some resources,
this may produce a lot of noise. The error log [verbosity][vlog] for a metric or resource can be set with errorLogV on
the resource or metric:

yaml

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind: ...
      errorLogV: 0  # 0 = default for errors
      metrics:
        - name: uptime
          errorLogV: 10  # only log at high verbosity

[vlog]: https://github.com/go-logr/logr#why-v-levels

Path Syntax

Paths are specified as a list of strings. Each string is a path segment, resolved dynamically against the data of the custom resource.
If any part of a path is missing, the result is nil.

Examples:

yaml

simple path lookup

[spec, replicas]                         # spec.replicas == 1
indexing an array

[spec, order, "0", value]                # spec.order[0].value = true
finding an element in a list by key=value  

[status, conditions, "[name=a]", value]  # status.conditions[0].value = 45
if the value to be matched is a number or boolean, the value is compared as a number or boolean  

[status, conditions, "[value=66]", name]  # status.conditions[1].name = "b"
For generally matching against a field in an object schema, use the following syntax:

[metadata, "name=foo"] # if v, ok := metadata[name]; ok && v == "foo" { return v; } else { / ignore / }

Wildcard matching of version and kind fields

The Custom Resource State (CRS hereon) configuration also allows you to monitor all versions and/or kinds that come under a group. It watches
the installed CRDs for this purpose. Taking the aforementioned Foo object as reference, the configuration below allows
you to monitor all objects under all versions and all kinds that come under the myteam.io group.

yaml

kind: CustomResourceStateMetrics
spec:
  resources:
    - groupVersionKind:
        group: "myteam.io"
        version: "*" # Set to v1 to monitor all kinds under myteam.io/v1. Wildcard matches all installed versions that come under this group.
        kind: "*" # Set to Foo to monitor all Foo objects under the myteam.io group (under all versions). Wildcard matches all installed kinds that come under this group (and version, if specified).
      metrics:
        - name: "myobject_info"
          help: "Foo Bar Baz"
          each:
            type: Info
            info:
              path: [metadata]
              labelsFromPath:
                object: [name]
                namespace: [namespace]

The configuration above produces these metrics.

yaml

kube_customresource_myobject_info{customresource_group="myteam.io",customresource_kind="Foo",customresource_version="v1",namespace="ns",object="foo"} 1
kube_customresource_myobject_info{customresource_group="myteam.io",customresource_kind="Bar",customresource_version="v1",namespace="ns",object="bar"} 1

#### Note

* Only versions that the API server actually serves (served: true in the CRD's spec.versions) are discovered. A version that is still declared in the CRD but no longer served cannot be listed or watched, so the wildcard will not resolve to it, and pinning it explicitly will not generate any metrics. Such skipped versions are logged at verbosity level 1 (-v=1). For cases where the GVKs defined in a CRD have multiple versions under a single group for the same kind, as expected, the wildcard value will resolve to all* versions, but a query for any specific version will return all resources under all versions, in that versions' representation. This basically means that for two such versionsA and B, if a resource exists under B, it will reflect in the metrics generated for A as well, in addition to any resources of itself, and vice-versa. This logic is based on the current list`ing behavior of the client-go library.
* The introduction of this feature further discourages (and discontinues) the use of native objects in the CRS featureset, since these do not have an explicit CRD associated with them, and conflict with internal stores defined specifically for such native resources. Please consider opening an issue or raising a PR if you'd like to expand on the current metric labelsets for them. Also, any such configuration will be ignored, and no metrics will be generated for the same.

---

README Copy Section

Documentation

Table of Contents Copy Chapter

Metrics Stages Copy Chapter

Opt-in Metrics Copy Chapter

Exposed Metrics Copy Chapter

Default Resources Copy Section

Optional Resources Copy Section

PromQL Queries Copy Chapter

Join Metrics Copy Chapter

Metrics from Custom Resources Copy Chapter

CLI Arguments Copy Chapter

Protecting /metrics endpoints Copy Chapter

Design/Metrics Best Practices Copy Section

Kube-State-Metrics - Timeseries best practices

Introduction Copy Chapter

General best practices Copy Chapter

Best practices for kube-state-metrics Copy Chapter

Avoid pre-computation Copy Section

Static object properties Copy Section

Dynamic object properties Copy Section

Linked properties Copy Section

Optional properties Copy Section

Timestamps Copy Section

Cardinality Copy Section

Stability Copy Chapter

Design/Metrics Store Performance Optimization Copy Section

Kube-State-Metrics - Performance Optimization Proposal

Glossary Copy Chapter

Problem Statement Copy Chapter

Goal Copy Chapter

Status Quo Copy Chapter

Proposal Copy Chapter

FAQ / Follow up improvements Copy Chapter

Developer/Cli Arguments Copy Section

Command line arguments

Available options Copy Chapter

Developer/Guide Copy Section

Developer Guide

Table of Contents Copy Chapter

Add New Kubernetes Resource Metric Collector Copy Section

Add New Metrics Copy Section

Metrics/Workload/Cronjob Metrics Copy Section

CronJob Metrics

Metrics/Workload/Daemonset Metrics Copy Section

DaemonSet Metrics

Metrics/Workload/Deployment Metrics Copy Section

Deployment Metrics

Metrics/Workload/Horizontalpodautoscaler Metrics Copy Section

Horizontal Pod Autoscaler Metrics

Metrics/Workload/Job Metrics Copy Section

Job Metrics

Metrics/Workload/Pod Metrics Copy Section

Pod Metrics

Useful metrics queries Copy Chapter

How to retrieve non-standard Pod state Copy Section

Metrics/Workload/Replicaset Metrics Copy Section

ReplicaSet metrics

Metrics/Workload/Replicationcontroller Metrics Copy Section

ReplicationController metrics

Metrics/Workload/Statefulset Metrics Copy Section

Stateful Set Metrics

Metrics/Storage/Configmap Metrics Copy Section

ConfigMap Metrics

Metrics/Storage/Persistentvolume Metrics Copy Section

PersistentVolume Metrics

Useful metrics queries Copy Chapter

How to retrieve non-standard PV state Copy Section

Metrics/Storage/Persistentvolumeclaim Metrics Copy Section

PersistentVolumeClaim Metrics

Useful metrics queries Copy Chapter

How to retrieve non-standard PVC state Copy Section

Metrics/Storage/Secret Metrics Copy Section

Secret Metrics

Metrics/Storage/Storageclass Metrics Copy Section

StorageClass Metrics

Metrics/Storage/Volumeattachment Metrics Copy Section

VolumeAttachment Metrics

Metrics/Service/Endpoint Metrics Copy Section

Endpoint Metrics

README

Table of Contents

Metrics Stages

Opt-in Metrics

Exposed Metrics

Default Resources

Optional Resources

PromQL Queries

Join Metrics

Metrics from Custom Resources

CLI Arguments

Protecting /metrics endpoints

Design/Metrics Best Practices

Introduction

General best practices

Best practices for kube-state-metrics

Avoid pre-computation

Static object properties

Dynamic object properties

Linked properties

Optional properties

Timestamps

Cardinality

Stability

Design/Metrics Store Performance Optimization

Glossary

Problem Statement

Goal

Status Quo

Proposal

FAQ / Follow up improvements

Developer/Cli Arguments

Available options

Developer/Guide

Table of Contents

Add New Kubernetes Resource Metric Collector

Add New Metrics

Metrics/Workload/Cronjob Metrics

Metrics/Workload/Daemonset Metrics

Metrics/Workload/Deployment Metrics

Metrics/Workload/Horizontalpodautoscaler Metrics

Metrics/Workload/Job Metrics

Metrics/Workload/Pod Metrics

Useful metrics queries

How to retrieve non-standard Pod state

Metrics/Workload/Replicaset Metrics

Metrics/Workload/Replicationcontroller Metrics

Metrics/Workload/Statefulset Metrics

Metrics/Storage/Configmap Metrics

Metrics/Storage/Persistentvolume Metrics

Useful metrics queries

How to retrieve non-standard PV state

Metrics/Storage/Persistentvolumeclaim Metrics

Useful metrics queries

How to retrieve non-standard PVC state

Metrics/Storage/Secret Metrics

Metrics/Storage/Storageclass Metrics

Metrics/Storage/Volumeattachment Metrics

Metrics/Service/Endpoint Metrics

Metrics/Service/Endpointslice Metrics

Metrics/Service/Ingress Metrics

Metrics/Service/Ingressclass Metrics

Metrics/Service/Service Metrics

Metrics/Policy/Limitrange Metrics

Metrics/Policy/Networkpolicy Metrics

Metrics/Policy/Poddisruptionbudget Metrics

Metrics/Policy/Resourcequota Metrics

Metrics/Extend/Customresourcestate Metrics

Configuration

RBAC-enabled Clusters

Examples

Metric types

Naming

Logging

Path Syntax

Wildcard matching of version and kind fields