kube-prometheus Documentation

Technical Documentation: prometheus-operator/kube-prometheus

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

kube-prometheus

[!WARNING]
Everything is experimental and may change significantly at any time.

This repository collects Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

The content of this project is written in jsonnet. This project could both be described as a package as well as a library.

Components included in this package:

• The Prometheus Operator
• Highly available Prometheus
• Highly available Alertmanager
• Prometheus node-exporter
• Prometheus blackbox-exporter
• Prometheus Adapter for Kubernetes Metrics APIs
• kube-state-metrics
• Grafana

Optional addon:

• Perses is CNCF Sandbox alternative to Grafana (see Using Perses instead of Grafana)

This stack is meant for cluster monitoring, so it is pre-configured to collect metrics from all Kubernetes components. In addition to that it delivers a default set of dashboards and alerting rules. Many of the useful dashboards and alerts come from the kubernetes-mixin project, similar to this project it provides composable jsonnet as a library for users to customize to their needs.

Prerequisites Copy Chapter

You will need a Kubernetes cluster, that's it! By default it is assumed, that the kubelet uses token authentication and authorization, as otherwise Prometheus needs a client certificate, which gives it full access to the kubelet, rather than just the metrics. Token authentication and authorization allows more fine grained and easier access control.

This means the kubelet configuration must contain these flags:

• --authentication-token-webhook=true This flag enables, that a ServiceAccount token can be used to authenticate against the kubelet(s). This can also be enabled by setting the kubelet configuration value authentication.webhook.enabled to true.
• --authorization-mode=Webhook This flag enables, that the kubelet will perform an RBAC request with the API to determine, whether the requesting entity (Prometheus in this case) is allowed to access a resource, in specific for this project the /metrics endpoint. This can also be enabled by setting the kubelet configuration value authorization.mode to Webhook.

This stack provides resource metrics by deploying
the Prometheus Adapter.
This adapter is an Extension API Server and Kubernetes needs to be have this feature enabled, otherwise the adapter has
no effect, but is still deployed.

Compatibility Copy Chapter

The following Kubernetes versions are supported and work as we test against these versions in their respective branches. But note that other versions might work!

[!NOTE]
In CI we will be testing only last two releases and main branch on a regular basis.

Quickstart Copy Chapter

This project is intended to be used as a library (i.e. the intent is not for you to create your own modified copy of this repository).

Though for a quickstart a compiled version of the Kubernetes manifests generated with this library (specifically with example.jsonnet) is checked into this repository in order to try the content out quickly. To try out the stack un-customized run:

• Create the monitoring stack using the config in the manifests directory:

  shell Copy

  # Create the namespace and CRDs, and then wait for them to be available before creating the remaining resources
  # Note that due to some CRD size we are using kubectl server-side apply feature which is generally available since kubernetes 1.22.
  # If you are using previous kubernetes versions this feature may not be available and you would need to use kubectl create instead.
  kubectl apply --server-side -f manifests/setup
  kubectl wait \
      --for condition=Established \
      --all CustomResourceDefinition \
      --namespace=monitoring
  kubectl apply -f manifests/

We create the namespace and CustomResourceDefinitions first to avoid race conditions when deploying the monitoring components.
Alternatively, the resources in both folders can be applied with a single command
kubectl apply --server-side -f manifests/setup -f manifests, but it may be necessary to run the command multiple times for all components to
be created successfully.

• To teardown the stack:

  shell Copy

  kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup

The official documentation contains the full version of this quick-start guide, and includes instructions on how to access Prometheus, AlertManager, and Grafana.

minikube Copy Section

To try out this stack, start minikube with the following command:

minikube delete && minikube start --container-runtime=containerd --kubernetes-version=v1.33.1 --memory=6g --bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.bind-address=0.0.0.0 --extra-config=controller-manager.bind-address=0.0.0.0

The kube-prometheus stack includes a resource metrics API server (prometheus-adapter by default), so the metrics-server addon is not necessary. Ensure the metrics-server addon is disabled on minikube:

minikube addons disable metrics-server

[!NOTE]
If you prefer to use the lightweight upstream metrics-server instead of prometheus-adapter, see Using metrics-server instead of prometheus-adapter.

Getting started Copy Chapter

Before deploying kube-prometheus in a production environment, read:

1. Customizing kube-prometheus
2. Customization examples
3. Accessing Graphical User Interfaces
4. Troubleshooting kube-prometheus

Documentation Copy Chapter

1. Continuous Delivery
2. Update to new version
3. For more documentation on the project refer to docs/ directory.

Contributing Copy Chapter

To contribute to kube-prometheus, refer to Contributing.

Join the discussion Copy Chapter

If you have any questions or feedback regarding kube-prometheus, join the kube-prometheus discussion. Alternatively, consider joining #prometheus-operator slack channel or project's bi-weekly Contributor Office Hours.

License Copy Chapter

Apache License 2.0, see LICENSE.

Overview (`developer-workspace/README.md`) Copy Section

Ephemeral developer workspaces

Aiming to provide better developer experience when making contributions to kube-prometheus, whether by actively developing new features/bug fixes or by reviewing pull requests, we want to provide ephemeral developer workspaces with everything already configured (as far as tooling makes it possible).

A developer workspace provides a brand new Kubernetes cluster, where kube-prometheus can be easily deployed and the contributor can easily see the impact that a pull request is proposing.

Today only Github Codespaces is supported. Unfortunately, Codespaces is not available for everyone. If you are fortunate to have access to it, you can open a new workspace from a specific branch, or even from Pull Requests.

After your workspace start, you can deploy a kube-prometheus inside a Kind cluster inside by running make deploy.

If you are reviewing a PR, you'll have a fully-functional kubernetes cluster, generating real monitoring data that can be used to review if the proposed changes works as described.

If you are working on new features/bug fixes, you can regenerate kube-prometheus's YAML manifests with make generate and deploy it again with make deploy.

Overview (`docs/migration-example/readme.md`) Copy Section

Example of conversion of a legacy my.jsonnet file Copy Chapter

An example conversion of a legacy custom jsonnet file to release-0.8
format can be seen by viewing and comparing this
release-0.3 jsonnet file (when the github
repo was under https://github.com/coreos/kube-prometheus...)
and the corresponding release-0.8 jsonnet file.

These two files have had necessary blank lines added so that they
can be compared side-by-side and line-by-line on screen.

The conversion covers both the change of stopping using ksonnet after
release-0.3 and also the major migration after release-0.7 as described in
migration-guide.md

The sample files are intended as an example of format conversion and
not necessarily best practice for the files in release-0.3 or release-0.8.

Below are three sample extracts of the conversion as an indication of the
changes required.

{
_config+:: {
// Override namespace
namespace: 'monitoring',

|jsonnet
local kp =
(import 'kube-prometheus/main.libsonnet') +
// kubeadm now achieved by setting platform value - see 9 lines below
(import 'kube-prometheus/addons/static-etcd.libsonnet') +
(import 'kube-prometheus/addons/podsecuritypolicies.libsonnet') +
{
values+:: {
common+: {
namespace: 'monitoring',
},

// Add kubeadm platform-specific items,
  // including kube-contoller-manager and kube-scheduler discovery
  kubePrometheus+: {
    platform: 'kubeadm',
  },

| release-0.3 | release-0.8 |
| --- | --- |
| ```jsonnet
    // Add additional ingresses
    // See https://github.com/coreos/kube-prometheus/...
    //           tree/master/examples/ingress.jsonnet
    ingress+:: {
      alertmanager:
        ingress.new() +


        ingress.mixin.metadata.withName('alertmanager') +
        ingress.mixin.metadata.withNamespace($._config.namespace) +
        ingress.mixin.metadata.withAnnotations({
          'kubernetes.io/ingress.class': 'nginx-api',
        }) +

        ingress.mixin.spec.withRules(
          ingressRule.new() +
          ingressRule.withHost(alert_manager_host) +
          ingressRule.mixin.http.withPaths(
            ingressRuleHttpPath.new() +




            ingressRuleHttpPath.mixin.backend
                               .withServiceName('alertmanager-operated') +
            ingressRuleHttpPath.mixin.backend.withServicePort(9093)
          ),
        ) +
        // Note we do not need a TLS secretName here as we are going to use the
        // nginx-ingress default secret which is a wildcard
        // secretName would need to be in the same namespace at this time,
        // see https://github.com/kubernetes/ingress-nginx/issues/2371
        ingress.mixin.spec.withTls(
          ingressTls.new() +
          ingressTls.withHosts(alert_manager_host)
        ),
  
  
``` | ```jsonnet
    // Add additional ingresses
    // See https://github.com/prometheus-operator/kube-prometheus/...
    //           blob/main/examples/ingress.jsonnet
    ingress+:: {
      alertmanager: {
        apiVersion: 'networking.k8s.io/v1',
        kind: 'Ingress',
        metadata: {
          name: 'alertmanager',
          namespace: $.values.common.namespace,
          annotations: {
            'kubernetes.io/ingress.class': 'nginx-api',
          },
        },
        spec: {
          rules: [{
            host: alert_manager_host,
            http: {
              paths: [{
                path: '/',
                pathType: 'Prefix',
                backend: {
                  service: {
                    name: 'alertmanager-operated',
                    port: {
                      number: 9093,
                    },
                  },
                },
              }],
            },
          }],
          tls: [{

            hosts: [alert_manager_host],
          }],
        },
      },
``` |




| release-0.3 | release-0.8 |
| --- | --- |
| ```jsonnet
    // Additional prometheus rules
    // See https://github.com/coreos/kube-prometheus/docs/...
    //           developing-prometheus-rules-and-grafana-dashboards.md
    //
    // cat my-prometheus-rules.yaml | \
    //   gojsontoyaml -yamltojson | \
    //   jq . > my-prometheus-rules.json
    prometheusRules+:: {














      groups+: import 'my-prometheus-rules.json',


    },
  };
  
  
  
  
``` | ```jsonnet
    // Additional prometheus rules
    // See https://github.com/prometheus-operator/kube-prometheus/blob/main/...
    //           docs/developing-prometheus-rules-and-grafana-dashboards.md...
    //           #pre-rendered-rules
    // cat my-prometheus-rules.yaml | \
    //   gojsontoyaml -yamltojson | \
    //   jq . > my-prometheus-rules.json
    prometheusMe: {
      rules: {
        apiVersion: 'monitoring.coreos.com/v1',
        kind: 'PrometheusRule',
        metadata: {
          name: 'my-prometheus-rule',
          namespace: $.values.common.namespace,
          labels: {
            'app.kubernetes.io/name': 'kube-prometheus',
            'app.kubernetes.io/part-of': 'kube-prometheus',
            prometheus: 'k8s',
            role: 'alert-rules',
          },
        },
        spec: {
          groups: import 'my-prometheus-rules.json',
        },
      },
    },
  };

...

+ { ['prometheus-my-' + name]: kp.prometheusMe[name] for name in std.objectFields(kp.prometheusMe) }
``` |



---

### Overview (`examples/continuous-delivery/argocd/README.md`)

## ArgoCD Example

This is the simplest, working example of an argocd app, the JSON object built is now an array of objects as that is the prefered format for ArgoCD. And ArgoCD specific annotations are added to manifests.

Requirements:

- **ArgoCD 1.7+**

- Follow the vendor generation steps at the root of this repository and generate a `vendored` folder (referenced in `application.yaml`).

- Make sure that argocd-cm has `application.instanceLabelKey` set to something else than `app.kubernetes.io/instance`, otherwise it will cause problems with prometheus target discovery. (see also [Why Is My App Out Of Sync Even After Syncing?](https://argo-cd.readthedocs.io/en/stable/faq/#why-is-my-app-out-of-sync-even-after-syncing))

---

### Overview (`jsonnet/kube-prometheus/platforms/README.md`)

# Adding a new platform specific configuration

Adding a new platform specific configuration requires to update the [customization example](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/platform-specific.md) and the [platforms.libsonnet](platforms.libsonnet) file by adding the platform to the list of existing ones. This allow the new platform to be discoverable and easily configurable by the users.

---

### EKS Cni Support (`docs/EKS-cni-support.md`)

# CNI monitoring special configuration updates for EKS

AWS EKS uses [CNI](https://github.com/aws/amazon-vpc-cni-k8s) networking plugin for pod networking in Kubernetes using Elastic Network Interfaces on AWS

One fatal issue that can occur is that you run out of IP addresses in your eks cluster. (Generally happens due to error configs where pods keep scheduling).

You can monitor the `awscni` using kube-promethus with :

```jsonnet mdox-exec="cat examples/eks-cni-example.jsonnet"
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
    kubePrometheus+: {
      platform: 'eks',
    },
  },
  kubernetesControlPlane+: {
    prometheusRuleEksCNI+: {
      spec+: {
        groups+: [
          {
            name: 'example-group',
            rules: [
              {
                record: 'aws_eks_available_ip',
                expr: 'sum by(instance) (awscni_total_ip_addresses) - sum by(instance) (awscni_assigned_ip_addresses) < 10',
              },
            ],
          },
        ],
      },
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) }

After you have the required yaml file please run

kubectl apply -f manifests/prometheus-serviceMonitorAwsEksCNI.yaml

GKE Cadvisor Support (`docs/GKE-cadvisor-support.md`) Copy Section

Kubelet / cAdvisor special configuration updates for GKE

Prior to GKE 1.11, the kubelet does not support token
authentication. Until it does, Prometheus must use HTTP (not HTTPS)
for scraping.

You can configure this behavior through kube-prometheus with:

local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') +
    (import 'kube-prometheus/kube-prometheus-insecure-kubelet.libsonnet') +
	{
        _config+:: {
		# ... config here
		}
    };

Or, you can patch and re-apply your existing manifests with:

On linux:

sed -i -e 's/https/http/g' manifests/prometheus-serviceMonitorKubelet.yaml

On MacOs:

sed -i '' -e 's/https/http/g' manifests/prometheus-serviceMonitorKubelet.yaml

After you have modified the yaml file please run

kubectl apply -f manifests/prometheus-serviceMonitorKubelet.yaml

Access Ui (`docs/access-ui.md`) Copy Section

weight: 300
toc: true
title: Access Dashboards
menu:
docs:
parent: kube
images: []

draft: false Copy Chapter

Prometheus, Grafana, and Alertmanager dashboards can be accessed quickly using kubectl port-forward after running the quickstart via the commands below.

Kubernetes 1.10 or later is required.

You can also learn how to expose Prometheus/Alertmanager/Grafana via Ingress

Prometheus Copy Chapter

kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090

Open Prometheus on http://localhost:9090 in your browser.

Check out the alerts and rules pages with the pre-configured rules and alerts!
This Prometheus is supposed to monitor your Kubernetes cluster and make sure to alert you if there’s a problem with it.

For your own applications we recommend running one or more other instances.

Grafana Copy Chapter

kubectl --namespace monitoring port-forward svc/grafana 3000

Open Grafana on localhost:3000 in your browser.
You can login with the username admin and password admin.

Perses (alternative to Grafana) Copy Chapter

If you have enabled the Perses addon instead of Grafana:

kubectl --namespace monitoring port-forward svc/perses 8080

Open Perses on localhost:8080 in your browser. The bundled community-mixins dashboards for the default kube-prometheus stack are loaded from PersesDashboard CRs, and a Prometheus datasource is auto-configured. See dashboard coverage for details.

Alertmanager Copy Chapter

kubectl --namespace monitoring port-forward svc/alertmanager-main 9093

Open Alertmanager on localhost:9093 in your browser.

Blackbox Exporter (`docs/blackbox-exporter.md`) Copy Section

weight: 304
toc: true
title: Blackbox Exporter
menu:
docs:
parent: kube
lead: This guide will help you deploying the blackbox-exporter with the Probe custom resource definition.
images: []
draft: false

date: "2021-03-08T08:49:31+00:00" Copy Chapter

Setting up a blackbox exporter

The prometheus-operator defines a Probe resource type that can be used to describe blackbox checks. To execute these, a separate component called blackbox_exporter has to be deployed, which can be scraped to retrieve the results of these checks. You can use kube-prometheus to set up such a blackbox exporter within your Kubernetes cluster.

Adding blackbox exporter manifests to an existing `kube-prometheus` configuration Copy Chapter

1. Override blackbox-related configuration parameters as needed.
2. Add the following to the list of renderers to render the blackbox exporter manifests:

{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) }

Configuration parameters influencing the blackbox exporter Copy Chapter

• _config.namespace: the namespace where the various generated resources (ConfigMap, Deployment, Service, ServiceAccount and ServiceMonitor) will reside. This does not affect where you can place Probe objects; that is determined by the configuration of the Prometheus resource. This option is shared with other kube-prometheus components; defaults to default.
• _config.imageRepos.blackboxExporter: the name of the blackbox exporter image to deploy. Defaults to quay.io/prometheus/blackbox-exporter.
• _config.versions.blackboxExporter: the tag of the blackbox exporter image to deploy. Defaults to the version kube-prometheus was tested with.
• _config.imageRepos.configmapReloader: the name of the ConfigMap reloader image to deploy. Defaults to ghcr.io/jimmidyson/configmap-reload.
• _config.versions.configmapReloader: the tag of the ConfigMap reloader image to deploy. Defaults to the version kube-prometheus was tested with.
• _config.resources.blackbox-exporter.requests: the requested resources; this is used for each container. Defaults to 10m CPU and 20Mi RAM. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for details.
• _config.resources.blackbox-exporter.limits: the resource limits; this is used for each container. Defaults to 20m CPU and 40Mi RAM. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for details.
• _config.blackboxExporter.port: the exposed HTTPS port of the exporter. This is what Prometheus can scrape for metrics related to the blackbox exporter itself. Defaults to 9115.
• _config.blackboxExporter.internalPort: the internal plaintext port of the exporter. Prometheus scrapes configured via Probe objects cannot access the HTTPS port right now, so you have to specify this port in the url field. Defaults to 19115.
• _config.blackboxExporter.replicas: the number of exporter replicas to be deployed. Defaults to 1.
• _config.blackboxExporter.matchLabels: map of the labels to be used to select resources belonging to the instance deployed. Defaults to { 'app.kubernetes.io/name': 'blackbox-exporter' }
• _config.blackboxExporter.assignLabels: map of the labels applied to components of the instance deployed. Defaults to all the labels included in the matchLabels option, and additionally app.kubernetes.io/version is set to the version of the blackbox exporter.
• _config.blackboxExporter.modules: the modules available in the blackbox exporter installation, i.e. the types of checks it can perform. The default value includes most of the modules defined in the default blackbox exporter configuration: http_2xx, http_post_2xx, tcp_connect, pop3s_banner, ssh_banner, and irc_banner. icmp is omitted so the exporter can be run with minimum privileges, but you can add it back if needed - see the example below. See https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md for the configuration format, except you have to use JSON instead of YAML here.
• _config.blackboxExporter.privileged: whether the blackbox-exporter container should be running as non-root (false) or root with heavily-restricted capability set (true). Defaults to true if you have any ICMP modules defined (which need the extra permissions) and false otherwise.

Complete example Copy Chapter

local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
      blackboxExporter+:: {
        modules+:: {
          icmp: {
            prober: 'icmp',
          },
        },
      },
    },
  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

After installing the generated manifests, you can create Probe resources, for example:

kind: Probe
apiVersion: monitoring.coreos.com/v1
metadata:
  name: example-com-website
  namespace: monitoring
spec:
  interval: 60s
  module: http_2xx
  prober:
    url: blackbox-exporter.monitoring.svc.cluster.local:19115
  targets:
    staticConfig:
      static:
      - http://example.com
      - https://example.com

Community Support (`docs/community-support.md`) Copy Section

Community support

For bugs, you can use the GitHub issue tracker.

For questions, you can use the GitHub discussions forum.

Many of the kube-prometheus project's contributors and users can also be found on the #prometheus-operator channel of the Kubernetes Slack.

kube-prometheus is the aggregation of many projects that all have different
channels to reach out for help and support. This community strives at
supporting all users and you should never be afraid of asking us first. However
if your request relates specifically to one of the projects listed below, it is
often more efficient to reach out to the project directly. If you are unsure,
please feel free to open an issue in this repository and we will redirect you
if applicable.

prometheus-operator Copy Chapter

For documentation, check the project's documentation directory.

For questions, use the #prometheus-operator channel on the Kubernetes Slack.

For bugs, use the GitHub issue tracker.

Prometheus, Alertmanager, node_exporter Copy Chapter

For documentation, check the Prometheus online docs. There is a
section with links to blog
posts, recorded talks and presentations. This repository
(not affiliated to the Prometheus project) has also a list of curated resources
related to the Prometheus ecosystem.

For questions, see the Prometheus community page for the various channels.

There is also a #prometheus channel on the CNCF Slack.

kube-state-metrics Copy Chapter

For documentation, see the project's docs directory.

For questions, use the #kube-state-metrics channel on the Kubernetes Slack.

For bugs, use the GitHub issue tracker.

Kubernetes Copy Chapter

For documentation, check the Kubernetes docs.

For questions, use the community forums and the Kubernetes Slack. Check also the community page.

For bugs, use the GitHub issue tracker.

Prometheus adapter Copy Chapter

For documentation, check the project's README.

For questions, use the #sig-instrumentation channel on the Kubernetes Slack.

For bugs, use the GitHub issue tracker.

Grafana Copy Chapter

For documentation, check the Grafana docs.

For questions, use the community forums.

For bugs, use the GitHub issue tracker.

kubernetes-mixin Copy Chapter

For documentation, check the project's README.

For questions, use #monitoring-mixins channel on the Kubernetes Slack.

For bugs, use the GitHub issue tracker.

Jsonnet Copy Chapter

For documentation, check the Jsonnet website.

For questions, use the mailing list.

Customizing (`docs/customizing.md`) Copy Section

Customizing Kube-Prometheus

This section:

• describes how to customize the kube-prometheus library via compiling the kube-prometheus manifests yourself (as an alternative to the README.md quickstart section).
• still doesn't require you to make a copy of this entire repository, but rather only a copy of a few select files.

Installing Copy Chapter

The content of this project consists of a set of jsonnet files making up a library to be consumed.

Install this library in your own project with jsonnet-bundler (the jsonnet package manager):

$ mkdir my-kube-prometheus; cd my-kube-prometheus
$ jb init  # Creates the initial/empty `jsonnetfile.json`
# Install the kube-prometheus dependency
$ jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main # Creates `vendor/` & `jsonnetfile.lock.json`, and fills in `jsonnetfile.json`

$ wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/example.jsonnet -O example.jsonnet
$ wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/build.sh -O build.sh
$ chmod +x build.sh

jb can be installed with go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest

An e.g. of how to install a given version of this library: jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main

In order to update the kube-prometheus dependency, simply use the jsonnet-bundler update functionality:

$ jb update

Generating Copy Chapter

e.g. of how to compile the manifests: ./build.sh example.jsonnet

before compiling, install gojsontoyaml tool with go install github.com/brancz/gojsontoyaml@latest and jsonnet with go install github.com/google/go-jsonnet/cmd/jsonnet@latest

Here's example.jsonnet:

Note: some of the following components must be configured beforehand. See configuration and customization-examples.

/* Detailed source-code truncated for AI context efficiency. */

And here's the build.sh script (which uses vendor/ to render all manifests in a json structure of {filename: manifest-content}):

#!/usr/bin/env bash

# This script uses arg $1 (name of *.jsonnet file to use) to generate the manifests/*.yaml files.

set -e
set -x
# only exit with zero if all commands of the pipeline exit successfully
set -o pipefail

# Make sure to use project tooling
PATH="$(pwd)/tmp/bin:${PATH}"

# Make sure to start with a clean 'manifests' dir
rm -rf manifests
mkdir -p manifests/setup

# Calling gojsontoyaml is optional, but we would like to generate yaml, not json
jsonnet -J vendor -m manifests "${1-example.jsonnet}" | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml' -- {}

# Make sure to remove json files
find manifests -type f ! -name '*.yaml' -delete
mv manifests/kustomization.yaml .

Note you need jsonnet (go install github.com/google/go-jsonnet/cmd/jsonnet@latest) and gojsontoyaml (go install github.com/brancz/gojsontoyaml@latest) installed to run build.sh. If you just want json output, not yaml, then you can skip the pipe and everything afterwards.

This script runs the jsonnet code, then reads each key of the generated json and uses that as the file name, and writes the value of that key to that file, and converts each json manifest to yaml.

Configuring Copy Chapter

Jsonnet has the concept of hidden fields. These are fields, that are not going to be rendered in a result. This is used to configure the kube-prometheus components in jsonnet. In the example jsonnet code of the above Generating section, you can see an example of this, where the namespace is being configured to be monitoring. In order to not override the whole object, use the +:: construct of jsonnet, to merge objects, this way you can override individual settings, but retain all other settings and defaults.

The available fields and their default values can be seen in main.libsonnet. Note that many of the fields get their default values from variables, and for example the version numbers are imported from versions.json.

Configuration is mainly done in the values map. You can see this being used in the example.jsonnet to set the namespace to monitoring. This is done in the common field, which all other components take their default value from. See for example how Alertmanager is configured in main.libsonnet:

alertmanager: {
      name: 'main',
      // Use the namespace specified under values.common by default.
      namespace: $.values.common.namespace,
      version: $.values.common.versions.alertmanager,
      image: $.values.common.images.alertmanager,
      mixin+: { ruleLabels: $.values.common.ruleLabels },
    },

The grafana definition is located in a different project (https://github.com/brancz/kubernetes-grafana ), but needed configuration can be customized from the same top level values field. For example to allow anonymous access to grafana, add the following values section:

grafana+:: {
        config: { // http://docs.grafana.org/installation/configuration/
          sections: {
            "auth.anonymous": {enabled: true},
          },
        },
      },

Apply the kube-prometheus stack Copy Chapter

The previous generation step has created a bunch of manifest files in the manifest/ folder.
Now simply use kubectl to install Prometheus and Grafana as per your configuration:

# Update the namespace and CRDs, and then wait for them to be available before creating the remaining resources
$ kubectl apply --server-side -f manifests/setup
$ kubectl apply -f manifests/

Note that due to some CRD size we are using kubectl server-side apply feature which is generally available since
kubernetes 1.22. If you are using previous kubernetes versions this feature may not be available and you would need to
use kubectl create instead.

Alternatively, the resources in both folders can be applied with a single command
kubectl apply --server-side -Rf manifests, but it may be necessary to run the command multiple times for all components to
be created successfully.

Check the monitoring namespace (or the namespace you have specific in namespace: ) and make sure the pods are running. Prometheus and Grafana should be up and running soon.

Minikube Example Copy Chapter

To use an easy to reproduce example, see minikube.jsonnet, which uses the minikube setup as demonstrated in Prerequisites. Because we would like easy access to our Prometheus, Alertmanager and Grafana UIs, minikube.jsonnet exposes the services as NodePort type services.

Deploy Kind (`docs/deploy-kind.md`) Copy Section

weight: 301
toc: true
title: Deploy to kind
menu:
docs:
parent: kube
lead: This guide will help you deploying kube-prometheus on Kubernetes kind.
images: []

draft: false Copy Chapter

Time to explain how!

Your chance of contributing!

Endpoints Migration (`docs/endpoints-migration.md`) Copy Section

Migration from Endpoints to EndpointSlice

kube-prometheus 0.17+ automatically configures Prometheus to use EndpointSlice instead of Endpoints for Kubernetes service discovery (Endpoints have been deprecated in Kubernetes 1.33).

While the migration should be seamless for "regular" pods, it requires a few manual steps for components running as host services (e.g. node_exporter and kubelet):

1. The node_exporter and kubelet ServiceMonitors rely on the Prometheus operator's kubelet controller which manages the kube-system/kubelet Service.
2. With kube-prometheus 0.17, the Prometheus operator starts with both --kubelet-endpoints=true and --kubelet-endpointslice=true to ensure that a) the operator synchronizes the EndpointSlice object(s) backing the kube-system/kubelet Service and b) Kubernetes stops mirroring the kube-system/kubelet Endpoints object to EndpointSlice object(s) (otherwise the operator and kube-controller-manager would fight for the same resources).
3. After verifying that all targets are correctly discovered, it is ok to modify the operator's deployment and use --kubelet-endpoints=false instead. This will become the default in a future version of kube-prometheus.
4. The kube-system/kubelet Endpoints object should be removed manually.

To verify the status of the Endpoints and EndpointSlice objects, run:

kubectl get -n kube-system endpoints kubelet
kubectl get -n kube-system endpointslice -l endpointslice.kubernetes.io/managed-by=prometheus-operator

Kube Prometheus On Kubeadm (`docs/kube-prometheus-on-kubeadm.md`) Copy Section

weight: 302
toc: true
title: Deploy to kubeadm
menu:
docs:
parent: kube
lead: This guide will help you deploying kube-prometheus on Kubernetes kubeadm.
images: []

draft: false Copy Chapter

The kubeadm tool is linked by Kubernetes as the offical way to deploy and manage self-hosted clusters. kubeadm does a lot of heavy lifting by automatically configuring your Kubernetes cluster with some common options. This guide is intended to show you how to deploy Prometheus, Prometheus Operator and Kube Prometheus to get you started monitoring your cluster that was deployed with kubeadm.

This guide assumes you have a basic understanding of how to use the functionality the Prometheus Operator implements. If you haven't yet, we recommend reading through the getting started guide as well as the alerting guide.

kubeadm Pre-requisites Copy Chapter

This guide assumes you have some familiarity with kubeadm or at least have deployed a cluster using kubeadm. By default, kubeadm does not expose two of the services that we will be monitoring. Therefore, in order to get the most out of the kube-prometheus package, we need to make some quick tweaks to the Kubernetes cluster. Since we will be monitoring the kube-controller-manager and kube-scheduler, we must expose them to the cluster.

By default, kubeadm runs these pods on your master and bound to 127.0.0.1. There are a couple of ways to change this. The recommended way to change these features is to use the kubeadm config file. An example configuration file can be used:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
controlPlaneEndpoint: "192.168.1.173:6443"
apiServer:
  extraArgs:
    authorization-mode: "Node,RBAC"
controllerManager:
  extraArgs:
    bind-address: "0.0.0.0"
scheduler:
  extraArgs:
    bind-address: "0.0.0.0"
certificatesDir: "/etc/kubernetes/pki"
etcd:
  # one of local or external
  local:
    dataDir: "/var/lib/etcd"
kubernetesVersion: "v1.23.1"
networking:
  dnsDomain: "cluster.local"
  serviceSubnet: "10.96.0.0/12"
imageRepository: "registry.k8s.io"

Notice the .scheduler.extraArgs and .controllerManager.extraArgs. This exposes the kube-controller-manager and kube-scheduler services to the rest of the cluster. If you have kubernetes core components as pods in the kube-system namespace, ensure that the kube-prometheus-exporter-kube-scheduler and kube-prometheus-exporter-kube-controller-manager services' spec.selector values match those of pods.

In previous versions of Kubernetes, we had to make a change to the kubelet setting with regard to cAdvisor monitoring on the control-plane as well as all the nodes. But this is no longer required due to the change of Kubernetes.

In case you already have a Kubernetes deployed with kubeadm, change the address kube-controller-manager and kube-scheduler listens in addition to previous kubelet change:

sed -e "s/- --bind-address=127.0.0.1/- --bind-address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-controller-manager.yaml
sed -e "s/- --bind-address=127.0.0.1/- --bind-address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-scheduler.yaml

With these changes, your Kubernetes cluster is ready.

Metric Sources Copy Chapter

Monitoring a Kubernetes cluster with Prometheus is a natural choice as Kubernetes components themselves are instrumented with Prometheus metrics, therefore those components simply have to be discovered by Prometheus and most of the cluster is monitored.

Metrics that are rather about cluster state than a single component's metrics is exposed by the add-on component kube-state-metrics.

Additionally, to have an overview of cluster nodes' resources the Prometheus node_exporter is used. The node_exporter allows monitoring a node's resources: CPU, memory and disk utilization and more.

Once you complete this guide you will monitor the following:

• cluster state via kube-state-metrics
• nodes via the node_exporter
• kubelets
• apiserver
• kube-scheduler
• kube-controller-manager

Getting Up and Running Fast with Kube-Prometheus Copy Chapter

To help get started more quickly with monitoring Kubernetes clusters, kube-prometheus was created. It is a collection of manifests including dashboards and alerting rules that can easily be deployed. It utilizes the Prometheus Operator and all the manifests demonstrated in this guide.

This section represent a quick installation and is not intended to teach you about all the components. The easiest way to get started is to clone this repository and use the kube-prometheus section of the code.

git clone https://github.com/prometheus-operator/kube-prometheus
cd kube-prometheus/

First, create the namespace in which you want the monitoring tool suite to be running.

export NAMESPACE='monitoring'
kubectl create namespace "$NAMESPACE"

Now we will create the components for the Prometheus operator

kubectl --namespace="$NAMESPACE" apply -f manifests/prometheus-operator

This will create all the Prometheus Operator components. You might need to wait a short amount of time before the Custom Resource Definitions are available in the cluster. You can wait for them:

until kubectl --namespace="$NAMESPACE" get alertmanagers.monitoring.coreos.com > /dev/null 2>&1; do sleep 1; printf "."; done

Next, we will install the node exporter and then kube-state-metrics:

kubectl --namespace="$NAMESPACE" apply -f manifests/node-exporter
kubectl --namespace="$NAMESPACE" apply -f manifests/kube-state-metrics

Then, we can deploy the grafana credentials. By default, the username/password will be admin/admin, you should change these for your production clusters.

kubectl --namespace="$NAMESPACE" apply -f manifests/grafana/grafana-credentials.yaml

Then install grafana itself:

kubectl --namespace="$NAMESPACE" apply -f manifests/grafana

Next up is the Prometheus object itself. We will deploy the application, and then the roles/role-bindings.

find manifests/prometheus -type f ! -name prometheus-k8s-roles.yaml ! -name prometheus-k8s-role-bindings.yaml -exec kubectl --namespace "$NAMESPACE" apply -f {} \;
kubectl apply -f manifests/prometheus/prometheus-k8s-roles.yaml
kubectl apply -f manifests/prometheus/prometheus-k8s-role-bindings.yaml

Finally, install the Alertmanager

kubectl --namespace="$NAMESPACE" apply -f manifests/alertmanager

Now you should have a working cluster. After all the pods are ready, you should be able to reach:

• Prometheus UI on node port 30900
• Alertmanager UI on node port 30903
• Grafana on node port 30902

These can of course be changed via the Service definitions. It is recommended to look at the Exposing Prometheus and Alert Manager documentation for more detailed information on how to expose these services.

Migration Guide (`docs/migration-guide.md`) Copy Section

Migration guide from release-0.7 and earlier

Why? Copy Chapter

Thanks to our community we identified a lot of short-commings of previous design, varying from issues with global state to UX problems. Hoping to fix at least part of those issues we decided to do a complete refactor of the codebase.

Overview Copy Chapter

Breaking Changes Copy Section

• global _config object is removed and the new values object is a partial replacement
• imageRepos field was removed and the project no longer tries to compose image strings. Use $.values.common.images to override default images.
• prometheus alerting and recording rules are split into multiple PrometheusRule objects
• kubernetes control plane ServiceMonitors and Services are now part of the new kubernetesControlPlane top-level object instead of prometheus object
• jsonnet/kube-prometheus/kube-prometheus.libsonnet file was renamed to jsonnet/kube-prometheus/main.libsonnet and slimmed down to bare minimum
• jsonnet/kube-prometheus/kube-prometheus*-.libsonnet files were move either to jsonnet/kube-prometheus/addons/ or jsonnet/kube-prometheus/platforms/ depending on the feature they provided
• all component libraries are now function- and not object-based
• monitoring-mixins are included inside each component and not globally. prometheusRules, prometheusAlerts, and grafanaDashboards are accessible only per component via mixin object (ex. $.alertmanager.mixin.prometheusAlerts)
• default repository branch changed from master to main
• labels on resources have changes, kubectl apply will not work correctly due to those field being immutable. Deleting the resource first before applying is a workaround if you are using the kubectl CLI. (This only applies to Deployments and DaemonSets.)

New Features Copy Section

• concept of addons, components, and platforms was introduced
• all main components are now represented internally by a function with default values and required parameters (see #Component-configuration for more information)
• $.values holds main configuration parameters and should be used to set basic stack configuration.
• common parameters across all components are stored now in $.values.common
• removed dependency on deprecated ksonnet library

Details Copy Chapter

Components, Addons, Platforms Copy Section

Those concepts were already present in the repository but it wasn't clear which file is holding what. After refactoring we categorized jsonnet code into 3 buckets and put them into separate directories:

• components - main building blocks for kube-prometheus, written as functions responsible for creating multiple objects representing kubernetes manifests. For example all objects for node_exporter deployment are bundled in components/node_exporter.libsonnet library
• addons - everything that can enhance kube-prometheus deployment. Those are small snippets of code adding a small feature, for example adding anti-affinity to pods via addons/anti-affinity.libsonnet. Addons are meant to be used in object-oriented way like local kp = (import 'kube-prometheus/main.libsonnet') + (import 'kube-prometheus/addons/all-namespaces.libsonnet')
• platforms - currently those are addons specialized to allow deploying kube-prometheus project on a specific platform.

Component configuration Copy Section

Refactoring main components to use functions allowed us to define APIs for said components. Each function has a default set of parameters that can be overridden or that are required to be set by a user. Those default parameters are represented in each component by defaults map at the top of each library file, for example in node_exporter.libsonnet.

This API is meant to ease the use of kube-prometheus as parameters can be passed from a JSON file and don't need to be in jsonnet format. However, if you need to modify particular parts of the stack, jsonnet allows you to do this and we are also not restricting such access in any way. An example of such modifications can be seen in any of our addons, like the addons/anti-affinity.libsonnet one.

Mixin integration Copy Section

Previously kube-prometheus project joined all mixins on a global level. However with a wider adoption of monitoring mixins this turned out to be a problem, especially apparent when two mixins started to use the same configuration field for different purposes. To fix this we moved all mixins into their own respective components:

• alertmanager mixin -> alertmanager.libsonnet
• kubernetes mixin -> k8s-control-plane.libsonnet
• kube-state-metrics mixin -> kube-state-metrics.libsonnet
• node_exporter mixin -> node_exporter.libsonnet
• prometheus and thanos sidecar mixins -> prometheus.libsonnet
• prometheus-operator mixin -> prometheus-operator.libsonnet
• kube-prometheus alerts and rules -> components/mixin/custom.libsonnet

etcd mixin is a special case as we add it inside an addon in addons/static-etcd.libsonnet

This results in creating multiple PrometheusRule objects instead of having one giant object as before. It also means each mixin is configured separately and accessing mixin objects is done via $.<component>.mixin.

Examples Copy Chapter

All examples from examples/ directory were adapted to the new codebase. Please take a look at them for guideance

Legacy migration Copy Chapter

An example of conversion of a legacy release-0.3 my.jsonnet file to release-0.8 can be found in migration-example

Advanced usage examples Copy Chapter

For more advanced usage examples, you can take a look at openshift/cluster-monitoring-operator which uses kube-prometheus components as standalone libraries to build a custom solution.

Final note Copy Chapter

Refactoring was a huge undertaking and possibly this document didn't describe in enough detail how to help you with migration to the new stack. If that is the case, please reach out to us by using GitHub discussions feature or directly on #prometheus-operator kubernetes slack channel.

Monitoring External Etcd (`docs/monitoring-external-etcd.md`) Copy Section

weight: 305
toc: true
title: Monitoring external etcd
menu:
docs:
parent: kube
lead: This guide will help you monitoring an external etcd cluster.
images: []

draft: false Copy Chapter

When the etcd cluster is not hosted inside Kubernetes.
This is often the case with Kubernetes setups. This approach has been tested with kube-aws but the same principals apply to other tools.

Note that etcd.jsonnet & static-etcd.libsonnet (which are described by a section of the customization) do the following:

• Put the three etcd TLS client files (CA & cert & key) into a secret in the namespace, and have Prometheus Operator load the secret.
• Create the following (to expose etcd metrics - port 2379): a Service, Endpoint, & ServiceMonitor.

Step 1: Open the port

You now need to allow the nodes Prometheus are running on to talk to the etcd on the port 2379 (if 2379 is the port used by etcd to expose the metrics)

If using kube-aws, you will need to edit the etcd security group inbound, specifying the security group of your Kubernetes node (worker) as the source.

kube-aws and EIP or ENI inconsistency Copy Chapter

With kube-aws, each etcd node has two IP addresses:

• EC2 instance IP
• EIP or ENI (depending on the chosen method in yuour cluster.yaml)

For some reason, some etcd node answer to :2379/metrics on the intance IP (eth0), some others on the EIP|ENI address (eth1). See issue
It would be of course much better if we could hit the EPI/ENI all the time as they don't change even if the underlying EC2 intance goes down.
If specifying the Instance IP (eth0) in the Prometheus Operator ServiceMonitor, and the EC2 intance goes down, one would have to update the ServiceMonitor.

Another idea woud be to use the DNS entries of etcd, but those are not currently supported for EndPoints objects in Kubernetes.

Step 2: verify

Go to the Prometheus UI on :9090/config and check that you have an etcd job entry:

- job_name: monitoring/etcd-k8s/0
  scrape_interval: 30s
  scrape_timeout: 10s
  ...

On the :9090/targets page:

• You should see "etcd" with the UP state. If not, check the Error column for more information.
• If no "etcd" targets are even shown on this page, prometheus isn't attempting to scrape it.

Monitoring Other Namespaces (`docs/monitoring-other-namespaces.md`) Copy Section

weight: 306
toc: true
title: Monitoring other Namespaces
menu:
docs:
parent: kube
lead: This guide will help you monitoring applications in other namespaces.
images: []

draft: false Copy Chapter

By default the RBAC rules are only enabled for the Default and kube-system namespaces.

Setup

You have to give the list of the namespaces that you want to be able to monitor.
This is done in the variable prometheus.roleSpecificNamespaces. You usually set this in your .jsonnet file when building the manifests.

Example to create the needed Role and RoleBinding for the Namespace foo :

/* Detailed source-code truncated for AI context efficiency. */

Security (`docs/security.md`) Copy Section

Security Copy Chapter

The manifests generated in this repository are subject to a security audit in CI via kubescape.
The scan can be run locally via make kubescape.

While we aim for best practices in terms of security by default, due to the nature of the project, we are required to make the exceptions in the following components:

node-exporter

• Host Port is set. Kubernetes already sets a Host Port by default when Host Network is enabled.. Since nothing can be done here, we configure it to our preference port.
• Host PID is set to true, since node-exporter requires direct access to the host namespace to gather statistics.
• Host Network is set to true, since node-exporter requires direct access to the host network to gather statistics.
• automountServiceAccountToken is set to true on Pod level as kube-rbac-proxy sidecar requires connection to kubernetes API server.

prometheus-adapter

• automountServiceAccountToken is set to true on Pod level as application requires connection to kubernetes API server.

blackbox-exporter

• automountServiceAccountToken is set to true on Pod level as kube-rbac-proxy sidecar requires connection to kubernetes API server.

kube-state-metrics

• automountServiceAccountToken is set to true on Pod level as kube-rbac-proxy sidecars requires connection to kubernetes API server.

prometheus-operator

• automountServiceAccountToken is set to true on Pod level as kube-rbac-proxy sidecars requires connection to kubernetes API server.

Troubleshooting (`docs/troubleshooting.md`) Copy Section

Troubleshooting

See the general guidelines for getting support from the community.

Error retrieving kubelet metrics Copy Chapter

Should the Prometheus /targets page show kubelet targets, but not able to successfully scrape the metrics, then most likely it is a problem with the authentication and authorization setup of the kubelets.

As described in the README.md Prerequisites section, in order to retrieve metrics from the kubelet token authentication and authorization must be enabled. Some Kubernetes setup tools do not enable this by default.

• If you are using Google's GKE product, see cAdvisor support.
• If you are using AWS EKS, see AWS EKS CNI support.
• If you are using Weave Net, see Weave Net support.

Authentication problem Copy Section

The Prometheus /targets page will show the kubelet job with the error 403 Unauthorized, when token authentication is not enabled. Ensure, that the --authentication-token-webhook=true flag is enabled on all kubelet configurations.

Authorization problem Copy Section

The Prometheus /targets page will show the kubelet job with the error 401 Unauthorized, when token authorization is not enabled. Ensure that the --authorization-mode=Webhook flag is enabled on all kubelet configurations.

kube-state-metrics resource usage Copy Chapter

In some environments, kube-state-metrics may need additional
resources. One driver for more resource needs, is a high number of
namespaces. There may be others.

kube-state-metrics resource allocation is managed by
addon-resizer
You can control it's parameters by setting variables in the
config. They default to:

kubeStateMetrics+:: {
      baseCPU: '100m',
      cpuPerNode: '2m',
      baseMemory: '150Mi',
      memoryPerNode: '30Mi',
    }

Error retrieving kube-proxy metrics Copy Chapter

By default, kubeadm will configure kube-proxy to listen on 127.0.0.1 for metrics. Because of this prometheus would not be able to scrape these metrics. This would have to be changed to 0.0.0.0 in one of the following two places:

1. Before cluster initialization, the config file passed to kubeadm init should have KubeProxyConfiguration manifest with the field metricsBindAddress set to 0.0.0.0:10249
2. If the k8s cluster is already up and running, we'll have to modify the configmap kube-proxy in the namespace kube-system and set the metricsBindAddress field. After this kube-proxy daemonset would have to be restarted with
   kubectl -n kube-system rollout restart daemonset kube-proxy

Conflict with an existing metrics-server Copy Chapter

kube-prometheus deploys the Prometheus Adapter, which serves the Kubernetes resource metrics API by registering the v1beta1.metrics.k8s.io APIService. A cluster can only have one provider of that APIService, so if a metrics-server is already installed the two conflict.

Installing kube-prometheus on top of an existing metrics-server typically fails with:

Error from server (AlreadyExists): error when creating "manifests/prometheus-adapter-apiService.yaml": apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io" already exists

and uninstalling kube-prometheus afterwards can delete the shared v1beta1.metrics.k8s.io APIService, breaking kubectl top and Horizontal Pod Autoscalers that relied on the existing metrics-server.

Because both provide the same resource metrics API, only one can be active. You have two options:

1. Remove the existing metrics-server and use prometheus-adapter (the default).
2. Switch kube-prometheus to use metrics-server instead of prometheus-adapter by setting values.common.resourceMetricsAPI:: 'metrics-server'. See Using metrics-server instead of prometheus-adapter for details.

To remove an existing metrics-server installed with Helm:

helm -n kube-system uninstall metrics-server

If it was installed from manifests, delete its resources, for example:

kubectl delete service/metrics-server -n kube-system
kubectl delete deployment.apps/metrics-server -n kube-system
kubectl delete apiservices.apiregistration.k8s.io v1beta1.metrics.k8s.io
kubectl delete clusterrole system:aggregated-metrics-reader
kubectl delete clusterrole system:metrics-server
kubectl delete clusterrolebinding metrics-server:system:auth-delegator
kubectl delete clusterrolebinding system:metrics-server
kubectl delete rolebinding metrics-server-auth-reader -n kube-system
kubectl delete serviceaccount metrics-server -n kube-system

Update (`docs/update.md`) Copy Section

Update kube-prometheus

You may wish to fetch changes made on this project so they are available to you.

Update jb Copy Chapter

jb may have been updated so it's a good idea to get the latest version of this binary:

$ go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest

Update kube-prometheus Copy Chapter

The command below will sync with upstream project:

$ jb update

Compile the manifests and apply Copy Chapter

Once updated, just follow the instructions under Generating and Apply the kube-prometheus stack from customizing.md doc to apply the changes to your cluster.

Migration from previous versions Copy Chapter

If you are migrating from release-0.7 branch or earlier please read what changed and how to migrate in our guide.

Refer to migration document for more information about migration from 0.3 and 0.8 versions of kube-prometheus.

Weave Net Support (`docs/weave-net-support.md`) Copy Section

Setup Weave Net monitoring using kube-prometheus

Weave Net is a resilient and simple to use CNI provider for Kubernetes. A well monitored and observed CNI provider helps in troubleshooting Kubernetes networking problems. Weave Net emits prometheus metrics for monitoring Weave Net. There are many ways to install Weave Net in your cluster. One of them is using kops.

Following this document, you can setup Weave Net monitoring for your cluster using kube-prometheus.

Contents Copy Chapter

Using kube-prometheus and kubectl you will be able install the following for monitoring Weave Net in your cluster:

1. Service for Weave Net The service which the ServiceMonitor scrapes.
2. ServiceMonitor for Weave Net Service monitor to scrape the Weave Net metrics and bring it to Prometheus.
3. Prometheus Alerts for Weave Net This will setup all the important Weave Net metrics you should be alerted on.
4. Grafana Dashboard for Weave Net This will setup the per Weave Net pod level monitoring for Weave Net.
5. Grafana Dashboard for Weave Net(Cluster) This will setup the cluster level monitoring for Weave Net.

Instructions Copy Chapter

• You can monitor Weave Net using an example like below. Please note that some alert configurations are environment specific and may require modifications of alert thresholds. For example: The FastDP flows have never gone below 15000 for us. But if this value is say 20000 for you then you can use an example like below to update the alert. The alerts which may require threshold modifications are WeaveNetFastDPFlowsLow and WeaveNetIPAMUnreachable.

/* Detailed source-code truncated for AI context efficiency. */

• After you have the required yamls file please run

kubectl create -f prometheus-serviceWeaveNet.yaml
kubectl create -f prometheus-serviceMonitorWeaveNet.yaml
kubectl apply -f  prometheus-rules.yaml
kubectl apply -f grafana-dashboardDefinitions.yaml
kubectl apply -f grafana-deployment.yaml

Windows (`docs/windows.md`) Copy Section

Windows

The Windows hostprocess addon adds the dashboards and rules from kubernetes-monitoring/kubernetes-mixin.

It also deploys windows_exporter as a hostprocess pod as Kubernetes now supports HostProcess containers on Windows nodes (as of v1.22). The cluster should be using containerd runtime.

local kp = (import 'kube-prometheus/main.libsonnet') +
  (import 'kube-prometheus/addons/windows-hostprocess.libsonnet') +
  {
    values+:: {
      windowsExporter+:: {
        image: "ghcr.io/prometheus-community/windows-exporter",
        version: "0.21.0",
      },
    },
  };

{ ['windows-exporter-' + name]: kp.windowsExporter[name] for name in std.objectFields(kp.windowsExporter) }

See the full example for setup.

If the cluster is running docker runtime then use the other Windows addon. The Windows addon does not deploy windows_exporter. Docker based Windows does not support running with windows_exporter in a pod so this add on uses additional scrape configuration to set up a static config to scrape the node ports where windows_exporter is configured.

The addon requires you to specify the node ips and ports where it can find the windows_exporter. See the full example for setup.

local kp = (import 'kube-prometheus/main.libsonnet') +
  (import 'kube-prometheus/addons/windows.libsonnet') +
  {
    values+:: {
      windowsScrapeConfig+:: {
          static_configs: {
              targets: ["10.240.0.65:5000", "10.240.0.63:5000"],
          },
      },
    },
  };

Customizations Alertmanager Configuration (`docs/customizations/alertmanager-configuration.md`) Copy Section

Alertmanager configuration Copy Section

The Alertmanager configuration is located in the values.alertmanager.config configuration field. In order to set a custom Alertmanager configuration simply set this field.

((import 'kube-prometheus/main.libsonnet') + {
   values+:: {
     alertmanager+: {
       config: |||
         global:
           resolve_timeout: 10m
         route:
           group_by: ['job']
           group_wait: 30s
           group_interval: 5m
           repeat_interval: 12h
           receiver: 'null'
           routes:
           - match:
               alertname: Watchdog
             receiver: 'null'
         receivers:
         - name: 'null'
       |||,
     },
   },
 }).alertmanager.secret

In the above example the configuration has been inlined, but can just as well be an external file imported in jsonnet via the importstr function.

((import 'kube-prometheus/main.libsonnet') + {
   values+:: {
     alertmanager+: {
       config: importstr 'alertmanager-config.yaml',
     },
   },
 }).alertmanager.secret

Customizations Components Name Namespace Overrides (`docs/customizations/components-name-namespace-overrides.md`) Copy Section

Components' name and namespace overrides Copy Section

It is possible to override the namespace where kube-prometheus is going to be deployed, like the example below:

local kp = (import 'kube-prometheus/main.libsonnet') +
{
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
};

If prefered, it can be changed individually by component. It is also possible to change the name of Prometheus and Alertmanager Custom Resources, like shown below:

/* Detailed source-code truncated for AI context efficiency. */

Customizations Developing Prometheus Rules And Grafana Dashboards (`docs/customizations/developing-prometheus-rules-and-grafana-dashboards.md`) Copy Section

weight: 307
toc: true
title: Prometheus Rules and Grafana Dashboards
menu:
docs:
parent: kube
lead: This guide will help you adding Prometheus Rules and Grafana Dashboards on top of kube-prometheus
images: []

draft: false Copy Chapter

kube-prometheus ships with a set of default Prometheus rules and Grafana dashboards. At some point one might like to extend them, the purpose of this document is to explain how to do this.

All manifests of kube-prometheus are generated using jsonnet.
Prometheus rules and Grafana dashboards in specific follow the
Prometheus Monitoring Mixins proposal.

For both the Prometheus rules and the Grafana dashboards Kubernetes ConfigMaps are generated within kube-prometheus. In order to add additional rules and dashboards simply merge them onto the existing json objects. This document illustrates examples for rules as well as dashboards.

As a basis, all examples in this guide are based on the base example of the kube-prometheus readme:

/* Detailed source-code truncated for AI context efficiency. */

Prometheus rules Copy Chapter

Alerting rules Copy Section

As per the Prometheus Monitoring Mixins proposal
Prometheus alerting rules are under the key prometheusAlerts in the top level object.
Additional alerting rules can be added by merging into the existing object.

The format is exactly the Prometheus format, so there should be no changes necessary should you have existing rules that you want to include.

Note that alerts can also be included into this file, using the jsonnet import function.
In this example it is just inlined in order to demonstrate their use in a single file.

/* Detailed source-code truncated for AI context efficiency. */

Recording rules Copy Section

In order to add a recording rule, simply do the same with the prometheusRules field.

Note that rules can just as well be included into this file, using the jsonnet import function.
In this example it is just inlined in order to demonstrate their use in a single file.

/* Detailed source-code truncated for AI context efficiency. */

Pre-rendered rules Copy Section

We acknowledge, that users may need to transition existing rules, and therefore allow an option to add additional pre-rendered rules. Luckily the yaml and json formats are very close so the yaml rules just need to be converted to json without any manual interaction needed. Just a tool to convert yaml to json is needed:

go get -u -v github.com/brancz/gojsontoyaml

And convert the existing rule file:

cat existingrule.yaml | gojsontoyaml -yamltojson > existingrule.json

Then import it in jsonnet:

local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
  exampleApplication: {
    prometheusRuleExample: {
      apiVersion: 'monitoring.coreos.com/v1',
      kind: 'PrometheusRule',
      metadata: {
        name: 'my-prometheus-rule',
        namespace: $.values.common.namespace,
      },
      spec: {
        groups: (import 'existingrule.json').groups,
      },
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ ['example-application-' + name]: kp.exampleApplication[name] for name in std.objectFields(kp.exampleApplication) }

Changing default rules Copy Section

Along with adding additional rules, we give the user the option to filter or adjust the existing rules imported by kube-prometheus/main.libsonnet.
The recording rules can be found in kube-prometheus/components/mixin/rules
and kubernetes-mixin/rules.
The alerting rules can be found in kube-prometheus/components/mixin/alerts
and kubernetes-mixin/alerts.

Knowing which rules to change, the user can now use functions from the Jsonnet standard library to make these changes.
Below are examples of both a filter and an adjustment being made to the default rules.
These changes can be assigned to a local variable and then added to the local kp object as seen in the examples above.

Filter

Here the alert KubeStatefulSetReplicasMismatch is being filtered out of the group kubernetes-apps.
The default rule can be seen here.
You first need to find out in which component the rule is defined (here it is kuberentesControlPlane).

local filter = {
  kubernetesControlPlane+: {
    prometheusRule+: {
      spec+: {
        groups: std.map(
          function(group)
            if group.name == 'kubernetes-apps' then
              group {
                rules: std.filter(
                  function(rule)
                    rule.alert != 'KubeStatefulSetReplicasMismatch',
                  group.rules
                ),
              }
            else
              group,
          super.groups
        ),
      },
    },
  },
};

Adjustment

Here the expression for another alert in the same component is updated from its previous value.
The default rule can be seen here.

local update = {
  kubernetesControlPlane+: {
    prometheusRule+: {
      spec+: {
        groups: std.map(
          function(group)
            if group.name == 'kubernetes-apps' then
              group {
                rules: std.map(
                  function(rule)
                    if rule.alert == 'KubePodCrashLooping' then
                      rule {
                        expr: 'rate(kube_pod_container_status_restarts_total{namespace=kube-system,job="kube-state-metrics"}[10m]) * 60 * 5 > 0',
                      }
                    else
                      rule,
                  group.rules
                ),
              }
            else
              group,
          super.groups
        ),
      },
    },
  },
};

Using the example from above about adding in pre-rendered rules, the new local variables can be added in as follows:

/* Detailed source-code truncated for AI context efficiency. */

Dashboards Copy Chapter

Dashboards can either be added using jsonnet or simply a pre-rendered json dashboard.

Jsonnet dashboard Copy Section

We recommend using the grafonnet library for jsonnet,
which gives you a simple DSL to generate Grafana dashboards.
Following the Prometheus Monitoring Mixins proposal
additional dashboards are added to the grafanaDashboards key, located in the top level object.
To add new jsonnet dashboards, simply add one.

Note that dashboards can just as well be included into this file, using the jsonnet import function.
In this example it is just inlined in order to demonstrate their use in a single file.

/* Detailed source-code truncated for AI context efficiency. */

Pre-rendered Grafana dashboards Copy Section

As jsonnet is a superset of json, the jsonnet import function can be used to include Grafana dashboard json blobs.
In this example we are importing a provided example dashboard.

local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+:: {
      namespace: 'monitoring',
    },
    grafana+: {
      dashboards+:: {  // use this method to import your dashboards to Grafana
        'my-dashboard.json': (import 'example-grafana-dashboard.json'),
      },
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

In case you have lots of json dashboard exported out from grafana UI the above approach is going to take lots of time.
To improve performance we can use rawDashboards field and provide it's value as json string by using importstr

local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+:: {
      namespace: 'monitoring',
    },
    grafana+: {
      rawDashboards+:: {
        'my-dashboard.json': (importstr 'example-grafana-dashboard.json'),
      },
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

Mixins Copy Section

Kube-prometheus comes with a couple of default mixins as the Kubernetes-mixin and the Node-exporter mixin,
however there are many more mixins.
To use other mixins, kube-prometheus has a jsonnet library for creating a PrometheusRule CRD and Grafana dashboards from a mixin.
Below is an example of creating a mixin object that has Prometheus rules and Grafana dashboards:

// Import the library function for adding mixins
local addMixin = (import 'kube-prometheus/lib/mixin.libsonnet');

// Create your mixin
local myMixin = addMixin({
  name: 'myMixin',
  mixin: import 'my-mixin/mixin.libsonnet',
});

The myMixin object will have two objects - prometheusRules and grafanaDashboards. The grafanaDashboards object will be needed to be added to the dashboards field as in the example below:

values+:: {
  grafana+:: {
    dashboards+:: myMixin.grafanaDashboards

The prometheusRules object is a PrometheusRule CRD. It should be defined as its own jsonnet object.
If you define multiple mixins in a single jsonnet object, there is a possibility that they will overwrite each others'
configuration and there will be unintended effects.
Therefore, use the prometheusRules object as its own jsonnet object:

...
{ ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ 'external-mixins/my-mixin-prometheus-rules': myMixin.prometheusRules } // one object for each mixin

As mentioned above each mixin is configurable and you would configure the mixin as in the example below:

local myMixin = addMixin({
  name: 'myMixin',
  mixin: (import 'my-mixin/mixin.libsonnet') + {
    _config+:: {
      myMixinSelector: 'my-selector',
      interval: '30d', // example
    },
  },
});

The library has also two optional parameters - the namespace for the PrometheusRule CRD and the dashboard folder for the Grafana dashboards.
The below example shows how to use both:

local myMixin = addMixin({
  name: 'myMixin',
  namespace: 'prometheus', // default is monitoring
  dashboardFolder: 'Observability',
  mixin: (import 'my-mixin/mixin.libsonnet') + {
    _config+:: {
      myMixinSelector: 'my-selector',
      interval: '30d', // example
    },
  },
});

The created prometheusRules object will have the metadata field namespace added and the usage will remain the same.
However, the grafanaDasboards will be added to the folderDashboards field instead of the dashboards field as shown in the example below:

values+:: {
  grafana+:: {
    folderDashboards+:: {
        Kubernetes: {
            ...
        },
        Misc: {
            'grafana-home.json': import 'dashboards/misc/grafana-home.json',
        },
    } + myMixin.grafanaDashboards

Full example of including etcd mixin using method described above:

/* Detailed source-code truncated for AI context efficiency. */

Customizations Dropping Unwanted Dashboards (`docs/customizations/dropping-unwanted-dashboards.md`) Copy Section

Dropping unwanted dashboards Copy Section

When deploying kube-prometheus, your Grafana instance is deployed with a lot of dashboards by default. All those dashboards are comming from upstream projects like kubernetes-mixin, prometheus-mixin and node-exporter-mixin, among others.

In case you find out that you don't need some of them, you can choose to remove those dashboards like in the example below, which removes the alertmanager-overview.json dashboard.

/* Detailed source-code truncated for AI context efficiency. */

Customizations Exposing Prometheus Alertmanager Grafana Ingress (`docs/customizations/exposing-prometheus-alertmanager-grafana-ingress.md`) Copy Section

weight: 303
toc: true
title: Expose via Gateway API
menu:
docs:
parent: kube
lead: This guide will help you exposing Prometheus, Alertmanager and Grafana using the Gateway API.
images: []

draft: false Copy Chapter

In order to access the web interfaces via the Internet, the Gateway API is the recommended option. This guide explains how the Gateway API can be used to expose the Prometheus, Alertmanager and Grafana UIs, that are included in the kube-prometheus project.

Note: before continuing, it is recommended to first get familiar with the kube-prometheus stack by itself.

Prerequisites Copy Chapter

Apart from a running Kubernetes cluster with a running kube-prometheus stack, a Gateway API controller must be installed and a GatewayClass must be available. The Gateway API is implemented by a number of controllers, for example Istio, Envoy Gateway and Contour; refer to the list of implementations and follow your controller's installation instructions. The gatewayClassName referenced below must match a GatewayClass that exists in your cluster.

Setting up routing Copy Chapter

The Gateway API splits the configuration into two resources: a Gateway, which defines the entry point (its listeners, ports and protocols) and is usually created once and shared, and an HTTPRoute, which attaches to a Gateway and forwards traffic to a Service. The setup of HTTPRoute objects is the same for Prometheus, Alertmanager and Grafana. Therefore this guide demonstrates it in detail for Prometheus as it can easily be adapted for the other applications.

As monitoring data may contain sensitive data, these endpoints should be protected with authentication. The Gateway API does not define a portable authentication mechanism. Authentication is instead configured through the APIs of your Gateway controller — for example an external authorization filter or a policy attached to the Gateway or HTTPRoute — or by placing an authenticating proxy such as oauth2-proxy in front of the Services. Consult the documentation of your Gateway API implementation for the supported options.

Also, the applications provide external links to themselves in alerts and various places. When a Gateway is used in front of the applications these links need to be based on the external URL's. This can be configured for each application in jsonnet.

/* Detailed source-code truncated for AI context efficiency. */

In order to expose Alertmanager and Grafana, simply create additional HTTPRoute objects, but simply pointing at the alertmanager-main or grafana Service instead of the prometheus-k8s Service. Make sure to also use the correct numeric port respectively, for Alertmanager it is 9093, for Grafana it is 3000. Be sure to also specify the appropriate external URL. Note that the external URL for Grafana is set in a different way than the external URL for Prometheus or Alertmanager. See ingress.jsonnet for how to set the Grafana external URL.

In order to render the Gateway and HTTPRoute objects similar to the other objects use as demonstrated in the main readme:

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ ['gateway-api-' + name]: kp.gatewayAPI[name] for name in std.objectFields(kp.gatewayAPI) }

Note, that in comparison only the last line was added, the rest is identical to the original.

See ingress.jsonnet for an example implementation.

Adding the Gateway namespace to NetworkPolicies Copy Chapter

NetworkPolicies restricting access to the components are added by default. These can either be removed as in
networkpolicies-disabled.jsonnet or modified to allow traffic from the namespace where your Gateway controller's data plane runs, as
described here.

This is an example for Alertmanager, but the same can be applied to Grafana and Prometheus. Replace gateway-system with the namespace of your Gateway controller.

{
  alertmanager+:: {
    networkPolicy+: {
      spec+: {
        ingress: [
          super.ingress[0] + {
            from+: [
              {
                namespaceSelector: {
                  matchLabels: {
                    'kubernetes.io/metadata.name': 'gateway-system',
                  },
                },
              },
            ],
          },
        ] + super.ingress[1:],
      },
    },
  },
}

Customizations Metrics Server (`docs/customizations/metrics-server.md`) Copy Section

Using metrics-server instead of prometheus-adapter Copy Section

By default, kube-prometheus deploys prometheus-adapter to serve the Kubernetes resource metrics API (metrics.k8s.io). This is a full-featured solution that derives CPU and memory metrics from Prometheus queries, and can also serve custom and external metrics APIs.

However, for use cases where only the resource metrics API is needed (for example, when using KEDA for custom metrics scaling), the lightweight upstream metrics-server may be preferable.

[!IMPORTANT]
prometheus-adapter is being deprecated by SIG Instrumentation. The project recommends migrating to KEDA for custom/external metrics use cases. Since metrics-server handles the resource metrics API natively without requiring Prometheus queries, it is the natural replacement for the resource metrics portion. We plan to make metrics-server the default resourceMetricsAPI in a future release of kube-prometheus.

Switching to metrics-server

Set the resourceMetricsAPI field in values.common to 'metrics-server':

(import 'kube-prometheus/main.libsonnet') +
{
  values+:: {
    common+: {
      resourceMetricsAPI:: 'metrics-server',
    },
    metricsServer+: {
      kubeletInsecureTLS:: true,
    },
  },
}

This replaces the prometheus-adapter manifests with metrics-server manifests. Only one resource metrics API provider can be active at a time since they both register the v1beta1.metrics.k8s.io APIService.

When to set kubeletInsecureTLS

On clusters that do not use proper kubelet serving certificates (for example, kind, minikube, or clusters without a kubelet-serving CA), set kubeletInsecureTLS:: true. On production clusters with a properly configured kubelet PKI, this should be left at the default (false).

Emitting manifests

If you are writing your own top-level jsonnet (instead of using example.jsonnet), emit the resource metrics manifests using the helper library:

(import 'kube-prometheus/lib/resource-metrics-api.libsonnet')(kp)

This automatically selects the correct component based on the resourceMetricsAPI flag.

Single-node clusters

The default configuration deploys 2 replicas with hard pod anti-affinity, which requires at least 2 nodes. On single-node clusters (minikube, single-node kind, etc.), set replicas:: 1 or switch to soft anti-affinity:

values+:: {
  metricsServer+: {
    replicas:: 1,
  },
},

Available configuration

The following hidden fields can be overridden in values.metricsServer:

Generating and deploying manifests

To generate manifests using the provided example:

make manifests-metrics-server

This builds from examples/metrics-server.jsonnet and outputs YAML manifests to manifests/. Then deploy:

kubectl apply --server-side -f manifests/setup
kubectl apply -f manifests/

Full example

See examples/metrics-server.jsonnet for a complete working example that deploys the full kube-prometheus stack with metrics-server instead of prometheus-adapter.

Customizations Monitoring Additional Namespaces (`docs/customizations/monitoring-additional-namespaces.md`) Copy Section

Monitoring additional namespaces Copy Section

In order to monitor additional namespaces, the Prometheus server requires the appropriate Role and RoleBinding to be able to discover targets from that namespace. By default the Prometheus server is limited to the three namespaces it requires: default, kube-system and the namespace you configure the stack to run in via $.values.namespace. This is specified in $.values.prometheus.namespaces, to add new namespaces to monitor, simply append the additional namespaces:

local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },

    prometheus+: {
      namespaces+: ['my-namespace', 'my-second-namespace'],
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

Defining the ServiceMonitor for each additional Namespace

In order to Prometheus be able to discovery and scrape services inside the additional namespaces specified in previous step you need to define a ServiceMonitor resource.

Typically it is up to the users of a namespace to provision the ServiceMonitor resource, but in case you want to generate it with the same tooling as the rest of the cluster monitoring infrastructure, this is a guide on how to achieve this.

You can define ServiceMonitor resources in your jsonnet spec. See the snippet bellow:

/* Detailed source-code truncated for AI context efficiency. */

NOTE: make sure your service resources have the right labels (eg. 'app': 'myapp') applied. Prometheus uses kubernetes labels to discover resources inside the namespaces.

Customizations Monitoring All Namespaces (`docs/customizations/monitoring-all-namespaces.md`) Copy Section

Monitoring all namespaces Copy Section

In case you want to monitor all namespaces in a cluster, you can add the following mixin. Also, make sure to empty the namespaces defined in prometheus so that roleBindings are not created against them.

local kp = (import 'kube-prometheus/main.libsonnet') +
           (import 'kube-prometheus/addons/all-namespaces.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
    prometheus+: {
      namespaces: [],
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

NOTE: This configuration can potentially make your cluster insecure especially in a multi-tenant cluster. This is because this gives Prometheus visibility over the whole cluster which might not be expected in a scenario when certain namespaces are locked down for security reasons.

Proceed with creating ServiceMonitors for the services in the namespaces you actually want to monitor

Customizations Node Ports (`docs/customizations/node-ports.md`) Copy Section

NodePorts Copy Section

Another mixin that may be useful for exploring the stack is to expose the UIs of Prometheus, Alertmanager and Grafana on NodePorts:

(import 'kube-prometheus/main.libsonnet') +
(import 'kube-prometheus/addons/node-ports.libsonnet')

Customizations Perses (`docs/customizations/perses.md`) Copy Section

weight: 320
toc: true
title: Using Perses instead of Grafana
menu:
docs:
parent: kube
lead: Replace Grafana with Perses for Kubernetes cluster dashboards
images: []

draft: false Copy Chapter

[!NOTE]
Perses is currently integrated as an addon. Once the integration is mature, it may be promoted to a built-in toggle (similar to the metrics-server / prometheus-adapter switch).

Perses is a CNCF Sandbox observability visualization platform. The perses addon deploys Perses and Kubernetes observability dashboards from community-mixins via the perses-operator. In jsonnet, Grafana is omitted by setting grafana: {} (see examples/perses.jsonnet). On an existing cluster, kubectl apply does not delete Grafana — remove it explicitly after you are ready (see Remove Grafana).

What the addon deploys Copy Chapter

The addon also patches the existing Prometheus NetworkPolicy so Perses pods can query Prometheus.

[!NOTE]
Security: Perses runs with authentication disabled by default, matching the bundled Grafana setup in kube-prometheus. Enable auth and secure cookies for production deployments.

Storage: Dashboards and datasource configuration are stored in Kubernetes CRs. The Perses server uses ephemeral file storage for runtime state; pod restarts do not remove PersesDashboard or PersesGlobalDatasource objects.

Generate and apply Copy Chapter

Generate the full stack with Perses instead of Grafana:

make manifests-perses

[!IMPORTANT]
make manifests-perses regenerates the entire manifests/ tree from examples/perses.jsonnet (not only Perses objects).

Apply the generated manifests:

# Apply CRDs and namespace first
kubectl apply --server-side -f manifests/setup
kubectl wait \
    --for condition=Established \
    --all CustomResourceDefinition \
    --namespace=monitoring

# Apply the remaining manifests
kubectl apply -f manifests/

Access the Perses UI Copy Section

kubectl --namespace monitoring port-forward svc/perses 8080:8080

Open Perses at http://localhost:8080. The operator creates a Service named after the Perses CR (perses by default). Dashboards and the Prometheus datasource are loaded from CRs.

Using a custom jsonnet file Copy Section

See examples/perses.jsonnet. Generate with ./build.sh my-perses.jsonnet.

Remove Grafana (optional) Copy Chapter

On an existing kube-prometheus cluster, kubectl apply does not remove resources that disappeared from the generated manifests, so Grafana keeps running alongside Perses. That is fine for evaluation — you can run both UIs in parallel while verifying coverage.

Once you are satisfied with Perses, delete Grafana by label:

kubectl -n monitoring delete --ignore-not-found=true \
  deployment,service,serviceaccount,servicemonitor,networkpolicy,prometheusrule,secret,configmap \
  -l app.kubernetes.io/name=grafana

Customization Copy Chapter

Override versions Copy Section

{
  values+:: {
    common+: {
      versions+: {
        perses: '0.54.0',
        persesOperator: '0.5.0',
      },
    },
  },
}

Override images (e.g. internal registry) Copy Section

Default images are persesdev/perses:v<version> and persesdev/perses-operator:v<version>.

{
  values+:: {
    common+: {
      images+: {
        perses: 'my-registry.example.com/perses:v0.54.0',
        persesOperator: 'my-registry.example.com/perses-operator:v0.5.0',
      },
    },
  },
}

Dashboard query selectors Copy Section

The Perses addon imports pre-generated dashboards from community-mixins, which use kubernetes-mixin default job labels. kube-prometheus overrides those selectors for Grafana via kubernetesControlPlane; the addon applies the same rewiring when importing dashboards:

The addon also sets prometheus.externalLabels.cluster to kube-prometheus so the cluster dashboard variable resolves (required by kubernetes-mixin dashboards). Override either if your scrape labels differ:

{
  values+:: {
    prometheus+: {
      externalLabels+: {
        cluster: 'my-cluster',
      },
    },
    perses+: {
      cadvisorJobSelector: 'job="kubelet", metrics_path="/metrics/cadvisor"',
      kubeApiserverJobSelector: 'job="apiserver"',
    },
  },
}

Change the Prometheus datasource URL Copy Section

By default the datasource proxies queries through the Perses server to http://prometheus-k8s.<namespace>.svc.cluster.local:9090 (see perses.libsonnet). Patch the generated component to use a different URL:

local kp =
  (import 'kube-prometheus/main.libsonnet') +
  (import 'kube-prometheus/addons/perses.libsonnet') +
  {
    values+:: {
      common+: { namespace: 'monitoring' },
    },
    perses+: {
      prometheusGlobalDatasource+: {
        spec+: {
          config+: {
            plugin+: {
              spec+: {
                proxy+: {
                  spec+: {
                    url: 'http://my-prometheus.monitoring.svc.cluster.local:9090',
                  },
                },
              },
            },
          },
        },
      },
    },
  };

Select dashboard packages Copy Section

Override dashboardComponents to change which community-mixins packages are imported. Defaults are kubernetes, prometheus, alertmanager, and node-exporter.

Trim to a subset:

{
  values+:: {
    perses+: {
      dashboardComponents: ['kubernetes', 'prometheus', 'alertmanager'],
    },
  },
}

Include optional packages (for example blackbox-exporter and Perses overview):

{
  values+:: {
    perses+: {
      dashboardComponents: [
        'kubernetes',
        'prometheus',
        'alertmanager',
        'node-exporter',
        'blackbox-exporter',
        'perses',
      ],
    },
  },
}

Available component names match community-mixins dashboards.libsonnet.

Dashboard coverage Copy Chapter

[!NOTE]
The counts and component lists below reflect the state of community-mixins vendored at the time of this release. Community-mixins is under active development and may add new dashboards or components at any time. Check the upstream repository for the latest coverage.

Dashboards are imported from community-mixins via its dashboards.libsonnet helper (namespace, datasource, and labels), then rewired for kube-prometheus scrape labels.

Default packages (23 `PersesDashboard` CRs) Copy Section

Optional packages Copy Section

Enable these via dashboardComponents (see Select dashboard packages). Thanos and etcd panels may be empty until those targets are scraped.

These kube-prometheus components have no Perses dashboard in community-mixins yet: kube-state-metrics, prometheus-operator, prometheus-adapter, and metrics-server.

Authoritative configuration is in perses.libsonnet. Run make manifests-perses after addon or mixin updates.

Switching back to Grafana Copy Chapter

To revert to Grafana, regenerate default manifests and reapply, then remove Perses resources that are no longer in the manifests:

make manifests
kubectl apply --server-side -f manifests/setup
kubectl wait --for condition=Established --all CustomResourceDefinition --namespace=monitoring
kubectl apply -f manifests/

# Perses CRs (instance and dashboards are namespaced; global datasource is cluster-scoped)
kubectl -n monitoring delete --ignore-not-found=true perses/perses
kubectl delete --ignore-not-found=true \
  persesglobaldatasources.perses.dev/prometheus-datasource
kubectl -n monitoring delete --ignore-not-found=true \
  persesdashboards.perses.dev --all

# Operator resources (all carry app.kubernetes.io/part-of=perses-operator)
kubectl -n monitoring delete --ignore-not-found=true \
  deployment,service,serviceaccount,servicemonitor,networkpolicy,prometheusrule,role,rolebinding \
  -l app.kubernetes.io/part-of=perses-operator
kubectl delete --ignore-not-found=true clusterrole,clusterrolebinding \
  -l app.kubernetes.io/part-of=perses-operator

Optionally, remove the perses.dev CRDs if you no longer need them:

kubectl delete --ignore-not-found=true crd \
  perses.perses.dev \
  persesdashboards.perses.dev \
  persesdatasources.perses.dev \
  persesglobaldatasources.perses.dev

Testing Copy Chapter

make test-e2e-perses runs Go end-to-end tests against a cluster that already has the Perses stack deployed. It does not generate or apply manifests — follow Generate and apply first, then:

export KUBECONFIG=~/.kube/config   # point at your test cluster
make test-e2e-perses

CI follows the same order: generate manifests, deploy to a kind cluster, then run make test-e2e-perses (see .github/workflows/ci.yaml).

References Copy Chapter

• Perses project
• perses-operator
• community-mixins

Customizations Platform Specific (`docs/customizations/platform-specific.md`) Copy Section

Running kube-prometheus on specific platforms Copy Section

A common example is that not all Kubernetes clusters are created exactly the same way, meaning the configuration to monitor them may be slightly different. For the following clusters there are mixins available to easily configure them:

• aws
• bootkube
• eks
• gke
• kops
• kops_coredns
• kubeadm
• kubespray

These mixins are selectable via the platform field of kubePrometheus:

(import 'kube-prometheus/main.libsonnet') +
{
  values+:: {
    common+: {
      platform: 'example-platform',
    },
  },
}

Customizations Pod Anti Affinity (`docs/customizations/pod-anti-affinity.md`) Copy Section

Pod Anti-Affinity Copy Section

To prevent Prometheus and Alertmanager instances from being deployed onto the same node when
possible, one can include the kube-prometheus-anti-affinity.libsonnet mixin:

local kp = (import 'kube-prometheus/main.libsonnet') +
           (import 'kube-prometheus/addons/anti-affinity.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

Customizations Prometheus Agent (`docs/customizations/prometheus-agent.md`) Copy Section

Prometheus-agent mode Copy Section

ATTENTION: Although it is possible to run Prometheus in Agent mode with Prometheus-Operator, it requires strategic merge patches. This practice is not recommended and we do not provide support if Prometheus doesn't work as you expect. Try it at your own risk!

/* Detailed source-code truncated for AI context efficiency. */

Customizations Static Etcd Configuration (`docs/customizations/static-etcd-configuration.md`) Copy Section

Static etcd configuration Copy Section

In order to configure a static etcd cluster to scrape there is a simple static-etcd.libsonnet mixin prepared.

An example of how to use it can be seen below:

/* Detailed source-code truncated for AI context efficiency. */

If you'd like to monitor an etcd instance that lives outside the cluster, see Monitoring external etcd for more information.

Note that monitoring etcd in minikube is currently not possible because of how etcd is setup. (minikube's etcd binds to 127.0.0.1:2379 only, and within host networking namespace.)

Customizations Strip Limits (`docs/customizations/strip-limits.md`) Copy Section

Stripping container resource limits Copy Section

Sometimes in small clusters, the CPU/memory limits can get high enough for alerts to be fired continuously. To prevent this, one can strip off the predefined limits.
To do that, one can import the following mixin

local kp = (import 'kube-prometheus/main.libsonnet') +
           (import 'kube-prometheus/addons/strip-limits.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

Customizations Using Custom Container Registry (`docs/customizations/using-custom-container-registry.md`) Copy Section

Internal Registry Copy Section

Some Kubernetes installations source all their images from an internal registry. kube-prometheus supports this use case and helps the user synchronize every image it uses to the internal registry and generate manifests pointing at the internal registry.

To produce the docker pull/tag/push commands that will synchronize upstream images to internal-registry.com/organization (after having run the jb command to populate the vendor directory):

$ jsonnet -J vendor -S --tla-str repository=internal-registry.com/organization examples/sync-to-internal-registry.jsonnet
$ docker pull k8s.gcr.io/addon-resizer:1.8.4
$ docker tag k8s.gcr.io/addon-resizer:1.8.4 internal-registry.com/organization/addon-resizer:1.8.4
$ docker push internal-registry.com/organization/addon-resizer:1.8.4
$ docker pull quay.io/prometheus/alertmanager:v0.16.2
$ docker tag quay.io/prometheus/alertmanager:v0.16.2 internal-registry.com/organization/alertmanager:v0.16.2
$ docker push internal-registry.com/organization/alertmanager:v0.16.2
...

The output of this command can be piped to a shell to be executed by appending | sh.

Then to generate manifests with internal-registry.com/organization, use the withImageRepository mixin:

local mixin = import 'kube-prometheus/addons/config-mixins.libsonnet';
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
} + mixin.withImageRepository('internal-registry.com/organization');

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }

--- METRICS ---

• Files Extracted: 38
• Estimated Token Budget: ~31217 tokens
• Canonical Reference: https://codewiki.google/github.com/prometheus-operator/kube-prometheus