# Technical Documentation: prometheus-operator/kube-prometheus

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/prometheus-operator/kube-prometheus)

### Overview (`README.md`)

# kube-prometheus

[](https://github.com/prometheus-operator/kube-prometheus/actions)
[](http://slack.k8s.io/)

> [!WARNING]
> Everything is experimental and may change significantly at any time.

This repository collects Kubernetes manifests, [Grafana](http://grafana.com/) dashboards, and [Prometheus rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with [Prometheus](https://prometheus.io/) using the Prometheus Operator.

The content of this project is written in [jsonnet](http://jsonnet.org/). This project could both be described as a package as well as a library.

**Components included in this package**:

* The [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator)
* Highly available [Prometheus](https://prometheus.io/)
* Highly available [Alertmanager](https://github.com/prometheus/alertmanager)
* [Prometheus node-exporter](https://github.com/prometheus/node_exporter)
* [Prometheus blackbox-exporter](https://github.com/prometheus/blackbox_exporter)
* [Prometheus Adapter for Kubernetes Metrics APIs](https://github.com/kubernetes-sigs/prometheus-adapter)
* [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)
* [Grafana](https://grafana.com/)

**Optional addon:**

* [Perses](https://perses.dev/) is CNCF Sandbox alternative to Grafana (see [Using Perses instead of Grafana](docs/customizations/perses.md))

This stack is meant for cluster monitoring, so it is pre-configured to collect metrics from all Kubernetes components. In addition to that it delivers a default set of dashboards and alerting rules. Many of the useful dashboards and alerts come from the [kubernetes-mixin project](https://github.com/kubernetes-monitoring/kubernetes-mixin), similar to this project it provides composable jsonnet as a library for users to customize to their needs.

## Prerequisites

You will need a Kubernetes cluster, that's it! By default it is assumed, that the kubelet uses token authentication and authorization, as otherwise Prometheus needs a client certificate, which gives it full access to the kubelet, rather than just the metrics. Token authentication and authorization allows more fine grained and easier access control.

This means the kubelet configuration must contain these flags:

* `--authentication-token-webhook=true` This flag enables, that a `ServiceAccount` token can be used to authenticate against the kubelet(s). This can also be enabled by setting the kubelet configuration value `authentication.webhook.enabled` to `true`.
* `--authorization-mode=Webhook` This flag enables, that the kubelet will perform an RBAC request with the API to determine, whether the requesting entity (Prometheus in this case) is allowed to access a resource, in specific for this project the `/metrics` endpoint. This can also be enabled by setting the kubelet configuration value `authorization.mode` to `Webhook`.

This stack provides [resource metrics](https://github.com/kubernetes/metrics#resource-metrics-api) by deploying
the [Prometheus Adapter](https://github.com/kubernetes-sigs/prometheus-adapter).
This adapter is an Extension API Server and Kubernetes needs to be have this feature enabled, otherwise the adapter has
no effect, but is still deployed.

## Compatibility

The following Kubernetes versions are supported and work as we test against these versions in their respective branches. But note that other versions might work!

> [!NOTE]
> In CI we will be testing only last two releases and main branch on a regular basis.

| kube-prometheus stack                                                                      | Kubernetes 1.33 | Kubernetes 1.34 | Kubernetes 1.35 | Kubernetes 1.36 | Kubernetes 1.37 |
|--------------------------------------------------------------------------------------------|-----------------|-----------------|-----------------|-----------------|-----------------|
| [`release-0.17`](https://github.com/prometheus-operator/kube-prometheus/tree/release-0.17) | ✔               | ✔               | ✔               | x               | x               |
| [`release-0.18`](https://github.com/prometheus-operator/kube-prometheus/tree/release-0.18) | ✔               | ✔               | ✔               | ✔               | x               |
| [`release-0.19`](https://github.com/prometheus-operator/kube-prometheus/tree/release-0.19) | x               | x               | ✔               | ✔               | ✔               |
| [`main`](https://github.com/prometheus-operator/kube-prometheus/tree/main)                 | x               | x               | ✔               | ✔               | ✔               |

## Quickstart

This project is intended to be used as a library (i.e. the intent is not for you to create your own modified copy of this repository).

Though for a quickstart a compiled version of the Kubernetes [manifests](manifests) generated with this library (specifically with `example.jsonnet`) is checked into this repository in order to try the content out quickly. To try out the stack un-customized run:

* Create the monitoring stack using the config in the `manifests` directory:

  ```shell
  # Create the namespace and CRDs, and then wait for them to be available before creating the remaining resources
  # Note that due to some CRD size we are using kubectl server-side apply feature which is generally available since kubernetes 1.22.
  # If you are using previous kubernetes versions this feature may not be available and you would need to use kubectl create instead.
  kubectl apply --server-side -f manifests/setup
  kubectl wait \
      --for condition=Established \
      --all CustomResourceDefinition \
      --namespace=monitoring
  kubectl apply -f manifests/
  ```

We create the namespace and CustomResourceDefinitions first to avoid race conditions when deploying the monitoring components.
Alternatively, the resources in both folders can be applied with a single command
`kubectl apply --server-side -f manifests/setup -f manifests`, but it may be necessary to run the command multiple times for all components to
be created successfully.

* To teardown the stack:

  ```shell
  kubectl delete --ignore-not-found=true -f manifests/ -f manifests/setup
  ```

The [official documentation](http://prometheus-operator.dev/docs/getting-started/installation/) contains the full version of this quick-start guide, and includes [instructions](https://prometheus-operator.dev/kube-prometheus/kube/access-ui/) on how to access Prometheus, AlertManager, and Grafana.

### minikube

To try out this stack, start [minikube](https://github.com/kubernetes/minikube) with the following command:

```shell
minikube delete && minikube start --container-runtime=containerd --kubernetes-version=v1.33.1 --memory=6g --bootstrapper=kubeadm --extra-config=kubelet.authentication-token-webhook=true --extra-config=kubelet.authorization-mode=Webhook --extra-config=scheduler.bind-address=0.0.0.0 --extra-config=controller-manager.bind-address=0.0.0.0
```

The kube-prometheus stack includes a resource metrics API server (prometheus-adapter by default), so the metrics-server addon is not necessary. Ensure the metrics-server addon is disabled on minikube:

```shell
minikube addons disable metrics-server
```

> [!NOTE]
> If you prefer to use the lightweight upstream metrics-server instead of prometheus-adapter, see [Using metrics-server instead of prometheus-adapter](docs/customizations/metrics-server.md).

## Getting started

Before deploying kube-prometheus in a production environment, read:

1. [Customizing kube-prometheus](docs/customizing.md)
2. [Customization examples](docs/customizations)
3. [Accessing Graphical User Interfaces](docs/access-ui.md)
4. [Troubleshooting kube-prometheus](docs/troubleshooting.md)

## Documentation

1. [Continuous Delivery](examples/continuous-delivery)
2. [Update to new version](docs/update.md)
3. For more documentation on the project refer to `docs/` directory.

## Contributing

To contribute to kube-prometheus, refer to [Contributing](CONTRIBUTING.md).

## Join the discussion

If you have any questions or feedback regarding kube-prometheus, join the [kube-prometheus discussion](https://github.com/prometheus-operator/kube-prometheus/discussions). Alternatively, consider joining [#prometheus-operator](https://kubernetes.slack.com/archives/CFFDS2Z7F) slack channel or project's bi-weekly [Contributor Office Hours](https://docs.google.com/document/d/1-fjJmzrwRpKmSPHtXN5u6VZnn39M28KqyQGBEJsqUOk/edit#).

## License

Apache License 2.0, see [LICENSE](https://github.com/prometheus-operator/kube-prometheus/blob/main/LICENSE).

---

### Overview (`developer-workspace/README.md`)

# Ephemeral developer workspaces

Aiming to provide better developer experience when making contributions to kube-prometheus, whether by actively developing new features/bug fixes or by reviewing pull requests, we want to provide ephemeral developer workspaces with everything already configured (as far as tooling makes it possible).

A developer workspace provides a brand new Kubernetes cluster, where kube-prometheus can be easily deployed and the contributor can easily see the impact that a pull request is proposing.

Today only [Github Codespaces](https://github.com/features/codespaces) is supported. Unfortunately, Codespaces is not available for everyone. If you are fortunate to have access to it, you can open a new workspace from a specific branch, or even from Pull Requests.





After your workspace start, you can deploy a kube-prometheus inside a Kind cluster inside by running `make deploy`.

If you are reviewing a PR, you'll have a fully-functional kubernetes cluster, generating real monitoring data that can be used to review if the proposed changes works as described.

If you are working on new features/bug fixes, you can regenerate kube-prometheus's YAML manifests with `make generate` and deploy it again with `make deploy`.

---

### Overview (`docs/migration-example/readme.md`)

## Example of conversion of a legacy my.jsonnet file

An example conversion of a legacy custom jsonnet file to release-0.8
format can be seen by viewing and comparing this
[release-0.3 jsonnet file](my.release-0.3.jsonnet) (when the github
repo was under `https://github.com/coreos/kube-prometheus...`)
and the corresponding [release-0.8 jsonnet file](my.release-0.8.jsonnet).

These two files have had necessary blank lines added so that they
can be compared side-by-side and line-by-line on screen.

The conversion covers both the change of stopping using ksonnet after
release-0.3 and also the major migration after release-0.7 as described in
[migration-guide.md](../migration-guide.md)

The sample files are intended as an example of format conversion and
not necessarily best practice for the files in release-0.3 or release-0.8.

Below are three sample extracts of the conversion as an indication of the
changes required.

<table>
<tr>
<th> release-0.3 </th>
<th> release-0.8 </th>
</tr>
<tr>
<td>

```jsonnet
local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-kubeadm.libsonnet') +
  (import 'kube-prometheus/kube-prometheus-static-etcd.libsonnet') +

  {
    _config+:: {
      // Override namespace
      namespace: 'monitoring',
  
  
  
  
   
   
   
```

</td>
<td>

```jsonnet
local kp =
  (import 'kube-prometheus/main.libsonnet') +
  // kubeadm now achieved by setting platform value - see 9 lines below
  (import 'kube-prometheus/addons/static-etcd.libsonnet') +
  (import 'kube-prometheus/addons/podsecuritypolicies.libsonnet') +
  {
    values+:: {
      common+: {
        namespace: 'monitoring',
      },

      // Add kubeadm platform-specific items,
      // including kube-contoller-manager and kube-scheduler discovery
      kubePrometheus+: {
        platform: 'kubeadm',
      },
```

</td>
</tr>
</table>
<table>
<tr>
<th> release-0.3 </th>
<th> release-0.8 </th>
</tr>
<tr>
<td>

```jsonnet
    // Add additional ingresses
    // See https://github.com/coreos/kube-prometheus/...
    //           tree/master/examples/ingress.jsonnet
    ingress+:: {
      alertmanager:
        ingress.new() +


        ingress.mixin.metadata.withName('alertmanager') +
        ingress.mixin.metadata.withNamespace($._config.namespace) +
        ingress.mixin.metadata.withAnnotations({
          'kubernetes.io/ingress.class': 'nginx-api',
        }) +

        ingress.mixin.spec.withRules(
          ingressRule.new() +
          ingressRule.withHost(alert_manager_host) +
          ingressRule.mixin.http.withPaths(
            ingressRuleHttpPath.new() +




            ingressRuleHttpPath.mixin.backend
                               .withServiceName('alertmanager-operated') +
            ingressRuleHttpPath.mixin.backend.withServicePort(9093)
          ),
        ) +
        // Note we do not need a TLS secretName here as we are going to use the
        // nginx-ingress default secret which is a wildcard
        // secretName would need to be in the same namespace at this time,
        // see https://github.com/kubernetes/ingress-nginx/issues/2371
        ingress.mixin.spec.withTls(
          ingressTls.new() +
          ingressTls.withHosts(alert_manager_host)
        ),
  
  
```

</td>
<td>

```jsonnet
    // Add additional ingresses
    // See https://github.com/prometheus-operator/kube-prometheus/...
    //           blob/main/examples/ingress.jsonnet
    ingress+:: {
      alertmanager: {
        apiVersion: 'networking.k8s.io/v1',
        kind: 'Ingress',
        metadata: {
          name: 'alertmanager',
          namespace: $.values.common.namespace,
          annotations: {
            'kubernetes.io/ingress.class': 'nginx-api',
          },
        },
        spec: {
          rules: [{
            host: alert_manager_host,
            http: {
              paths: [{
                path: '/',
                pathType: 'Prefix',
                backend: {
                  service: {
                    name: 'alertmanager-operated',
                    port: {
                      number: 9093,
                    },
                  },
                },
              }],
            },
          }],
          tls: [{

            hosts: [alert_manager_host],
          }],
        },
      },
```

</td>
</tr>
</table>
<table>
<tr>
<th> release-0.3 </th>
<th> release-0.8 </th>
</tr>
<tr>
<td>

```jsonnet
    // Additional prometheus rules
    // See https://github.com/coreos/kube-prometheus/docs/...
    //           developing-prometheus-rules-and-grafana-dashboards.md
    //
    // cat my-prometheus-rules.yaml | \
    //   gojsontoyaml -yamltojson | \
    //   jq . > my-prometheus-rules.json
    prometheusRules+:: {














      groups+: import 'my-prometheus-rules.json',


    },
  };
  
  
  
  
```

</td>
<td>

```jsonnet
    // Additional prometheus rules
    // See https://github.com/prometheus-operator/kube-prometheus/blob/main/...
    //           docs/developing-prometheus-rules-and-grafana-dashboards.md...
    //           #pre-rendered-rules
    // cat my-prometheus-rules.yaml | \
    //   gojsontoyaml -yamltojson | \
    //   jq . > my-prometheus-rules.json
    prometheusMe: {
      rules: {
        apiVersion: 'monitoring.coreos.com/v1',
        kind: 'PrometheusRule',
        metadata: {
          name: 'my-prometheus-rule',
          namespace: $.values.common.namespace,
          labels: {
            'app.kubernetes.io/name': 'kube-prometheus',
            'app.kubernetes.io/part-of': 'kube-prometheus',
            prometheus: 'k8s',
            role: 'alert-rules',
          },
        },
        spec: {
          groups: import 'my-prometheus-rules.json',
        },
      },
    },
  };

...

+ { ['prometheus-my-' + name]: kp.prometheusMe[name] for name in std.objectFields(kp.prometheusMe) }
```

</td>
</tr>
</table>

---

### Overview (`examples/continuous-delivery/argocd/README.md`)

## ArgoCD Example

This is the simplest, working example of an argocd app, the JSON object built is now an array of objects as that is the prefered format for ArgoCD. And ArgoCD specific annotations are added to manifests.

Requirements:

- **ArgoCD 1.7+**

- Follow the vendor generation steps at the root of this repository and generate a `vendored` folder (referenced in `application.yaml`).

- Make sure that argocd-cm has `application.instanceLabelKey` set to something else than `app.kubernetes.io/instance`, otherwise it will cause problems with prometheus target discovery. (see also [Why Is My App Out Of Sync Even After Syncing?](https://argo-cd.readthedocs.io/en/stable/faq/#why-is-my-app-out-of-sync-even-after-syncing))

---

### Overview (`jsonnet/kube-prometheus/platforms/README.md`)

# Adding a new platform specific configuration

Adding a new platform specific configuration requires to update the [customization example](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/customizations/platform-specific.md) and the [platforms.libsonnet](platforms.libsonnet) file by adding the platform to the list of existing ones. This allow the new platform to be discoverable and easily configurable by the users.

---

### EKS Cni Support (`docs/EKS-cni-support.md`)

# CNI monitoring special configuration updates for EKS

AWS EKS uses [CNI](https://github.com/aws/amazon-vpc-cni-k8s) networking plugin for pod networking in Kubernetes using Elastic Network Interfaces on AWS

One fatal issue that can occur is that you run out of IP addresses in your eks cluster. (Generally happens due to error configs where pods keep scheduling).

You can monitor the `awscni` using kube-promethus with :

```jsonnet mdox-exec="cat examples/eks-cni-example.jsonnet"
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
    kubePrometheus+: {
      platform: 'eks',
    },
  },
  kubernetesControlPlane+: {
    prometheusRuleEksCNI+: {
      spec+: {
        groups+: [
          {
            name: 'example-group',
            rules: [
              {
                record: 'aws_eks_available_ip',
                expr: 'sum by(instance) (awscni_total_ip_addresses) - sum by(instance) (awscni_assigned_ip_addresses) < 10',
              },
            ],
          },
        ],
      },
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) }
```

After you have the required yaml file please run

```
kubectl apply -f manifests/prometheus-serviceMonitorAwsEksCNI.yaml
```

---

### GKE Cadvisor Support (`docs/GKE-cadvisor-support.md`)

# Kubelet / cAdvisor special configuration updates for GKE

Prior to GKE 1.11, the kubelet does not support token
authentication. Until it does, Prometheus must use HTTP (not HTTPS)
for scraping.

You can configure this behavior through kube-prometheus with:

```
local kp = (import 'kube-prometheus/kube-prometheus.libsonnet') +
    (import 'kube-prometheus/kube-prometheus-insecure-kubelet.libsonnet') +
	{
        _config+:: {
		# ... config here
		}
    };
```

Or, you can patch and re-apply your existing manifests with:

On linux:

```
sed -i -e 's/https/http/g' manifests/prometheus-serviceMonitorKubelet.yaml
```

On MacOs:

```
sed -i '' -e 's/https/http/g' manifests/prometheus-serviceMonitorKubelet.yaml
```

After you have modified the yaml file please run

```
kubectl apply -f manifests/prometheus-serviceMonitorKubelet.yaml
```

---

### Access Ui (`docs/access-ui.md`)

---
weight: 300
toc: true
title: Access Dashboards
menu:
    docs:
        parent: kube
images: []
draft: false
---

Prometheus, Grafana, and Alertmanager dashboards can be accessed quickly using `kubectl port-forward` after running the quickstart via the commands below.

> Kubernetes 1.10 or later is required.

You can also learn how to [expose Prometheus/Alertmanager/Grafana via Ingress](https://prometheus-operator.dev/kube-prometheus/kube/exposing-prometheus-alertmanager-grafana-ingress/)

## Prometheus

```shell
kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090
```

Open Prometheus on [http://localhost:9090](http://localhost:9090) in your browser.

Check out the [alerts](http://localhost:9090/alerts) and [rules](http://localhost:9090/rules) pages with the pre-configured rules and alerts!
This Prometheus is supposed to monitor your Kubernetes cluster and make sure to alert you if there’s a problem with it.

For your own applications we recommend running one or more other instances.

## Grafana

```shell
kubectl --namespace monitoring port-forward svc/grafana 3000
```

Open Grafana on [localhost:3000](https://localhost:3000) in your browser.
You can login with the username `admin` and password `admin`.

## Perses (alternative to Grafana)

If you have enabled the [Perses addon](https://prometheus-operator.dev/kube-prometheus/kube/perses/) instead of Grafana:

```shell
kubectl --namespace monitoring port-forward svc/perses 8080
```

Open Perses on [localhost:8080](http://localhost:8080) in your browser. The bundled [community-mixins](https://github.com/perses/community-mixins) dashboards for the default kube-prometheus stack are loaded from `PersesDashboard` CRs, and a Prometheus datasource is auto-configured. See [dashboard coverage](https://prometheus-operator.dev/kube-prometheus/kube/perses/#dashboard-coverage) for details.

## Alertmanager

```shell
kubectl --namespace monitoring port-forward svc/alertmanager-main 9093
```

Open Alertmanager on [localhost:9093](http://localhost:9093) in your browser.

---

### Blackbox Exporter (`docs/blackbox-exporter.md`)

---
weight: 304
toc: true
title: Blackbox Exporter
menu:
    docs:
        parent: kube
lead: This guide will help you deploying the blackbox-exporter with the Probe custom resource definition.
images: []
draft: false
description: This guide will help you deploying the blackbox-exporter with the Probe custom resource definition.
date: "2021-03-08T08:49:31+00:00"
---

# Setting up a blackbox exporter

The `prometheus-operator` defines a `Probe` resource type that can be used to describe blackbox checks. To execute these, a separate component called [`blackbox_exporter`](https://github.com/prometheus/blackbox_exporter) has to be deployed, which can be scraped to retrieve the results of these checks. You can use `kube-prometheus` to set up such a blackbox exporter within your Kubernetes cluster.

## Adding blackbox exporter manifests to an existing `kube-prometheus` configuration

1. Override blackbox-related configuration parameters as needed.
2. Add the following to the list of renderers to render the blackbox exporter manifests:

```
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) }
```

## Configuration parameters influencing the blackbox exporter

* `_config.namespace`: the namespace where the various generated resources (`ConfigMap`, `Deployment`, `Service`, `ServiceAccount` and `ServiceMonitor`) will reside. This does not affect where you can place `Probe` objects; that is determined by the configuration of the `Prometheus` resource. This option is shared with other `kube-prometheus` components; defaults to `default`.
* `_config.imageRepos.blackboxExporter`: the name of the blackbox exporter image to deploy. Defaults to `quay.io/prometheus/blackbox-exporter`.
* `_config.versions.blackboxExporter`: the tag of the blackbox exporter image to deploy. Defaults to the version `kube-prometheus` was tested with.
* `_config.imageRepos.configmapReloader`: the name of the ConfigMap reloader image to deploy. Defaults to `ghcr.io/jimmidyson/configmap-reload`.
* `_config.versions.configmapReloader`: the tag of the ConfigMap reloader image to deploy. Defaults to the version `kube-prometheus` was tested with.
* `_config.resources.blackbox-exporter.requests`: the requested resources; this is used for each container. Defaults to `10m` CPU and `20Mi` RAM. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for details.
* `_config.resources.blackbox-exporter.limits`: the resource limits; this is used for each container. Defaults to `20m` CPU and `40Mi` RAM. See https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ for details.
* `_config.blackboxExporter.port`: the exposed HTTPS port of the exporter. This is what Prometheus can scrape for metrics related to the blackbox exporter itself. Defaults to `9115`.
* `_config.blackboxExporter.internalPort`: the internal plaintext port of the exporter. Prometheus scrapes configured via `Probe` objects cannot access the HTTPS port right now, so you have to specify this port in the `url` field. Defaults to `19115`.
* `_config.blackboxExporter.replicas`: the number of exporter replicas to be deployed. Defaults to `1`.
* `_config.blackboxExporter.matchLabels`: map of the labels to be used to select resources belonging to the instance deployed. Defaults to `{ 'app.kubernetes.io/name': 'blackbox-exporter' }`
* `_config.blackboxExporter.assignLabels`: map of the labels applied to components of the instance deployed. Defaults to all the labels included in the `matchLabels` option, and additionally `app.kubernetes.io/version` is set to the version of the blackbox exporter.
* `_config.blackboxExporter.modules`: the modules available in the blackbox exporter installation, i.e. the types of checks it can perform. The default value includes most of the modules defined in the default blackbox exporter configuration: `http_2xx`, `http_post_2xx`, `tcp_connect`, `pop3s_banner`, `ssh_banner`, and `irc_banner`. `icmp` is omitted so the exporter can be run with minimum privileges, but you can add it back if needed - see the example below. See https://github.com/prometheus/blackbox_exporter/blob/master/CONFIGURATION.md for the configuration format, except you have to use JSON instead of YAML here.
* `_config.blackboxExporter.privileged`: whether the `blackbox-exporter` container should be running as non-root (`false`) or root with heavily-restricted capability set (`true`). Defaults to `true` if you have any ICMP modules defined (which need the extra permissions) and `false` otherwise.

## Complete example

```jsonnet
local kp =
  (import 'kube-prometheus/kube-prometheus.libsonnet') +
  {
    _config+:: {
      namespace: 'monitoring',
      blackboxExporter+:: {
        modules+:: {
          icmp: {
            prober: 'icmp',
          },
        },
      },
    },
  };

{ ['setup/0namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{
  ['setup/prometheus-operator-' + name]: kp.prometheusOperator[name]
  for name in std.filter((function(name) name != 'serviceMonitor'), std.objectFields(kp.prometheusOperator))
} +
// serviceMonitor is separated so that it can be created after the CRDs are ready
{ 'prometheus-operator-serviceMonitor': kp.prometheusOperator.serviceMonitor } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['blackbox-exporter-' + name]: kp.blackboxExporter[name] for name in std.objectFields(kp.blackboxExporter) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
```

After installing the generated manifests, you can create `Probe` resources, for example:

```yaml
kind: Probe
apiVersion: monitoring.coreos.com/v1
metadata:
  name: example-com-website
  namespace: monitoring
spec:
  interval: 60s
  module: http_2xx
  prober:
    url: blackbox-exporter.monitoring.svc.cluster.local:19115
  targets:
    staticConfig:
      static:
      - http://example.com
      - https://example.com
```

---

### Community Support (`docs/community-support.md`)

# Community support

For bugs, you can use the GitHub [issue tracker](https://github.com/prometheus-operator/kube-prometheus/issues/new/choose).

For questions, you can use the GitHub [discussions forum](https://github.com/prometheus-operator/kube-prometheus/discussions).

Many of the `kube-prometheus` project's contributors and users can also be found on the #prometheus-operator channel of the [Kubernetes Slack](https://slack.k8s.io/).

`kube-prometheus` is the aggregation of many projects that all have different
channels to reach out for help and support. This community strives at
supporting all users and you should never be afraid of asking us first. However
if your request relates specifically to one of the projects listed below, it is
often more efficient to reach out to the project directly. If you are unsure,
please feel free to open an issue in this repository and we will redirect you
if applicable.

## prometheus-operator

For documentation, check the project's [documentation directory](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation).

For questions, use the #prometheus-operator channel on the [Kubernetes Slack](https://slack.k8s.io/).

For bugs, use the GitHub [issue tracker](https://github.com/prometheus-operator/prometheus-operator/issues/new/choose).

## Prometheus, Alertmanager, node_exporter

For documentation, check the Prometheus [online docs](https://prometheus.io/docs/). There is a
[section](https://prometheus.io/docs/introduction/media/) with links to blog
posts, recorded talks and presentations. This [repository](https://github.com/roaldnefs/awesome-prometheus)
(not affiliated to the Prometheus project) has also a list of curated resources
related to the Prometheus ecosystem.

For questions, see the Prometheus [community page](https://prometheus.io/community/) for the various channels.

There is also a #prometheus channel on the [CNCF Slack](https://slack.cncf.io/).

## kube-state-metrics

For documentation, see the project's [docs directory](https://github.com/kubernetes/kube-state-metrics/tree/main/docs).

For questions, use the #kube-state-metrics channel on the [Kubernetes Slack](https://slack.k8s.io/).

For bugs, use the GitHub [issue tracker](https://github.com/kubernetes/kube-state-metrics/issues/new/choose).

## Kubernetes

For documentation, check the [Kubernetes docs](https://kubernetes.io/docs/home/).

For questions, use the [community forums](https://discuss.kubernetes.io/) and the [Kubernetes Slack](https://slack.k8s.io/). Check also the [community page](https://kubernetes.io/community/#discuss).

For bugs, use the GitHub [issue tracker](https://github.com/kubernetes/kubernetes/issues/new/choose).

## Prometheus adapter

For documentation, check the project's [README](https://github.com/DirectXMan12/k8s-prometheus-adapter/blob/master/README.md).

For questions, use the #sig-instrumentation channel on the [Kubernetes Slack](https://slack.k8s.io/).

For bugs, use the GitHub [issue tracker](https://github.com/DirectXMan12/k8s-prometheus-adapter/issues/new).

## Grafana

For documentation, check the [Grafana docs](https://grafana.com/docs/grafana/latest/).

For questions, use the [community forums](https://community.grafana.com/).

For bugs, use the GitHub [issue tracker](https://github.com/grafana/grafana/issues/new/choose).

## kubernetes-mixin

For documentation, check the project's [README](https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/README.md).

For questions, use #monitoring-mixins channel on the [Kubernetes Slack](https://slack.k8s.io/).

For bugs, use the GitHub [issue tracker](https://github.com/kubernetes-monitoring/kubernetes-mixin/issues/new).

## Jsonnet

For documentation, check the [Jsonnet](https://jsonnet.org/) website.

For questions, use the [mailing list](https://groups.google.com/forum/#!forum/jsonnet).

---

### Customizing (`docs/customizing.md`)

# Customizing Kube-Prometheus

This section:
* describes how to customize the kube-prometheus library via compiling the kube-prometheus manifests yourself (as an alternative to the [README.md quickstart section](../README.md#quickstart)).
* still doesn't require you to make a copy of this entire repository, but rather only a copy of a few select files.

## Installing

The content of this project consists of a set of [jsonnet](http://jsonnet.org/) files making up a library to be consumed.

Install this library in your own project with [jsonnet-bundler](https://github.com/jsonnet-bundler/jsonnet-bundler#install) (the jsonnet package manager):

```shell
$ mkdir my-kube-prometheus; cd my-kube-prometheus
$ jb init  # Creates the initial/empty `jsonnetfile.json`
# Install the kube-prometheus dependency
$ jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main # Creates `vendor/` & `jsonnetfile.lock.json`, and fills in `jsonnetfile.json`

$ wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/example.jsonnet -O example.jsonnet
$ wget https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/build.sh -O build.sh
$ chmod +x build.sh
```

> `jb` can be installed with `go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest`

> An e.g. of how to install a given version of this library: `jb install github.com/prometheus-operator/kube-prometheus/jsonnet/kube-prometheus@main`

In order to update the kube-prometheus dependency, simply use the jsonnet-bundler update functionality:

```shell
$ jb update
```

## Generating

e.g. of how to compile the manifests: `./build.sh example.jsonnet`

> before compiling, install `gojsontoyaml` tool with `go install github.com/brancz/gojsontoyaml@latest` and `jsonnet` with `go install github.com/google/go-jsonnet/cmd/jsonnet@latest`

Here's [example.jsonnet](../example.jsonnet):

> Note: some of the following components must be configured beforehand. See [configuration](#configuring) and [customization-examples](customizations).

```
/* Detailed source-code truncated for AI context efficiency. */
```

And here's the [build.sh](../build.sh) script (which uses `vendor/` to render all manifests in a json structure of `{filename: manifest-content}`):

```sh mdox-exec="cat ./build.sh"
#!/usr/bin/env bash

# This script uses arg $1 (name of *.jsonnet file to use) to generate the manifests/*.yaml files.

set -e
set -x
# only exit with zero if all commands of the pipeline exit successfully
set -o pipefail

# Make sure to use project tooling
PATH="$(pwd)/tmp/bin:${PATH}"

# Make sure to start with a clean 'manifests' dir
rm -rf manifests
mkdir -p manifests/setup

# Calling gojsontoyaml is optional, but we would like to generate yaml, not json
jsonnet -J vendor -m manifests "${1-example.jsonnet}" | xargs -I{} sh -c 'cat {} | gojsontoyaml > {}.yaml' -- {}

# Make sure to remove json files
find manifests -type f ! -name '*.yaml' -delete
mv manifests/kustomization.yaml .
```

> Note you need `jsonnet` (`go install github.com/google/go-jsonnet/cmd/jsonnet@latest`) and `gojsontoyaml` (`go install github.com/brancz/gojsontoyaml@latest`) installed to run `build.sh`. If you just want json output, not yaml, then you can skip the pipe and everything afterwards.

This script runs the jsonnet code, then reads each key of the generated json and uses that as the file name, and writes the value of that key to that file, and converts each json manifest to yaml.

## Configuring

Jsonnet has the concept of hidden fields. These are fields, that are not going to be rendered in a result. This is used to configure the kube-prometheus components in jsonnet. In the example jsonnet code of the above [Generating section](#generating), you can see an example of this, where the `namespace` is being configured to be `monitoring`. In order to not override the whole object, use the `+::` construct of jsonnet, to merge objects, this way you can override individual settings, but retain all other settings and defaults.

The available fields and their default values can be seen in [main.libsonnet](../jsonnet/kube-prometheus/main.libsonnet). Note that many of the fields get their default values from variables, and for example the version numbers are imported from [versions.json](../jsonnet/kube-prometheus/versions.json).

Configuration is mainly done in the `values` map. You can see this being used in the `example.jsonnet` to set the namespace to `monitoring`. This is done in the `common` field, which all other components take their default value from. See for example how Alertmanager is configured in `main.libsonnet`:

```
    alertmanager: {
      name: 'main',
      // Use the namespace specified under values.common by default.
      namespace: $.values.common.namespace,
      version: $.values.common.versions.alertmanager,
      image: $.values.common.images.alertmanager,
      mixin+: { ruleLabels: $.values.common.ruleLabels },
    },
```

The grafana definition is located in a different project (https://github.com/brancz/kubernetes-grafana ), but needed configuration can be customized from the same top level `values` field. For example to allow anonymous access to grafana, add the following `values` section:

```
      grafana+:: {
        config: { // http://docs.grafana.org/installation/configuration/
          sections: {
            "auth.anonymous": {enabled: true},
          },
        },
      },
```

## Apply the kube-prometheus stack

The previous generation step has created a bunch of manifest files in the manifest/ folder.
Now simply use `kubectl` to install Prometheus and Grafana as per your configuration:

```shell
# Update the namespace and CRDs, and then wait for them to be available before creating the remaining resources
$ kubectl apply --server-side -f manifests/setup
$ kubectl apply -f manifests/
```

> Note that due to some CRD size we are using kubectl server-side apply feature which is generally available since
> kubernetes 1.22. If you are using previous kubernetes versions this feature may not be available and you would need to
> use `kubectl create` instead.

Alternatively, the resources in both folders can be applied with a single command
`kubectl apply --server-side -Rf manifests`, but it may be necessary to run the command multiple times for all components to
be created successfully.

Check the monitoring namespace (or the namespace you have specific in `namespace: `) and make sure the pods are running. Prometheus and Grafana should be up and running soon.

## Minikube Example

To use an easy to reproduce example, see [minikube.jsonnet](../examples/minikube.jsonnet), which uses the minikube setup as demonstrated in [Prerequisites](../README.md#prerequisites). Because we would like easy access to our Prometheus, Alertmanager and Grafana UIs, `minikube.jsonnet` exposes the services as NodePort type services.

---

### Deploy Kind (`docs/deploy-kind.md`)

---
weight: 301
toc: true
title: Deploy to kind
menu:
    docs:
        parent: kube
lead: This guide will help you deploying kube-prometheus on Kubernetes kind.
images: []
draft: false
description: This guide will help you deploying kube-prometheus on Kubernetes kind.
---

Time to explain how!

Your chance of [**contributing**](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/deploy-kind.md)!

---

### Endpoints Migration (`docs/endpoints-migration.md`)

# Migration from Endpoints to EndpointSlice

`kube-prometheus` 0.17+ automatically configures Prometheus to use EndpointSlice instead of Endpoints for Kubernetes service discovery (Endpoints have been deprecated in Kubernetes 1.33).

While the migration should be seamless for "regular" pods, it requires a few manual steps for components running as host services (e.g. node_exporter and kubelet):
1. The node_exporter and kubelet ServiceMonitors rely on the Prometheus operator's kubelet controller which manages the `kube-system/kubelet` Service.
2. With `kube-prometheus` 0.17, the Prometheus operator starts with both `--kubelet-endpoints=true` and `--kubelet-endpointslice=true` to ensure that a) the operator synchronizes the EndpointSlice object(s) backing the `kube-system/kubelet` Service and b) Kubernetes stops mirroring the `kube-system/kubelet` Endpoints object to EndpointSlice object(s) (otherwise the operator and kube-controller-manager would fight for the same resources).
3. After verifying that all targets are correctly discovered, it is ok to modify the operator's deployment and use `--kubelet-endpoints=false` instead. This will become the default in a future version of `kube-prometheus`.
4. The `kube-system/kubelet` Endpoints object should be removed manually.

To verify the status of the Endpoints and EndpointSlice objects, run:

```shell
kubectl get -n kube-system endpoints kubelet
kubectl get -n kube-system endpointslice -l endpointslice.kubernetes.io/managed-by=prometheus-operator
```

---

### Kube Prometheus On Kubeadm (`docs/kube-prometheus-on-kubeadm.md`)

---
weight: 302
toc: true
title: Deploy to kubeadm
menu:
    docs:
        parent: kube
lead: This guide will help you deploying kube-prometheus on Kubernetes kubeadm.
images: []
draft: false
description: This guide will help you deploying kube-prometheus on Kubernetes kubeadm.
---

The [kubeadm](https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/) tool is linked by Kubernetes as the offical way to deploy and manage self-hosted clusters. kubeadm does a lot of heavy lifting by automatically configuring your Kubernetes cluster with some common options. This guide is intended to show you how to deploy Prometheus, Prometheus Operator and Kube Prometheus to get you started monitoring your cluster that was deployed with kubeadm.

This guide assumes you have a basic understanding of how to use the functionality the Prometheus Operator implements. If you haven't yet, we recommend reading through the [getting started guide](https://prometheus-operator.dev/docs/platform/platform-guide/) as well as the [alerting guide](https://prometheus-operator.dev/docs/developer/alerting/).

## kubeadm Pre-requisites

This guide assumes you have some familiarity with `kubeadm` or at least have deployed a cluster using `kubeadm`. By default, `kubeadm` does not expose two of the services that we will be monitoring. Therefore, in order to get the most out of the `kube-prometheus` package, we need to make some quick tweaks to the Kubernetes cluster. Since we will be monitoring the `kube-controller-manager` and `kube-scheduler`, we must expose them to the cluster.

By default, `kubeadm` runs these pods on your master and bound to `127.0.0.1`. There are a couple of ways to change this. The recommended way to change these features is to use the [kubeadm config file](https://kubernetes.io/docs/reference/generated/kubeadm/#config-file). An example configuration file can be used:

```yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
controlPlaneEndpoint: "192.168.1.173:6443"
apiServer:
  extraArgs:
    authorization-mode: "Node,RBAC"
controllerManager:
  extraArgs:
    bind-address: "0.0.0.0"
scheduler:
  extraArgs:
    bind-address: "0.0.0.0"
certificatesDir: "/etc/kubernetes/pki"
etcd:
  # one of local or external
  local:
    dataDir: "/var/lib/etcd"
kubernetesVersion: "v1.23.1"
networking:
  dnsDomain: "cluster.local"
  serviceSubnet: "10.96.0.0/12"
imageRepository: "registry.k8s.io"
```

Notice the `.scheduler.extraArgs` and `.controllerManager.extraArgs`. This exposes the `kube-controller-manager` and `kube-scheduler` services to the rest of the cluster. If you have kubernetes core components as pods in the kube-system namespace, ensure that the `kube-prometheus-exporter-kube-scheduler` and `kube-prometheus-exporter-kube-controller-manager` services' `spec.selector` values match those of pods.

In previous versions of Kubernetes, we had to make a change to the `kubelet` setting with regard to `cAdvisor` monitoring on the control-plane as well as all the nodes. But this is **no longer required due to [the change of Kubernetes](https://github.com/kubernetes/kubernetes/issues/56523)**.

In case you already have a Kubernetes deployed with kubeadm, change the address kube-controller-manager and kube-scheduler listens in addition to previous kubelet change:

```
sed -e "s/- --bind-address=127.0.0.1/- --bind-address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-controller-manager.yaml
sed -e "s/- --bind-address=127.0.0.1/- --bind-address=0.0.0.0/" -i /etc/kubernetes/manifests/kube-scheduler.yaml
```

With these changes, your Kubernetes cluster is ready.

## Metric Sources

Monitoring a Kubernetes cluster with Prometheus is a natural choice as Kubernetes components themselves are instrumented with Prometheus metrics, therefore those components simply have to be discovered by Prometheus and most of the cluster is monitored.

Metrics that are rather about cluster state than a single component's metrics is exposed by the add-on component [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics).

Additionally, to have an overview of cluster nodes' resources the Prometheus [node_exporter](https://github.com/prometheus/node_exporter) is used. The node_exporter allows monitoring a node's resources: CPU, memory and disk utilization and more.

Once you complete this guide you will monitor the following:

* cluster state via kube-state-metrics
* nodes via the node_exporter
* kubelets
* apiserver
* kube-scheduler
* kube-controller-manager

## Getting Up and Running Fast with Kube-Prometheus

To help get started more quickly with monitoring Kubernetes clusters, [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) was created. It is a collection of manifests including dashboards and alerting rules that can easily be deployed. It utilizes the Prometheus Operator and all the manifests demonstrated in this guide.

This section represent a quick installation and is not intended to teach you about all the components. The easiest way to get started is to clone this repository and use the `kube-prometheus` section of the code.

```
git clone https://github.com/prometheus-operator/kube-prometheus
cd kube-prometheus/
```

First, create the namespace in which you want the monitoring tool suite to be running.

```
export NAMESPACE='monitoring'
kubectl create namespace "$NAMESPACE"
```

Now we will create the components for the Prometheus operator

```
kubectl --namespace="$NAMESPACE" apply -f manifests/prometheus-operator
```

This will create all the Prometheus Operator components. You might need to wait a short amount of time before the Custom Resource Definitions are available in the cluster. You can wait for them:

```
until kubectl --namespace="$NAMESPACE" get alertmanagers.monitoring.coreos.com > /dev/null 2>&1; do sleep 1; printf "."; done
```

Next, we will install the node exporter and then kube-state-metrics:

```
kubectl --namespace="$NAMESPACE" apply -f manifests/node-exporter
kubectl --namespace="$NAMESPACE" apply -f manifests/kube-state-metrics
```

Then, we can deploy the grafana credentials. By default, the username/password will be `admin/admin`, you should change these for your production clusters.

```
kubectl --namespace="$NAMESPACE" apply -f manifests/grafana/grafana-credentials.yaml
```

Then install grafana itself:

```
kubectl --namespace="$NAMESPACE" apply -f manifests/grafana
```

Next up is the `Prometheus` object itself. We will deploy the application, and then the roles/role-bindings.

```
find manifests/prometheus -type f ! -name prometheus-k8s-roles.yaml ! -name prometheus-k8s-role-bindings.yaml -exec kubectl --namespace "$NAMESPACE" apply -f {} \;
kubectl apply -f manifests/prometheus/prometheus-k8s-roles.yaml
kubectl apply -f manifests/prometheus/prometheus-k8s-role-bindings.yaml
```

Finally, install the [Alertmanager](https://prometheus-operator.dev/docs/developer/alerting)

```
kubectl --namespace="$NAMESPACE" apply -f manifests/alertmanager
```

Now you should have a working cluster. After all the pods are ready, you should be able to reach:

* Prometheus UI on node port `30900`
* Alertmanager UI on node port `30903`
* Grafana on node port `30902`

These can of course be changed via the Service definitions. It is recommended to look at the [Exposing Prometheus and Alert Manager](https://prometheus-operator.dev/docs/platform/exposing-prometheus-and-alertmanager/) documentation for more detailed information on how to expose these services.

---

### Migration Guide (`docs/migration-guide.md`)

# Migration guide from release-0.7 and earlier

## Why?

Thanks to our community we identified a lot of short-commings of previous design, varying from issues with global state to UX problems. Hoping to fix at least part of those issues we decided to do a complete refactor of the codebase.

## Overview

### Breaking Changes

- global `_config` object is removed and the new `values` object is a partial replacement
- `imageRepos` field was removed and the project no longer tries to compose image strings. Use `$.values.common.images` to override default images.
- prometheus alerting and recording rules are split into multiple `PrometheusRule` objects
- kubernetes control plane ServiceMonitors and Services are now part of the new `kubernetesControlPlane` top-level object instead of `prometheus` object
- `jsonnet/kube-prometheus/kube-prometheus.libsonnet` file was renamed to `jsonnet/kube-prometheus/main.libsonnet` and slimmed down to bare minimum
- `jsonnet/kube-prometheus/kube-prometheus*-.libsonnet` files were move either to `jsonnet/kube-prometheus/addons/` or `jsonnet/kube-prometheus/platforms/` depending on the feature they provided
- all component libraries are now function- and not object-based
- monitoring-mixins are included inside each component and not globally. `prometheusRules`, `prometheusAlerts`, and `grafanaDashboards` are accessible only per component via `mixin` object (ex. `$.alertmanager.mixin.prometheusAlerts`)
- default repository branch changed from `master` to `main`
- labels on resources have changes, `kubectl apply` will not work correctly due to those field being immutable. Deleting the resource first before applying is a workaround if you are using the kubectl CLI. (This only applies to `Deployments` and `DaemonSets`.)

### New Features

- concept of `addons`, `components`, and `platforms` was introduced
- all main `components` are now represented internally by a function with default values and required parameters (see #Component-configuration for more information)
- `$.values` holds main configuration parameters and should be used to set basic stack configuration.
- common parameters across all `components` are stored now in `$.values.common`
- removed dependency on deprecated ksonnet library

## Details

### Components, Addons, Platforms

Those concepts were already present in the repository but it wasn't clear which file is holding what. After refactoring we categorized jsonnet code into 3 buckets and put them into separate directories:
- `components` - main building blocks for kube-prometheus, written as functions responsible for creating multiple objects representing kubernetes manifests. For example all objects for node_exporter deployment are bundled in `components/node_exporter.libsonnet` library
- `addons` - everything that can enhance kube-prometheus deployment. Those are small snippets of code adding a small feature, for example adding anti-affinity to pods via [`addons/anti-affinity.libsonnet`](https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/addons/anti-affinity.libsonnet). Addons are meant to be used in object-oriented way like `local kp = (import 'kube-prometheus/main.libsonnet') + (import 'kube-prometheus/addons/all-namespaces.libsonnet')`
- `platforms` - currently those are `addons` specialized to allow deploying kube-prometheus project on a specific platform.

### Component configuration

Refactoring main components to use functions allowed us to define APIs for said components. Each function has a default set of parameters that can be overridden or that are required to be set by a user. Those default parameters are represented in each component by `defaults` map at the top of each library file, for example in [`node_exporter.libsonnet`](https://github.com/prometheus-operator/kube-prometheus/blob/1d2a0e275af97948667777739a18b24464480dc8/jsonnet/kube-prometheus/components/node-exporter.libsonnet#L3-L34).

This API is meant to ease the use of kube-prometheus as parameters can be passed from a JSON file and don't need to be in jsonnet format. However, if you need to modify particular parts of the stack, jsonnet allows you to do this and we are also not restricting such access in any way. An example of such modifications can be seen in any of our `addons`, like the [`addons/anti-affinity.libsonnet`](https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/addons/anti-affinity.libsonnet) one.

### Mixin integration

Previously kube-prometheus project joined all mixins on a global level. However with a wider adoption of monitoring mixins this turned out to be a problem, especially apparent when two mixins started to use the same configuration field for different purposes. To fix this we moved all mixins into their own respective components:
- alertmanager mixin -> `alertmanager.libsonnet`
- kubernetes mixin -> `k8s-control-plane.libsonnet`
- kube-state-metrics mixin -> `kube-state-metrics.libsonnet`
- node_exporter mixin -> `node_exporter.libsonnet`
- prometheus and thanos sidecar mixins -> `prometheus.libsonnet`
- prometheus-operator mixin -> `prometheus-operator.libsonnet`
- kube-prometheus alerts and rules -> `components/mixin/custom.libsonnet`

> etcd mixin is a special case as we add it inside an `addon` in `addons/static-etcd.libsonnet`

This results in creating multiple `PrometheusRule` objects instead of having one giant object as before. It also means each mixin is configured separately and accessing mixin objects is done via `$.<component>.mixin`.

## Examples

All examples from `examples/` directory were adapted to the new codebase. [Please take a look at them for guideance](https://github.com/prometheus-operator/kube-prometheus/tree/main/examples)

## Legacy migration

An example of conversion of a legacy release-0.3 my.jsonnet file to release-0.8 can be found in [migration-example](migration-example)

## Advanced usage examples

For more advanced usage examples, you can take a look at [openshift/cluster-monitoring-operator](https://github.com/openshift/cluster-monitoring-operator/tree/main/jsonnet) which uses kube-prometheus components as standalone libraries to build a custom solution.

## Final note

Refactoring was a huge undertaking and possibly this document didn't describe in enough detail how to help you with migration to the new stack. If that is the case, please reach out to us by using [GitHub discussions](https://github.com/prometheus-operator/kube-prometheus/discussions) feature or directly on [#prometheus-operator kubernetes slack channel](http://slack.k8s.io/).

---

### Monitoring External Etcd (`docs/monitoring-external-etcd.md`)

---
weight: 305
toc: true
title: Monitoring external etcd
menu:
    docs:
        parent: kube
lead: This guide will help you monitoring an external etcd cluster.
images: []
draft: false
description: This guide will help you monitoring an external etcd cluster.
---

When the etcd cluster is not hosted inside Kubernetes.
This is often the case with Kubernetes setups. This approach has been tested with kube-aws but the same principals apply to other tools.

Note that [etcd.jsonnet](../examples/etcd.jsonnet) & [static-etcd.libsonnet](../jsonnet/kube-prometheus/addons/static-etcd.libsonnet) (which are described by a section of the [customization](customizations/static-etcd-configuration.md)) do the following:
* Put the three etcd TLS client files (CA & cert & key) into a secret in the namespace, and have Prometheus Operator load the secret.
* Create the following (to expose etcd metrics - port 2379): a Service, Endpoint, & ServiceMonitor.

# Step 1: Open the port

You now need to allow the nodes Prometheus are running on to talk to the etcd on the port 2379 (if 2379 is the port used by etcd to expose the metrics)

If using kube-aws, you will need to edit the etcd security group inbound, specifying the security group of your Kubernetes node (worker) as the source.

## kube-aws and EIP or ENI inconsistency

With kube-aws, each etcd node has two IP addresses:

* EC2 instance IP
* EIP or ENI (depending on the chosen method in yuour cluster.yaml)

For some reason, some etcd node answer to :2379/metrics on the intance IP (eth0), some others on the EIP|ENI address (eth1). See [issue](https://github.com/kubernetes-incubator/kube-aws/issues/923)
It would be of course much better if we could hit the EPI/ENI all the time as they don't change even if the underlying EC2 intance goes down.
If specifying the Instance IP (eth0) in the Prometheus Operator ServiceMonitor, and the EC2 intance goes down, one would have to update the ServiceMonitor.

Another idea woud be to use the DNS entries of etcd, but those are not currently supported for EndPoints objects in Kubernetes.

# Step 2: verify

Go to the Prometheus UI on :9090/config and check that you have an etcd job entry:

```
- job_name: monitoring/etcd-k8s/0
  scrape_interval: 30s
  scrape_timeout: 10s
  ...
```

On the :9090/targets page:
* You should see "etcd" with the UP state. If not, check the Error column for more information.
* If no "etcd" targets are even shown on this page, prometheus isn't attempting to scrape it.

---

### Monitoring Other Namespaces (`docs/monitoring-other-namespaces.md`)

---
weight: 306
toc: true
title: Monitoring other Namespaces
menu:
    docs:
        parent: kube
lead: This guide will help you monitoring applications in other namespaces.
images: []
draft: false
description: This guide will help you monitoring applications in other namespaces.
---

By default the RBAC rules are only enabled for the `Default` and `kube-system` namespaces.

# Setup

You have to give the list of the namespaces that you want to be able to monitor.
This is done in the variable `prometheus.roleSpecificNamespaces`. You usually set this in your `.jsonnet` file when building the manifests.

Example to create the needed `Role` and `RoleBinding` for the Namespace `foo` :

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Security (`docs/security.md`)

## Security

The manifests generated in this repository are subject to a security audit in CI via [kubescape](https://github.com/armosec/kubescape).
The scan can be run locally via `make kubescape`.

While we aim for best practices in terms of security by default, due to the nature of the project, we are required to make the exceptions in the following components:

#### node-exporter
* Host Port is set. [Kubernetes already sets a Host Port by default when Host Network is enabled.](https://github.com/kubernetes/kubernetes/blob/1945829906546caf867992669a0bfa588edf8be6/pkg/apis/core/v1/defaults.go#L402-L411). Since nothing can be done here, we configure it to our preference port.
* Host PID is set to `true`, since node-exporter requires direct access to the host namespace to gather statistics.
* Host Network is set to `true`, since node-exporter requires direct access to the host network to gather statistics.
* `automountServiceAccountToken` is set to `true` on Pod level as kube-rbac-proxy sidecar requires connection to kubernetes API server.

#### prometheus-adapter
* `automountServiceAccountToken` is set to `true` on Pod level as application requires connection to kubernetes API server.

#### blackbox-exporter
* `automountServiceAccountToken` is set to `true` on Pod level as kube-rbac-proxy sidecar requires connection to kubernetes API server.

#### kube-state-metrics
* `automountServiceAccountToken` is set to `true` on Pod level as kube-rbac-proxy sidecars requires connection to kubernetes API server.

#### prometheus-operator
* `automountServiceAccountToken` is set to `true` on Pod level as kube-rbac-proxy sidecars requires connection to kubernetes API server.

---

### Troubleshooting (`docs/troubleshooting.md`)

# Troubleshooting

See the general [guidelines](community-support.md) for getting support from the community.

## Error retrieving kubelet metrics

Should the Prometheus `/targets` page show kubelet targets, but not able to successfully scrape the metrics, then most likely it is a problem with the authentication and authorization setup of the kubelets.

As described in the [README.md Prerequisites](../README.md#prerequisites) section, in order to retrieve metrics from the kubelet token authentication and authorization must be enabled. Some Kubernetes setup tools do not enable this by default.

- If you are using Google's GKE product, see [cAdvisor support](GKE-cadvisor-support.md).
- If you are using AWS EKS, see [AWS EKS CNI support](EKS-cni-support.md).
- If you are using Weave Net, see [Weave Net support](weave-net-support.md).

### Authentication problem

The Prometheus `/targets` page will show the kubelet job with the error `403 Unauthorized`, when token authentication is not enabled. Ensure, that the `--authentication-token-webhook=true` flag is enabled on all kubelet configurations.

### Authorization problem

The Prometheus `/targets` page will show the kubelet job with the error `401 Unauthorized`, when token authorization is not enabled. Ensure that the `--authorization-mode=Webhook` flag is enabled on all kubelet configurations.

## kube-state-metrics resource usage

In some environments, kube-state-metrics may need additional
resources. One driver for more resource needs, is a high number of
namespaces. There may be others.

kube-state-metrics resource allocation is managed by
[addon-resizer](https://github.com/kubernetes/autoscaler/tree/master/addon-resizer/nanny)
You can control it's parameters by setting variables in the
config. They default to:

```jsonnet
    kubeStateMetrics+:: {
      baseCPU: '100m',
      cpuPerNode: '2m',
      baseMemory: '150Mi',
      memoryPerNode: '30Mi',
    }
```

## Error retrieving kube-proxy metrics

By default, kubeadm will configure kube-proxy to listen on 127.0.0.1 for metrics. Because of this prometheus would not be able to scrape these metrics. This would have to be changed to 0.0.0.0 in one of the following two places:

1. Before cluster initialization, the config file passed to kubeadm init should have KubeProxyConfiguration manifest with the field metricsBindAddress set to 0.0.0.0:10249
2. If the k8s cluster is already up and running, we'll have to modify the configmap kube-proxy in the namespace kube-system and set the metricsBindAddress field. After this kube-proxy daemonset would have to be restarted with
   `kubectl -n kube-system rollout restart daemonset kube-proxy`

## Conflict with an existing metrics-server

kube-prometheus deploys the [Prometheus Adapter](https://github.com/kubernetes-sigs/prometheus-adapter), which serves the Kubernetes resource metrics API by registering the `v1beta1.metrics.k8s.io` APIService. A cluster can only have one provider of that APIService, so if a `metrics-server` is already installed the two conflict.

Installing kube-prometheus on top of an existing `metrics-server` typically fails with:

```
Error from server (AlreadyExists): error when creating "manifests/prometheus-adapter-apiService.yaml": apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io" already exists
```

and uninstalling kube-prometheus afterwards can delete the shared `v1beta1.metrics.k8s.io` APIService, breaking `kubectl top` and Horizontal Pod Autoscalers that relied on the existing metrics-server.

Because both provide the same resource metrics API, only one can be active. You have two options:

1. **Remove the existing metrics-server** and use prometheus-adapter (the default).
2. **Switch kube-prometheus to use metrics-server** instead of prometheus-adapter by setting `values.common.resourceMetricsAPI:: 'metrics-server'`. See [Using metrics-server instead of prometheus-adapter](customizations/metrics-server.md) for details.

To remove an existing metrics-server installed with Helm:

```shell
helm -n kube-system uninstall metrics-server
```

If it was installed from manifests, delete its resources, for example:

```shell
kubectl delete service/metrics-server -n kube-system
kubectl delete deployment.apps/metrics-server -n kube-system
kubectl delete apiservices.apiregistration.k8s.io v1beta1.metrics.k8s.io
kubectl delete clusterrole system:aggregated-metrics-reader
kubectl delete clusterrole system:metrics-server
kubectl delete clusterrolebinding metrics-server:system:auth-delegator
kubectl delete clusterrolebinding system:metrics-server
kubectl delete rolebinding metrics-server-auth-reader -n kube-system
kubectl delete serviceaccount metrics-server -n kube-system
```

---

### Update (`docs/update.md`)

# Update kube-prometheus

You may wish to fetch changes made on this project so they are available to you.

## Update jb

`jb` may have been updated so it's a good idea to get the latest version of this binary:

```shell
$ go install -a github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
```

## Update kube-prometheus

The command below will sync with upstream project:

```shell
$ jb update
```

## Compile the manifests and apply

Once updated, just follow the instructions under [Generating](customizing.md#generating) and [Apply the kube-prometheus stack](customizing.md#apply-the-kube-prometheus-stack) from [customizing.md doc](customizing.md) to apply the changes to your cluster.

## Migration from previous versions

If you are migrating from `release-0.7` branch or earlier please read [what changed and how to migrate in our guide](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/migration-guide.md).

Refer to [migration document](migration-example) for more information about migration from 0.3 and 0.8 versions of kube-prometheus.

---

### Weave Net Support (`docs/weave-net-support.md`)

# Setup Weave Net monitoring using kube-prometheus

[Weave Net](https://kubernetes.io/docs/concepts/cluster-administration/networking/#weave-net-from-weaveworks) is a resilient and simple to use CNI provider for Kubernetes. A well monitored and observed CNI provider helps in troubleshooting Kubernetes networking problems. [Weave Net](https://www.weave.works/docs/net/latest/concepts/how-it-works/) emits [prometheus metrics](https://www.weave.works/docs/net/latest/tasks/manage/metrics/) for monitoring Weave Net. There are many ways to install Weave Net in your cluster. One of them is using [kops](https://github.com/kubernetes/kops/blob/master/docs/networking.md).

Following this document, you can setup Weave Net monitoring for your cluster using kube-prometheus.

## Contents

Using kube-prometheus and kubectl you will be able install the following for monitoring Weave Net in your cluster:

1. [Service for Weave Net](https://gist.github.com/alok87/379c6234b582f555c141f6fddea9fbce) The service which the ServiceMonitor scrapes.
2. [ServiceMonitor for Weave Net](https://gist.github.com/alok87/e46a7f9a79ef6d1da6964a035be2cfb9) Service monitor to scrape the Weave Net metrics and bring it to Prometheus.
3. [Prometheus Alerts for Weave Net](https://stackoverflow.com/a/60447864) This will setup all the important Weave Net metrics you should be alerted on.
4. [Grafana Dashboard for Weave Net](https://grafana.com/grafana/dashboards/11789) This will setup the per Weave Net pod level monitoring for Weave Net.
5. [Grafana Dashboard for Weave Net(Cluster)](https://grafana.com/grafana/dashboards/11804) This will setup the cluster level monitoring for Weave Net.

## Instructions
- You can monitor Weave Net using an example like below. **Please note that some alert configurations are environment specific and may require modifications of alert thresholds**. For example: The FastDP flows have never gone below 15000 for us. But if this value is say 20000 for you then you can use an example like below to update the alert. The alerts which may require threshold modifications are `WeaveNetFastDPFlowsLow` and `WeaveNetIPAMUnreachable`.

```
/* Detailed source-code truncated for AI context efficiency. */
```

- After you have the required yamls file please run

```
kubectl create -f prometheus-serviceWeaveNet.yaml
kubectl create -f prometheus-serviceMonitorWeaveNet.yaml
kubectl apply -f  prometheus-rules.yaml
kubectl apply -f grafana-dashboardDefinitions.yaml
kubectl apply -f grafana-deployment.yaml
```

---

### Windows (`docs/windows.md`)

# Windows

The [Windows hostprocess addon](../examples/windows-hostprocess.jsonnet) adds the dashboards and rules from [kubernetes-monitoring/kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin#dashboards-for-windows-nodes).

It also deploys [windows_exporter](https://github.com/prometheus-community/windows_exporter) as a [hostprocess pod](https://github.com/prometheus-community/windows_exporter/blob/master/docs/kubernetes/kubernetes.md) as Kubernetes now supports HostProcess containers on Windows nodes (as of [v1.22](https://kubernetes.io/blog/2021/08/16/windows-hostprocess-containers/)). The cluster should be using containerd runtime.

```
local kp = (import 'kube-prometheus/main.libsonnet') +
  (import 'kube-prometheus/addons/windows-hostprocess.libsonnet') +
  {
    values+:: {
      windowsExporter+:: {
        image: "ghcr.io/prometheus-community/windows-exporter",
        version: "0.21.0",
      },
    },
  };

{ ['windows-exporter-' + name]: kp.windowsExporter[name] for name in std.objectFields(kp.windowsExporter) }
```

See the [full example](../examples/windows-hostprocess.jsonnet) for setup.

If the cluster is running docker runtime then use the other [Windows addon](../examples/windows.jsonnet). The Windows addon does not deploy windows_exporter. Docker based Windows does not support running with [windows_exporter](https://github.com/prometheus-community/windows_exporter) in a pod so this add on uses [additional scrape configuration](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md) to set up a static config to scrape the node ports where windows_exporter is configured.

The addon requires you to specify the node ips and ports where it can find the windows_exporter. See the [full example](../examples/windows.jsonnet) for setup.

```
local kp = (import 'kube-prometheus/main.libsonnet') +
  (import 'kube-prometheus/addons/windows.libsonnet') +
  {
    values+:: {
      windowsScrapeConfig+:: {
          static_configs: {
              targets: ["10.240.0.65:5000", "10.240.0.63:5000"],
          },
      },
    },
  };
```

---

### Customizations Alertmanager Configuration (`docs/customizations/alertmanager-configuration.md`)

### Alertmanager configuration

The Alertmanager configuration is located in the `values.alertmanager.config` configuration field. In order to set a custom Alertmanager configuration simply set this field.

```jsonnet mdox-exec="cat examples/alertmanager-config.jsonnet"
((import 'kube-prometheus/main.libsonnet') + {
   values+:: {
     alertmanager+: {
       config: |||
         global:
           resolve_timeout: 10m
         route:
           group_by: ['job']
           group_wait: 30s
           group_interval: 5m
           repeat_interval: 12h
           receiver: 'null'
           routes:
           - match:
               alertname: Watchdog
             receiver: 'null'
         receivers:
         - name: 'null'
       |||,
     },
   },
 }).alertmanager.secret
```

In the above example the configuration has been inlined, but can just as well be an external file imported in jsonnet via the `importstr` function.

```jsonnet mdox-exec="cat examples/alertmanager-config-external.jsonnet"
((import 'kube-prometheus/main.libsonnet') + {
   values+:: {
     alertmanager+: {
       config: importstr 'alertmanager-config.yaml',
     },
   },
 }).alertmanager.secret
```

---

### Customizations Components Name Namespace Overrides (`docs/customizations/components-name-namespace-overrides.md`)

### Components' name and namespace overrides

It is possible to override the namespace where kube-prometheus is going to be deployed, like the example below:

```jsonnet
local kp = (import 'kube-prometheus/main.libsonnet') +
{
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
};
```

If prefered, it can be changed individually by component. It is also possible to change the name of Prometheus and Alertmanager Custom Resources, like shown below:

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Customizations Developing Prometheus Rules And Grafana Dashboards (`docs/customizations/developing-prometheus-rules-and-grafana-dashboards.md`)

---
weight: 307
toc: true
title: Prometheus Rules and Grafana Dashboards
menu:
    docs:
        parent: kube
lead: This guide will help you adding Prometheus Rules and Grafana Dashboards on top of kube-prometheus
images: []
draft: false
description: This guide will help you adding Prometheus Rules and Grafana Dashboards on top of kube-prometheus
---

`kube-prometheus` ships with a set of default [Prometheus rules](https://prometheus.io/docs/prometheus/latest/configuration/recording_rules/) and [Grafana](http://grafana.com/) dashboards. At some point one might like to extend them, the purpose of this document is to explain how to do this.

All manifests of kube-prometheus are generated using [jsonnet](https://jsonnet.org/).
Prometheus rules and Grafana dashboards in specific follow the
[Prometheus Monitoring Mixins proposal](https://github.com/monitoring-mixins/docs/blob/master/design.pdf).

For both the Prometheus rules and the Grafana dashboards Kubernetes `ConfigMap`s are generated within kube-prometheus. In order to add additional rules and dashboards simply merge them onto the existing json objects. This document illustrates examples for rules as well as dashboards.

As a basis, all examples in this guide are based on the base example of the kube-prometheus [readme](https://github.com/prometheus-operator/kube-prometheus/blob/main/README.md):

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Prometheus rules

### Alerting rules

As per the [Prometheus Monitoring Mixins proposal](https://github.com/monitoring-mixins/docs/blob/master/design.pdf)
Prometheus alerting rules are under the key `prometheusAlerts` in the top level object.
Additional alerting rules can be added by merging into the existing object.

The format is exactly the Prometheus format, so there should be no changes necessary should you have existing rules that you want to include.

> Note that alerts can also be included into this file, using the jsonnet `import` function.
> In this example it is just inlined in order to demonstrate their use in a single file.

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Recording rules

In order to add a recording rule, simply do the same with the `prometheusRules` field.

> Note that rules can just as well be included into this file, using the jsonnet `import` function.
> In this example it is just inlined in order to demonstrate their use in a single file.

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Pre-rendered rules

We acknowledge, that users may need to transition existing rules, and therefore allow an option to add additional pre-rendered rules. Luckily the yaml and json formats are very close so the yaml rules just need to be converted to json without any manual interaction needed. Just a tool to convert yaml to json is needed:

```
go get -u -v github.com/brancz/gojsontoyaml
```

And convert the existing rule file:

```
cat existingrule.yaml | gojsontoyaml -yamltojson > existingrule.json
```

Then import it in jsonnet:

```jsonnet mdox-exec="cat examples/prometheus-additional-rendered-rule-example.jsonnet"
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
  exampleApplication: {
    prometheusRuleExample: {
      apiVersion: 'monitoring.coreos.com/v1',
      kind: 'PrometheusRule',
      metadata: {
        name: 'my-prometheus-rule',
        namespace: $.values.common.namespace,
      },
      spec: {
        groups: (import 'existingrule.json').groups,
      },
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['prometheus-adapter-' + name]: kp.prometheusAdapter[name] for name in std.objectFields(kp.prometheusAdapter) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ ['example-application-' + name]: kp.exampleApplication[name] for name in std.objectFields(kp.exampleApplication) }
```

### Changing default rules

Along with adding additional rules, we give the user the option to filter or adjust the existing rules imported by `kube-prometheus/main.libsonnet`.
The recording rules can be found in [kube-prometheus/components/mixin/rules](https://github.com/prometheus-operator/kube-prometheus/tree/main/jsonnet/kube-prometheus/components/mixin/rules)
and [kubernetes-mixin/rules](https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/rules).
The alerting rules can be found in [kube-prometheus/components/mixin/alerts](https://github.com/prometheus-operator/kube-prometheus/tree/main/jsonnet/kube-prometheus/components/mixin/alerts)
and [kubernetes-mixin/alerts](https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/alerts).

Knowing which rules to change, the user can now use functions from the [Jsonnet standard library](https://jsonnet.org/ref/stdlib.html) to make these changes.
Below are examples of both a filter and an adjustment being made to the default rules.
These changes can be assigned to a local variable and then added to the `local kp` object as seen in the examples above.

#### Filter

Here the alert `KubeStatefulSetReplicasMismatch` is being filtered out of the group `kubernetes-apps`.
The default rule can be seen [here](https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/alerts/apps_alerts.libsonnet).
You first need to find out in which component the rule is defined (here it is kuberentesControlPlane).

```jsonnet
local filter = {
  kubernetesControlPlane+: {
    prometheusRule+: {
      spec+: {
        groups: std.map(
          function(group)
            if group.name == 'kubernetes-apps' then
              group {
                rules: std.filter(
                  function(rule)
                    rule.alert != 'KubeStatefulSetReplicasMismatch',
                  group.rules
                ),
              }
            else
              group,
          super.groups
        ),
      },
    },
  },
};
```

#### Adjustment

Here the expression for another alert in the same component is updated from its previous value.
The default rule can be seen [here](https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/alerts/apps_alerts.libsonnet).

```jsonnet
local update = {
  kubernetesControlPlane+: {
    prometheusRule+: {
      spec+: {
        groups: std.map(
          function(group)
            if group.name == 'kubernetes-apps' then
              group {
                rules: std.map(
                  function(rule)
                    if rule.alert == 'KubePodCrashLooping' then
                      rule {
                        expr: 'rate(kube_pod_container_status_restarts_total{namespace=kube-system,job="kube-state-metrics"}[10m]) * 60 * 5 > 0',
                      }
                    else
                      rule,
                  group.rules
                ),
              }
            else
              group,
          super.groups
        ),
      },
    },
  },
};
```

Using the example from above about adding in pre-rendered rules, the new local variables can be added in as follows:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Dashboards

Dashboards can either be added using jsonnet or simply a pre-rendered json dashboard.

### Jsonnet dashboard

We recommend using the [grafonnet](https://github.com/grafana/grafonnet-lib/) library for jsonnet,
which gives you a simple DSL to generate Grafana dashboards.
Following the [Prometheus Monitoring Mixins proposal](https://github.com/monitoring-mixins/docs/blob/master/design.pdf)
additional dashboards are added to the `grafanaDashboards` key, located in the top level object.
To add new jsonnet dashboards, simply add one.

> Note that dashboards can just as well be included into this file, using the jsonnet `import` function.
> In this example it is just inlined in order to demonstrate their use in a single file.

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Pre-rendered Grafana dashboards

As jsonnet is a superset of json, the jsonnet `import` function can be used to include Grafana dashboard json blobs.
In this example we are importing a [provided example dashboard](https://github.com/prometheus-operator/kube-prometheus/tree/main/examples/example-grafana-dashboard.json).

```jsonnet mdox-exec="cat examples/grafana-additional-rendered-dashboard-example.jsonnet"
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+:: {
      namespace: 'monitoring',
    },
    grafana+: {
      dashboards+:: {  // use this method to import your dashboards to Grafana
        'my-dashboard.json': (import 'example-grafana-dashboard.json'),
      },
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
```

In case you have lots of json dashboard exported out from grafana UI the above approach is going to take lots of time.
To improve performance we can use `rawDashboards` field and provide it's value as json string by using `importstr`

```jsonnet mdox-exec="cat examples/grafana-additional-rendered-dashboard-example-2.jsonnet"
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+:: {
      namespace: 'monitoring',
    },
    grafana+: {
      rawDashboards+:: {
        'my-dashboard.json': (importstr 'example-grafana-dashboard.json'),
      },
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
```

### Mixins

Kube-prometheus comes with a couple of default mixins as the Kubernetes-mixin and the Node-exporter mixin,
however there [are many more mixins](https://monitoring.mixins.dev/).
To use other mixins, kube-prometheus has a jsonnet library for creating a PrometheusRule CRD and Grafana dashboards from a mixin.
Below is an example of creating a mixin object that has Prometheus rules and Grafana dashboards:

```jsonnet
// Import the library function for adding mixins
local addMixin = (import 'kube-prometheus/lib/mixin.libsonnet');

// Create your mixin
local myMixin = addMixin({
  name: 'myMixin',
  mixin: import 'my-mixin/mixin.libsonnet',
});
```

The myMixin object will have two objects - `prometheusRules` and `grafanaDashboards`. The `grafanaDashboards` object will be needed to be added to the `dashboards` field as in the example below:

```jsonnet
values+:: {
  grafana+:: {
    dashboards+:: myMixin.grafanaDashboards
```

The `prometheusRules` object is a PrometheusRule CRD. It should be defined as its own jsonnet object.
If you define multiple mixins in a single jsonnet object, there is a possibility that they will overwrite each others'
configuration and there will be unintended effects.
Therefore, use the `prometheusRules` object as its own jsonnet object:

```jsonnet
...
{ ['kubernetes-' + name]: kp.kubernetesControlPlane[name] for name in std.objectFields(kp.kubernetesControlPlane) }
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ 'external-mixins/my-mixin-prometheus-rules': myMixin.prometheusRules } // one object for each mixin
```

As mentioned above each mixin is configurable and you would configure the mixin as in the example below:

```jsonnet
local myMixin = addMixin({
  name: 'myMixin',
  mixin: (import 'my-mixin/mixin.libsonnet') + {
    _config+:: {
      myMixinSelector: 'my-selector',
      interval: '30d', // example
    },
  },
});
```

The library has also two optional parameters - the namespace for the `PrometheusRule` CRD and the dashboard folder for the Grafana dashboards.
The below example shows how to use both:

```jsonnet
local myMixin = addMixin({
  name: 'myMixin',
  namespace: 'prometheus', // default is monitoring
  dashboardFolder: 'Observability',
  mixin: (import 'my-mixin/mixin.libsonnet') + {
    _config+:: {
      myMixinSelector: 'my-selector',
      interval: '30d', // example
    },
  },
});
```

The created `prometheusRules` object will have the metadata field `namespace` added and the usage will remain the same.
However, the `grafanaDasboards` will be added to the `folderDashboards` field instead of the `dashboards` field as shown in the example below:

```jsonnet
values+:: {
  grafana+:: {
    folderDashboards+:: {
        Kubernetes: {
            ...
        },
        Misc: {
            'grafana-home.json': import 'dashboards/misc/grafana-home.json',
        },
    } + myMixin.grafanaDashboards
```

Full example of including etcd mixin using method described above:

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Customizations Dropping Unwanted Dashboards (`docs/customizations/dropping-unwanted-dashboards.md`)

### Dropping unwanted dashboards

When deploying kube-prometheus, your Grafana instance is deployed with a lot of dashboards by default. All those dashboards are comming from upstream projects like [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin), [prometheus-mixin](https://github.com/prometheus/prometheus/tree/main/documentation/prometheus-mixin) and [node-exporter-mixin](https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin), among others.

In case you find out that you don't need some of them, you can choose to remove those dashboards like in the example below, which removes the [`alertmanager-overview.json`](https://github.com/prometheus/alertmanager/blob/main/doc/alertmanager-mixin/dashboards/overview.libsonnet) dashboard.

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Customizations Exposing Prometheus Alertmanager Grafana Ingress (`docs/customizations/exposing-prometheus-alertmanager-grafana-ingress.md`)

---
weight: 303
toc: true
title: Expose via Gateway API
menu:
    docs:
        parent: kube
lead: This guide will help you exposing Prometheus, Alertmanager and Grafana using the Gateway API.
images: []
draft: false
description: This guide will help you exposing Prometheus, Alertmanager and Grafana using the Gateway API.
---

In order to access the web interfaces via the Internet, the [Gateway API](https://gateway-api.sigs.k8s.io/) is the recommended option. This guide explains how the Gateway API can be used to expose the Prometheus, Alertmanager and Grafana UIs, that are included in the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) project.

Note: before continuing, it is recommended to first get familiar with the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) stack by itself.

## Prerequisites

Apart from a running Kubernetes cluster with a running [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) stack, a [Gateway API](https://gateway-api.sigs.k8s.io/) controller must be installed and a `GatewayClass` must be available. The Gateway API is implemented by a number of controllers, for example Istio, Envoy Gateway and Contour; refer to the [list of implementations](https://gateway-api.sigs.k8s.io/) and follow your controller's installation instructions. The `gatewayClassName` referenced below must match a `GatewayClass` that exists in your cluster.

## Setting up routing

The Gateway API splits the configuration into two resources: a `Gateway`, which defines the entry point (its listeners, ports and protocols) and is usually created once and shared, and an `HTTPRoute`, which attaches to a `Gateway` and forwards traffic to a Service. The setup of `HTTPRoute` objects is the same for Prometheus, Alertmanager and Grafana. Therefore this guide demonstrates it in detail for Prometheus as it can easily be adapted for the other applications.

As monitoring data may contain sensitive data, these endpoints should be protected with authentication. The Gateway API does not define a portable authentication mechanism. Authentication is instead configured through the APIs of your Gateway controller — for example an external authorization filter or a policy attached to the `Gateway` or `HTTPRoute` — or by placing an authenticating proxy such as [oauth2-proxy](https://github.com/oauth2-proxy/oauth2-proxy) in front of the Services. Consult the documentation of your Gateway API implementation for the supported options.

Also, the applications provide external links to themselves in alerts and various places. When a Gateway is used in front of the applications these links need to be based on the external URL's. This can be configured for each application in jsonnet.

```
/* Detailed source-code truncated for AI context efficiency. */
```

In order to expose Alertmanager and Grafana, simply create additional `HTTPRoute` objects, but simply pointing at the `alertmanager-main` or `grafana` Service instead of the `prometheus-k8s` Service. Make sure to also use the correct numeric port respectively, for Alertmanager it is `9093`, for Grafana it is `3000`. Be sure to also specify the appropriate external URL. Note that the external URL for Grafana is set in a different way than the external URL for Prometheus or Alertmanager. See [ingress.jsonnet](https://github.com/prometheus-operator/kube-prometheus/tree/main/examples/ingress.jsonnet) for how to set the Grafana external URL.

In order to render the Gateway and `HTTPRoute` objects similar to the other objects use as demonstrated in the [main readme](https://github.com/prometheus-operator/kube-prometheus/tree/main/README.md):

```jsonnet
{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) } +
{ ['gateway-api-' + name]: kp.gatewayAPI[name] for name in std.objectFields(kp.gatewayAPI) }
```

Note, that in comparison only the last line was added, the rest is identical to the original.

See [ingress.jsonnet](https://github.com/prometheus-operator/kube-prometheus/tree/main/examples/ingress.jsonnet) for an example implementation.

## Adding the Gateway namespace to NetworkPolicies

NetworkPolicies restricting access to the components are added by default. These can either be removed as in
[networkpolicies-disabled.jsonnet](https://github.com/prometheus-operator/kube-prometheus/tree/main/examples/networkpolicies-disabled.jsonnet) or modified to allow traffic from the namespace where your Gateway controller's data plane runs, as
described here.

This is an example for Alertmanager, but the same can be applied to Grafana and Prometheus. Replace `gateway-system` with the namespace of your Gateway controller.

```jsonnet
{
  alertmanager+:: {
    networkPolicy+: {
      spec+: {
        ingress: [
          super.ingress[0] + {
            from+: [
              {
                namespaceSelector: {
                  matchLabels: {
                    'kubernetes.io/metadata.name': 'gateway-system',
                  },
                },
              },
            ],
          },
        ] + super.ingress[1:],
      },
    },
  },
}
```

---

### Customizations Metrics Server (`docs/customizations/metrics-server.md`)

### Using metrics-server instead of prometheus-adapter

By default, kube-prometheus deploys [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter) to serve the Kubernetes resource metrics API (`metrics.k8s.io`). This is a full-featured solution that derives CPU and memory metrics from Prometheus queries, and can also serve custom and external metrics APIs.

However, for use cases where only the resource metrics API is needed (for example, when using [KEDA](https://keda.sh/) for custom metrics scaling), the lightweight upstream [metrics-server](https://github.com/kubernetes-sigs/metrics-server) may be preferable.

> [!IMPORTANT]
> prometheus-adapter is [being deprecated](https://github.com/kubernetes-sigs/prometheus-adapter/issues/701) by SIG Instrumentation. The project recommends migrating to KEDA for custom/external metrics use cases. Since metrics-server handles the resource metrics API natively without requiring Prometheus queries, it is the natural replacement for the resource metrics portion. We plan to make `metrics-server` the default `resourceMetricsAPI` in a future release of kube-prometheus.

#### Switching to metrics-server

Set the `resourceMetricsAPI` field in `values.common` to `'metrics-server'`:

```jsonnet mdox-exec="cat examples/jsonnet-snippets/metrics-server.jsonnet"
(import 'kube-prometheus/main.libsonnet') +
{
  values+:: {
    common+: {
      resourceMetricsAPI:: 'metrics-server',
    },
    metricsServer+: {
      kubeletInsecureTLS:: true,
    },
  },
}
```

This replaces the prometheus-adapter manifests with metrics-server manifests. Only one resource metrics API provider can be active at a time since they both register the `v1beta1.metrics.k8s.io` APIService.

#### When to set kubeletInsecureTLS

On clusters that do not use proper kubelet serving certificates (for example, kind, minikube, or clusters without a kubelet-serving CA), set `kubeletInsecureTLS:: true`. On production clusters with a properly configured kubelet PKI, this should be left at the default (`false`).

#### Emitting manifests

If you are writing your own top-level jsonnet (instead of using `example.jsonnet`), emit the resource metrics manifests using the helper library:

```jsonnet
(import 'kube-prometheus/lib/resource-metrics-api.libsonnet')(kp)
```

This automatically selects the correct component based on the `resourceMetricsAPI` flag.

#### Single-node clusters

The default configuration deploys 2 replicas with hard pod anti-affinity, which requires at least 2 nodes. On single-node clusters (minikube, single-node kind, etc.), set `replicas:: 1` or switch to soft anti-affinity:

```jsonnet
values+:: {
  metricsServer+: {
    replicas:: 1,
  },
},
```

#### Available configuration

The following hidden fields can be overridden in `values.metricsServer`:

| Field                        | Default                                      | Description                                   |
|------------------------------|----------------------------------------------|-----------------------------------------------|
| `replicas`                   | `2`                                          | Number of metrics-server replicas             |
| `resources`                  | `{requests: {cpu: '100m', memory: '200Mi'}}` | Resource requests/limits                      |
| `kubeletInsecureTLS`         | `false`                                      | Skip kubelet TLS verification                 |
| `metricResolution`           | `'15s'`                                      | How often metrics are scraped from kubelets   |
| `securePort`                 | `10250`                                      | Metrics-server HTTPS port                     |
| `extraArgs`                  | `[]`                                         | Additional command-line arguments             |
| `podAntiAffinity`            | `'hard'`                                     | Pod anti-affinity type (`'hard'` or `'soft'`) |
| `podAntiAffinityTopologyKey` | `'kubernetes.io/hostname'`                   | Topology key for anti-affinity                |
| `insecureSkipTLSVerify`      | `true`                                       | APIService TLS verification                   |
| `priorityClassName`          | `'system-cluster-critical'`                  | Pod priority class                            |

#### Generating and deploying manifests

To generate manifests using the provided example:

```bash
make manifests-metrics-server
```

This builds from `examples/metrics-server.jsonnet` and outputs YAML manifests to `manifests/`. Then deploy:

```bash
kubectl apply --server-side -f manifests/setup
kubectl apply -f manifests/
```

#### Full example

See [examples/metrics-server.jsonnet](../../examples/metrics-server.jsonnet) for a complete working example that deploys the full kube-prometheus stack with metrics-server instead of prometheus-adapter.

---

### Customizations Monitoring Additional Namespaces (`docs/customizations/monitoring-additional-namespaces.md`)

### Monitoring additional namespaces

In order to monitor additional namespaces, the Prometheus server requires the appropriate `Role` and `RoleBinding` to be able to discover targets from that namespace. By default the Prometheus server is limited to the three namespaces it requires: default, kube-system and the namespace you configure the stack to run in via `$.values.namespace`. This is specified in `$.values.prometheus.namespaces`, to add new namespaces to monitor, simply append the additional namespaces:

```jsonnet mdox-exec="cat examples/additional-namespaces.jsonnet"
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },

    prometheus+: {
      namespaces+: ['my-namespace', 'my-second-namespace'],
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
```

#### Defining the ServiceMonitor for each additional Namespace

In order to Prometheus be able to discovery and scrape services inside the additional namespaces specified in previous step you need to define a ServiceMonitor resource.

> Typically it is up to the users of a namespace to provision the ServiceMonitor resource, but in case you want to generate it with the same tooling as the rest of the cluster monitoring infrastructure, this is a guide on how to achieve this.

You can define ServiceMonitor resources in your `jsonnet` spec. See the snippet bellow:

```
/* Detailed source-code truncated for AI context efficiency. */
```

> NOTE: make sure your service resources have the right labels (eg. `'app': 'myapp'`) applied. Prometheus uses kubernetes labels to discover resources inside the namespaces.

---

### Customizations Monitoring All Namespaces (`docs/customizations/monitoring-all-namespaces.md`)

### Monitoring all namespaces

In case you want to monitor all namespaces in a cluster, you can add the following mixin. Also, make sure to empty the namespaces defined in prometheus so that roleBindings are not created against them.

```jsonnet mdox-exec="cat examples/all-namespaces.jsonnet"
local kp = (import 'kube-prometheus/main.libsonnet') +
           (import 'kube-prometheus/addons/all-namespaces.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
    prometheus+: {
      namespaces: [],
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
```

> NOTE: This configuration can potentially make your cluster insecure especially in a multi-tenant cluster. This is because this gives Prometheus visibility over the whole cluster which might not be expected in a scenario when certain namespaces are locked down for security reasons.

Proceed with [creating ServiceMonitors for the services in the namespaces](monitoring-additional-namespaces.md#defining-the-servicemonitor-for-each-additional-namespace) you actually want to monitor

---

### Customizations Node Ports (`docs/customizations/node-ports.md`)

### NodePorts

Another mixin that may be useful for exploring the stack is to expose the UIs of Prometheus, Alertmanager and Grafana on NodePorts:

```jsonnet mdox-exec="cat examples/jsonnet-snippets/node-ports.jsonnet"
(import 'kube-prometheus/main.libsonnet') +
(import 'kube-prometheus/addons/node-ports.libsonnet')
```

---

### Customizations Perses (`docs/customizations/perses.md`)

---
weight: 320
toc: true
title: Using Perses instead of Grafana
menu:
    docs:
        parent: kube
lead: Replace Grafana with Perses for Kubernetes cluster dashboards
images: []
draft: false
description: Replace Grafana with Perses for Kubernetes cluster dashboards using the Perses addon
---

> [!NOTE]
> Perses is currently integrated as an addon. Once the integration is mature, it may be promoted to a built-in toggle (similar to the `metrics-server` / `prometheus-adapter` switch).

[Perses](https://perses.dev) is a CNCF Sandbox observability visualization platform. The `perses` addon deploys Perses and Kubernetes observability dashboards from [community-mixins](https://github.com/perses/community-mixins) via the [perses-operator](https://github.com/perses/perses-operator). In jsonnet, Grafana is omitted by setting `grafana: {}` (see [`examples/perses.jsonnet`](../../examples/perses.jsonnet)). On an existing cluster, `kubectl apply` does not delete Grafana — remove it explicitly after you are ready (see [Remove Grafana](#remove-grafana-optional)).

## What the addon deploys

| Resource              | Kind                             | Description                                                                       |
|-----------------------|----------------------------------|-----------------------------------------------------------------------------------|
| perses-operator       | Deployment, RBAC, ServiceAccount | Manages Perses CRs declaratively                                                  |
| 4 CRDs                | CustomResourceDefinition         | `Perses`, `PersesDashboard`, `PersesDatasource`, `PersesGlobalDatasource`         |
| Perses instance       | `Perses` CR                      | Runs the Perses server (port 8080)                                                |
| Prometheus datasource | `PersesGlobalDatasource` CR      | Proxies queries to `prometheus-k8s` Service                                       |
| Dashboards            | `PersesDashboard` CRs            | Default community-mixins packages (see [Dashboard coverage](#dashboard-coverage)) |
| ServiceMonitor        | `ServiceMonitor`                 | Scrapes operator metrics                                                          |
| PrometheusRule        | `PrometheusRule`                 | Operator alerting rules                                                           |

The addon also patches the existing Prometheus `NetworkPolicy` so Perses pods can query Prometheus.

> [!NOTE]
> **Security:** Perses runs with authentication disabled by default, matching the bundled Grafana setup in kube-prometheus. Enable auth and secure cookies for production deployments.
>
> **Storage:** Dashboards and datasource configuration are stored in Kubernetes CRs. The Perses server uses ephemeral file storage for runtime state; pod restarts do not remove `PersesDashboard` or `PersesGlobalDatasource` objects.

## Generate and apply

Generate the full stack with Perses instead of Grafana:

```shell
make manifests-perses
```

> [!IMPORTANT]
> `make manifests-perses` regenerates the entire `manifests/` tree from [`examples/perses.jsonnet`](../../examples/perses.jsonnet) (not only Perses objects).

Apply the generated manifests:

```shell
# Apply CRDs and namespace first
kubectl apply --server-side -f manifests/setup
kubectl wait \
    --for condition=Established \
    --all CustomResourceDefinition \
    --namespace=monitoring

# Apply the remaining manifests
kubectl apply -f manifests/
```

### Access the Perses UI

```shell
kubectl --namespace monitoring port-forward svc/perses 8080:8080
```

Open Perses at [http://localhost:8080](http://localhost:8080). The operator creates a Service named after the `Perses` CR (`perses` by default). Dashboards and the Prometheus datasource are loaded from CRs.

### Using a custom jsonnet file

See [`examples/perses.jsonnet`](../../examples/perses.jsonnet). Generate with `./build.sh my-perses.jsonnet`.

## Remove Grafana (optional)

On an **existing** kube-prometheus cluster, `kubectl apply` does not remove resources that disappeared from the generated manifests, so Grafana keeps running alongside Perses. That is fine for evaluation — you can run both UIs in parallel while verifying coverage.

Once you are satisfied with Perses, delete Grafana by label:

```shell
kubectl -n monitoring delete --ignore-not-found=true \
  deployment,service,serviceaccount,servicemonitor,networkpolicy,prometheusrule,secret,configmap \
  -l app.kubernetes.io/name=grafana
```

## Customization

### Override versions

```jsonnet
{
  values+:: {
    common+: {
      versions+: {
        perses: '0.54.0',
        persesOperator: '0.5.0',
      },
    },
  },
}
```

### Override images (e.g. internal registry)

Default images are `persesdev/perses:v<version>` and `persesdev/perses-operator:v<version>`.

```jsonnet
{
  values+:: {
    common+: {
      images+: {
        perses: 'my-registry.example.com/perses:v0.54.0',
        persesOperator: 'my-registry.example.com/perses-operator:v0.5.0',
      },
    },
  },
}
```

### Dashboard query selectors

The Perses addon imports pre-generated dashboards from [community-mixins](https://github.com/perses/community-mixins), which use kubernetes-mixin default job labels. kube-prometheus overrides those selectors for Grafana via `kubernetesControlPlane`; the addon applies the same rewiring when importing dashboards:

| community-mixins       | kube-prometheus                                   |
|------------------------|---------------------------------------------------|
| `job="cadvisor"`       | `job="kubelet", metrics_path="/metrics/cadvisor"` |
| `job="kube-apiserver"` | `job="apiserver"`                                 |

The addon also sets `prometheus.externalLabels.cluster` to `kube-prometheus` so the `cluster` dashboard variable resolves (required by kubernetes-mixin dashboards). Override either if your scrape labels differ:

```jsonnet
{
  values+:: {
    prometheus+: {
      externalLabels+: {
        cluster: 'my-cluster',
      },
    },
    perses+: {
      cadvisorJobSelector: 'job="kubelet", metrics_path="/metrics/cadvisor"',
      kubeApiserverJobSelector: 'job="apiserver"',
    },
  },
}
```

### Change the Prometheus datasource URL

By default the datasource proxies queries through the Perses server to `http://prometheus-k8s.<namespace>.svc.cluster.local:9090` (see [`perses.libsonnet`](../../jsonnet/kube-prometheus/addons/perses.libsonnet)). Patch the generated component to use a different URL:

```jsonnet
local kp =
  (import 'kube-prometheus/main.libsonnet') +
  (import 'kube-prometheus/addons/perses.libsonnet') +
  {
    values+:: {
      common+: { namespace: 'monitoring' },
    },
    perses+: {
      prometheusGlobalDatasource+: {
        spec+: {
          config+: {
            plugin+: {
              spec+: {
                proxy+: {
                  spec+: {
                    url: 'http://my-prometheus.monitoring.svc.cluster.local:9090',
                  },
                },
              },
            },
          },
        },
      },
    },
  };
```

### Select dashboard packages

Override `dashboardComponents` to change which community-mixins packages are imported. Defaults are `kubernetes`, `prometheus`, `alertmanager`, and `node-exporter`.

Trim to a subset:

```jsonnet
{
  values+:: {
    perses+: {
      dashboardComponents: ['kubernetes', 'prometheus', 'alertmanager'],
    },
  },
}
```

Include optional packages (for example blackbox-exporter and Perses overview):

```jsonnet
{
  values+:: {
    perses+: {
      dashboardComponents: [
        'kubernetes',
        'prometheus',
        'alertmanager',
        'node-exporter',
        'blackbox-exporter',
        'perses',
      ],
    },
  },
}
```

Available component names match [community-mixins `dashboards.libsonnet`](https://github.com/perses/community-mixins/blob/main/jsonnet/dashboards.libsonnet).

## Dashboard coverage

> [!NOTE]
> The counts and component lists below reflect the state of [community-mixins](https://github.com/perses/community-mixins) vendored at the time of this release. Community-mixins is under active development and may add new dashboards or components at any time. Check the upstream repository for the latest coverage.

Dashboards are imported from community-mixins via its [`dashboards.libsonnet`](https://github.com/perses/community-mixins/blob/main/jsonnet/dashboards.libsonnet) helper (namespace, datasource, and labels), then rewired for kube-prometheus scrape labels.

### Default packages (23 `PersesDashboard` CRs)

| Component       | Perses dashboards                                         | kube-prometheus component |
|-----------------|-----------------------------------------------------------|---------------------------|
| `kubernetes`    | 18 Kubernetes / control-plane and workload dashboards     | `kubernetesControlPlane`  |
| `prometheus`    | `prometheus-overview`, `prometheus-remote-write`          | `prometheus`              |
| `alertmanager`  | `alertmanager-overview`                                   | `alertmanager`            |
| `node-exporter` | `node-exporter-nodes`, `node-exporter-cluster-use-method` | `nodeExporter`            |

### Optional packages

Enable these via `dashboardComponents` (see [Select dashboard packages](#select-dashboard-packages)). Thanos and etcd panels may be empty until those targets are scraped.

| Component           | Perses dashboards             | kube-prometheus component / addon               |
|---------------------|-------------------------------|-------------------------------------------------|
| `blackbox-exporter` | `blackbox-overview`           | `blackboxExporter`                              |
| `perses`            | `perses-overview`             | Perses addon                                    |
| `thanos`            | 6 Thanos component dashboards | Prometheus Thanos sidecar (when enabled)        |
| `etcd`              | `etcd-overview`               | `static-etcd` addon or external etcd monitoring |

These kube-prometheus components have **no Perses dashboard in community-mixins yet**: `kube-state-metrics`, `prometheus-operator`, `prometheus-adapter`, and `metrics-server`.

Authoritative configuration is in [`perses.libsonnet`](../../jsonnet/kube-prometheus/addons/perses.libsonnet). Run `make manifests-perses` after addon or mixin updates.

## Switching back to Grafana

To revert to Grafana, regenerate default manifests and reapply, then remove Perses resources that are no longer in the manifests:

```shell
make manifests
kubectl apply --server-side -f manifests/setup
kubectl wait --for condition=Established --all CustomResourceDefinition --namespace=monitoring
kubectl apply -f manifests/

# Perses CRs (instance and dashboards are namespaced; global datasource is cluster-scoped)
kubectl -n monitoring delete --ignore-not-found=true perses/perses
kubectl delete --ignore-not-found=true \
  persesglobaldatasources.perses.dev/prometheus-datasource
kubectl -n monitoring delete --ignore-not-found=true \
  persesdashboards.perses.dev --all

# Operator resources (all carry app.kubernetes.io/part-of=perses-operator)
kubectl -n monitoring delete --ignore-not-found=true \
  deployment,service,serviceaccount,servicemonitor,networkpolicy,prometheusrule,role,rolebinding \
  -l app.kubernetes.io/part-of=perses-operator
kubectl delete --ignore-not-found=true clusterrole,clusterrolebinding \
  -l app.kubernetes.io/part-of=perses-operator
```

Optionally, remove the `perses.dev` CRDs if you no longer need them:

```shell
kubectl delete --ignore-not-found=true crd \
  perses.perses.dev \
  persesdashboards.perses.dev \
  persesdatasources.perses.dev \
  persesglobaldatasources.perses.dev
```

## Testing

`make test-e2e-perses` runs Go end-to-end tests against a cluster that already has the Perses stack deployed. It does **not** generate or apply manifests — follow [Generate and apply](#generate-and-apply) first, then:

```shell
export KUBECONFIG=~/.kube/config   # point at your test cluster
make test-e2e-perses
```

CI follows the same order: generate manifests, deploy to a kind cluster, then run `make test-e2e-perses` (see [`.github/workflows/ci.yaml`](../../.github/workflows/ci.yaml)).

## References

- [Perses project](https://perses.dev)
- [perses-operator](https://github.com/perses/perses-operator)
- [community-mixins](https://github.com/perses/community-mixins)

---

### Customizations Platform Specific (`docs/customizations/platform-specific.md`)

### Running kube-prometheus on specific platforms

A common example is that not all Kubernetes clusters are created exactly the same way, meaning the configuration to monitor them may be slightly different. For the following clusters there are mixins available to easily configure them:

* aws
* bootkube
* eks
* gke
* kops
* kops_coredns
* kubeadm
* kubespray

These mixins are selectable via the `platform` field of kubePrometheus:

```jsonnet mdox-exec="cat examples/jsonnet-snippets/platform.jsonnet"
(import 'kube-prometheus/main.libsonnet') +
{
  values+:: {
    common+: {
      platform: 'example-platform',
    },
  },
}
```

---

### Customizations Pod Anti Affinity (`docs/customizations/pod-anti-affinity.md`)

### Pod Anti-Affinity

To prevent `Prometheus` and `Alertmanager` instances from being deployed onto the same node when
possible, one can include the [kube-prometheus-anti-affinity.libsonnet](https://github.com/prometheus-operator/kube-prometheus/tree/main/jsonnet/kube-prometheus/addons/anti-affinity.libsonnet) mixin:

```jsonnet mdox-exec="cat examples/anti-affinity.jsonnet"
local kp = (import 'kube-prometheus/main.libsonnet') +
           (import 'kube-prometheus/addons/anti-affinity.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
```

---

### Customizations Prometheus Agent (`docs/customizations/prometheus-agent.md`)

### Prometheus-agent mode

***ATTENTION***: Although it is possible to run Prometheus in Agent mode with Prometheus-Operator, it requires strategic merge patches. This practice is not recommended and we do not provide support if Prometheus doesn't work as you expect. **Try it at your own risk!**

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Customizations Static Etcd Configuration (`docs/customizations/static-etcd-configuration.md`)

### Static etcd configuration

In order to configure a static etcd cluster to scrape there is a simple [static-etcd.libsonnet](https://github.com/prometheus-operator/kube-prometheus/tree/main/jsonnet/kube-prometheus/addons/static-etcd.libsonnet) mixin prepared.

An example of how to use it can be seen below:

```
/* Detailed source-code truncated for AI context efficiency. */
```

If you'd like to monitor an etcd instance that lives outside the cluster, see [Monitoring external etcd](../monitoring-external-etcd.md) for more information.

> Note that monitoring etcd in minikube is currently not possible because of how etcd is setup. (minikube's etcd binds to 127.0.0.1:2379 only, and within host networking namespace.)

---

### Customizations Strip Limits (`docs/customizations/strip-limits.md`)

### Stripping container resource limits

Sometimes in small clusters, the CPU/memory limits can get high enough for alerts to be fired continuously. To prevent this, one can strip off the predefined limits.
To do that, one can import the following mixin

```jsonnet mdox-exec="cat examples/strip-limits.jsonnet"
local kp = (import 'kube-prometheus/main.libsonnet') +
           (import 'kube-prometheus/addons/strip-limits.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
};

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
```

---

### Customizations Using Custom Container Registry (`docs/customizations/using-custom-container-registry.md`)

### Internal Registry

Some Kubernetes installations source all their images from an internal registry. kube-prometheus supports this use case and helps the user synchronize every image it uses to the internal registry and generate manifests pointing at the internal registry.

To produce the `docker pull/tag/push` commands that will synchronize upstream images to `internal-registry.com/organization` (after having run the `jb` command to populate the vendor directory):

```shell
$ jsonnet -J vendor -S --tla-str repository=internal-registry.com/organization examples/sync-to-internal-registry.jsonnet
$ docker pull k8s.gcr.io/addon-resizer:1.8.4
$ docker tag k8s.gcr.io/addon-resizer:1.8.4 internal-registry.com/organization/addon-resizer:1.8.4
$ docker push internal-registry.com/organization/addon-resizer:1.8.4
$ docker pull quay.io/prometheus/alertmanager:v0.16.2
$ docker tag quay.io/prometheus/alertmanager:v0.16.2 internal-registry.com/organization/alertmanager:v0.16.2
$ docker push internal-registry.com/organization/alertmanager:v0.16.2
...
```

The output of this command can be piped to a shell to be executed by appending `| sh`.

Then to generate manifests with `internal-registry.com/organization`, use the `withImageRepository` mixin:

```jsonnet mdox-exec="cat examples/internal-registry.jsonnet"
local mixin = import 'kube-prometheus/addons/config-mixins.libsonnet';
local kp = (import 'kube-prometheus/main.libsonnet') + {
  values+:: {
    common+: {
      namespace: 'monitoring',
    },
  },
} + mixin.withImageRepository('internal-registry.com/organization');

{ ['00namespace-' + name]: kp.kubePrometheus[name] for name in std.objectFields(kp.kubePrometheus) } +
{ ['0prometheus-operator-' + name]: kp.prometheusOperator[name] for name in std.objectFields(kp.prometheusOperator) } +
{ ['node-exporter-' + name]: kp.nodeExporter[name] for name in std.objectFields(kp.nodeExporter) } +
{ ['kube-state-metrics-' + name]: kp.kubeStateMetrics[name] for name in std.objectFields(kp.kubeStateMetrics) } +
{ ['alertmanager-' + name]: kp.alertmanager[name] for name in std.objectFields(kp.alertmanager) } +
{ ['prometheus-' + name]: kp.prometheus[name] for name in std.objectFields(kp.prometheus) } +
{ ['grafana-' + name]: kp.grafana[name] for name in std.objectFields(kp.grafana) }
```

---

--- METRICS ---
- Files Extracted: 38
- Estimated Token Budget: ~31236 tokens
- Canonical Reference: https://codewiki.google/github.com/prometheus-operator/kube-prometheus