## 1. Project Overview & Quickstart (grafana/mimir) ## File: README.md # Grafana Mimir Grafana Mimir is an open source software project that provides a scalable long-term storage for [Prometheus](https://prometheus.io). Some of the core strengths of Grafana Mimir include: - **Easy to install and maintain:** Grafana Mimir’s extensive documentation, tutorials, and deployment tooling make it quick to get started. Using its monolithic mode, you can get Grafana Mimir up and running with just one binary and no additional dependencies. Once deployed, the best-practice dashboards, alerts, and runbooks packaged with Grafana Mimir make it easy to monitor the health of the system. - **Massive scalability:** You can run Grafana Mimir's horizontally-scalable architecture across multiple machines, resulting in the ability to process orders of magnitude more time series than a single Prometheus instance. Internal testing shows that Grafana Mimir handles up to 1 billion active time series. - **Global view of metrics:** Grafana Mimir enables you to run queries that aggregate series from multiple Prometheus instances, giving you a global view of your systems. Its query engine extensively parallelizes query execution, so that even the highest-cardinality queries complete with blazing speed. - **Cheap, durable metric storage:** Grafana Mimir uses object storage for long-term data storage, allowing it to take advantage of this ubiquitous, cost-effective, high-durability technology. It is compatible with multiple object store implementations, including AWS S3, Google Cloud Storage, Azure Blob Storage, OpenStack Swift, as well as any S3-compatible object storage. - **High availability:** Grafana Mimir replicates incoming metrics, ensuring that no data is lost in the event of machine failure. Its horizontally scalable architecture also means that it can be restarted, upgraded, or downgraded with zero downtime, which means no interruptions to metrics ingestion or querying. - **Natively multi-tenant:** Grafana Mimir’s multi-tenant architecture enables you to isolate data and queries from independent teams or business units, making it possible for these groups to share the same cluster. Advanced limits and quality-of-service controls ensure that capacity is shared fairly among tenants. ## Migrating to Grafana Mimir If you're migrating to Grafana Mimir, refer to the following documents: - [Migrating from Thanos or Prometheus to Grafana Mimir](https://grafana.com/docs/mimir/latest/set-up/migrate/migrate-from-thanos-or-prometheus/). - [Migrating from Cortex to Grafana Mimir](https://grafana.com/docs/mimir/latest/set-up/migrate/migrate-from-cortex/) ## Deploying Grafana Mimir For information about how to deploy Grafana Mimir, refer to [Deploy Grafana Mimir](https://grafana.com/docs/mimir/latest/operators-guide/deploy-grafana-mimir/). ## Getting started If you’re new to Grafana Mimir, read the [Get started guide](https://grafana.com/docs/mimir/latest/get-started/). Before deploying Grafana Mimir in a production environment, read: 1. [An overview of Grafana Mimir’s architecture](https://grafana.com/docs/mimir/latest/operators-guide/architecture/) 1. [Configure Grafana Mimir](https://grafana.com/docs/mimir/latest/operators-guide/configure/) 1. [Run Grafana Mimir in production](https://grafana.com/docs/mimir/latest/operators-guide/run-production-environment/) ## Documentation Refer to the following links to access Grafana Mimir documentation: - [Latest release](https://grafana.com/docs/mimir/latest/) - [Upcoming release](https://grafana.com/docs/mimir/next/), at the tip of the `main` branch ## Contributing To contribute to Grafana Mimir, refer to [Contributing to Grafana Mimir](https://github.com/grafana/mimir/tree/main/docs/internal/contributing). ## Join the Grafana Mimir discussion If you have any questions or feedback regarding Grafana Mimir, join the [Grafana Mimir Discussion](https://github.com/grafana/mimir/discussions). Alternatively, consider joining the monthly [Grafana Mimir Community Call](https://docs.google.com/document/d/1E4jJcGicvLTyMEY6cUFFZUg_I8ytrBuW8r5yt1LyMv4). Your feedback is always welcome, and you can also share it via the [`#mimir` Slack channel](https://slack.grafana.com/). ## License Grafana Mimir is distributed under [AGPL-3.0-only](LICENSE). --- ## File: docs/README.md # Documentation ## Test Mimir and Helm chart documentation together locally Testing the documentation accurately requires a copy of Mimir documentation at the target version of the Helm chart. One way to do this is to use a git worktree to checkout the version branch of Mimir that matches the Helm chart target version. For example, if the Helm chart target version is `v2.9.x`, checkout the `release-2.9` branch: ```console $ git worktree add --checkout "$(git rev-parse --show-toplevel)/../mimir-v2.9.x" origin/release-2.9 Preparing worktree (detached HEAD 761114d8b) HEAD is now at 761114d8b Bump version to 2.9.0 (#5289) ``` ## 2. Official Technical Reference & Guides (grafana/docs) # Prometheus Monitoring Mixins > NOTE: This project is *beta* stage. A mixin is a set of Grafana dashboards and Prometheus rules and alerts, packaged together in a reuseable and extensible bundle. Mixins are written in [jsonnet](https://jsonnet.org/), and are typically installed and updated with [jsonnet-bundler](https://github.com/jsonnet-bundler/jsonnet-bundler). For more information about mixins, see: * [Prometheus Monitoring Mixins Design Doc](https://docs.google.com/document/d/1A9xvzwqnFVSOZ5fD3blKODXfsat5fg6ZhnKu9LK3lB4/view). A [cached pdf](design.pdf) is included in this repo. * For more motivation, see "[The RED Method: How to instrument your services](https://kccncna17.sched.com/event/CU8K/the-red-method-how-to-instrument-your-services-b-tom-wilkie-kausal?iframe=no&w=100%&sidebar=yes&bg=no)" talk from CloudNativeCon Austin 2018. The KLUMPs system demo'd became the basis for the kubernetes-mixin. * "[Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts and Exporters](https://www.youtube.com/watch?v=b7-DtFfsL6E)" talk from CloudNativeCon Copenhagen 2018. * "[Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts and Exporters](https://promcon.io/2018-munich/talks/prometheus-monitoring-mixins/)" talk from PromCon 2018 (slightly updated). ## How to use mixins. Mixins are designed to be vendored into the repo with your infrastructure config. To do this, use [jsonnet-bundler](https://github.com/jsonnet-bundler/jsonnet-bundler): You then have three options for deploying your dashboards 1. Generate the config files and deploy them yourself. 1. Use ksonnet to deploy this mixin along with Prometheus and Grafana. 1. Use kube-prometheus to deploy this mixin. ## Generate config files You can manually generate the alerts, dashboards and rules files, but first you must install some tools: ``` $ go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest # macOS $ brew install jsonnet-bundler # Archlinux AUR $ yay -S jsonnet ``` Then, grab the mixin and its dependencies: ``` $ git clone https://github.com// $ cd $ jb install ``` Finally, build the mixin: ``` $ make prometheus_alerts.yaml $ make prometheus_rules.yaml $ make dashboards_out ``` The `prometheus_alerts.yaml` and `prometheus_rules.yaml` file then need to passed to your Prometheus server, and the files in `dashboards_out` need to be imported into you Grafana server. The exact details will depending on how you deploy your monitoring stack to Kubernetes. ## Using with prometheus-ksonnet Alternatively you can also use the mixin with [prometheus-ksonnet](https://github.com/grafana/jsonnet-libs/tree/master/prometheus-ksonnet), a [ksonnet](https://github.com/ksonnet/ksonnet) module to deploy a fully-fledged Prometheus-based monitoring system for Kubernetes: Make sure you have the ksonnet v0.8.0: ``` $ brew install https://raw.githubusercontent.com/ksonnet/homebrew-tap/82ef24cb7b454d1857db40e38671426c18cd8820/ks.rb $ brew pin ks $ ks version ksonnet version: v0.8.0 jsonnet version: v0.9.5 client-go version: v1.6.8-beta.0+$Format:%h$ ``` In your config repo, if you don't have a ksonnet application, make a new one (will copy credentials from current context): ``` $ ks init $ cd $ ks env add default ``` Grab the kubernetes-jsonnet module using and its dependencies, which include the kubernetes-mixin: ``` $ go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb $ jb init $ jb install github.com/kausalco/public/prometheus-ksonnet ``` Assuming you want to run in the default namespace ('environment' in ksonnet parlance), add the follow to the file `environments/default/main.jsonnet`: ``` local prometheus = import "prometheus-ksonnet/prometheus-ksonnet.libsonnet"; prometheus { _config+:: { namespace: "default", }, } ``` Apply your config: ``` $ ks apply default ``` ## Using kube-prometheus See the kube-prometheus docs for [instructions on how to use mixins with kube-prometheus](https://github.com/coreos/kube-prometheus#kube-prometheus). ## Customising the mixin Mixins typically allows you to override the selectors used for various jobs, to match those used in your Prometheus set. This example uses the [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin). In a new directory, add a file `mixin.libsonnet`: ``` local kubernetes = import "kubernetes-mixin/mixin.libsonnet"; kubernetes { _config+:: { kubeStateMetricsSelector: 'job="kube-state-metrics"', cadvisorSelector: 'job="kubernetes-cadvisor"', nodeExporterSelector: 'job="kubernetes-node-exporter"', kubeletSelector: 'job="kubernetes-kubelet"', }, } ``` Then, install the kubernetes-mixin: ``` $ jb init $ jb install github.com/kubernetes-monitoring/kubernetes-mixin ``` Generate the alerts, rules and dashboards: ``` $ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusAlerts)' > alerts.yml $ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusRules)' >files/rules.yml $ jsonnet -J vendor -m files/dashboards -e '(import "mixin.libsonnet").grafanaDashboards' ``` ## Guidelines for alert names, labels, and annotations Prometheus alerts deliberately allow users to define their own schema for names, labels, and annotations. The following is a style guide recommended for alerts in monitoring mixins. Following this guide helps creating useful notification templates for all mixins and customizing mixin alerts in a unified fashion. The alert **name** is a terse description of the alerting condition, using camel case, without whitespace, starting with a capital letter. The first component of the name should be shared between all alerts of a mixin (or between a group of related alerts within a larger mixin). Examples: `NodeFilesystemAlmostOutOfFiles` (from the [node-exporter mixin](https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin), `PrometheusNotificationQueueRunningFull` (from the [Prometheus mixin](https://github.com/prometheus/prometheus/blob/master/documentation/prometheus-mixin)). To mark the severity of an alert, use a **label** called `severity` with one of the following label values: - `critical` for alerts that require immediate action. For a production system, those alerts will usually hit a pager. - `warning` for alerts that require action eventually but not urgently enough to wake someone up or require them to immediately interrupt what they are working on. A typical routing target for those alerts is some kind of ticket queueing or bug tracking system. - `info` for alerts that do not require any action by itself but mark something as “out of the ordinary”. Those alerts aren't usually routed anywhere, but can be inspected during troubleshooting. An alert can have the following **annotations**: - `summary` (mandatory): Essentially a more comprehensive and readable version of the alert name. Use a human-readable sentence, starting with a capital letter and ending with a period. Use a static string or, if dynamic expansion is needed, aim for expanding into the same string for alerts that are typically grouped together into one notification. In that way, it can be used as a common “headline” for all alerts in the notification template. Examples: `Filesystem has less than 3% inodes left.` (for the `NodeFilesystemAlmostOutOfFiles` alert mentioned above), `Prometheus alert notification queue predicted to run full in less than 30m.` (for the `PrometheusNotificationQueueRunningFull` alert mentioned above). - `description` (mandatory): A detailed description of a single alert, with most of the important information templated in. The description usually expands into a different string for every individual alert within a notification. A notification template can iterate through all the descriptions and format them into a list. Examples (again corresponding to the examples above): `Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left.`, `Alert notification queue of Prometheus %(prometheusName)s is running full.`. Note that we plan to add recommended optional annotations for a runbook link (presumably called `runbook_url`) and a dashboard link (`dashboard_url`). However, we still need to work out how to configure patterns for those URLs across mixins in a useful way.