node_exporter Documentation

1. Project Overview & Quickstart (prometheus/node_exporter) Copy Chapter

File: README.md Copy Chapter

Node exporter

Prometheus exporter for hardware and OS metrics exposed by *NIX kernels, written
in Go with pluggable metric collectors.

The Windows exporter is recommended for Windows users.
To expose NVIDIA GPU metrics, prometheus-dcgm
can be used.

Installation and Usage Copy Chapter

If you are new to Prometheus and node_exporter there is a simple step-by-step guide.

The node_exporter listens on HTTP port 9100 by default. See the --help output for more options.

Linux distribution packages Copy Section

On Enterprise Linux (RHEL, CentOS Stream, Rocky Linux, AlmaLinux) and Fedora, node_exporter is packaged in EPEL as golang-github-prometheus-node-exporter (available for EPEL 7, 8 and 9):

Ansible Copy Section

For automated installs with Ansible, there is the Prometheus Community role.

Docker Copy Section

The node_exporter is designed to monitor the host system. Deploying in containers requires
extra care in order to avoid monitoring the container itself.

For situations where containerized deployment is needed, some extra flags must be used to allow
the node_exporter access to the host namespaces.

Be aware that any non-root mount points you want to monitor will need to be bind-mounted
into the container.

If you start container for host monitoring, specify path.rootfs argument.
This argument must match path in bind-mount of host root. The node_exporter will use
path.rootfs as prefix to access host filesystem.

docker run -d \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  quay.io/prometheus/node-exporter:latest \
  --path.rootfs=/host

For Docker compose, similar flag changes are needed.

---
version: '3.8'

services:
  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    network_mode: host
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'

On some systems, the timex collector requires an additional Docker flag,
--cap-add=SYS_TIME, in order to access the required syscalls.

Collectors Copy Chapter

There is varying support for collectors on each operating system. The tables
below list all existing collectors and the supported systems.

Collectors are enabled by providing a --collector.<name> flag.
Collectors that are enabled by default can be disabled by providing a --no-collector.<name> flag.
To enable only some specific collector(s), use --collector.disable-defaults --collector.<name> ....

Include & Exclude flags Copy Section

A few collectors can be configured to include or exclude certain patterns using dedicated flags. The exclude flags are used to indicate "all except", while the include flags are used to say "none except". Note that these flags are mutually exclusive on collectors that support both.

Example:

--collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)

List:

Enabled by default Copy Section

Disabled by default Copy Section

node_exporter also implements a number of collectors that are disabled by default. Reasons for this vary by
collector, and may include:

• High cardinality
• Prolonged runtime that exceeds the Prometheus scrape_interval or scrape_timeout
• Significant resource demands on the host

You can enable additional collectors as desired by adding them to your
init system's or service supervisor's startup configuration for
node_exporter but caution is advised. Enable at most one at a time,
testing first on a non-production system, then by hand on a single
production node. When enabling additional collectors, you should
carefully monitor the change by observing the scrape_duration_seconds metric to ensure that collection completes
and does not time out. In addition, monitor the
scrape_samples_post_metric_relabeling metric to see the changes in
cardinality.

Deprecated Copy Section

These collectors are deprecated and will be removed in the next major release.

Perf Collector Copy Section

The perf collector may not work out of the box on some Linux systems due to kernel
configuration and security settings. To allow access, set the following sysctl
parameter:

sysctl -w kernel.perf_event_paranoid=X

• 2 allow only user-space measurements (default since Linux 4.6).
• 1 allow both kernel and user measurements (default before Linux 4.6).
• 0 allow access to CPU-specific data but not raw tracepoint samples.
• -1 no restrictions.

Depending on the configured value different metrics will be available, for most
cases 0 will provide the most complete set. For more information see man 2 perf_event_open.

By default, the perf collector will only collect metrics of the CPUs that
node_exporter is running on (ie
runtime.NumCPU. If this is
insufficient (e.g. if you run node_exporter with its CPU affinity set to
specific CPUs), you can specify a list of alternate CPUs by using the
--collector.perf.cpus flag. For example, to collect metrics on CPUs 2-6, you
would specify: --collector.perf --collector.perf.cpus=2-6. The CPU
configuration is zero indexed and can also take a stride value; e.g.
--collector.perf --collector.perf.cpus=1-10:5 would collect on CPUs
1, 5, and 10.

The perf collector is also able to collect
tracepoint
counts when using the --collector.perf.tracepoint flag. Tracepoints can be
found using perf list or
from debugfs. And example usage of this would be
--collector.perf.tracepoint="sched:sched_process_exec".

Sysctl Collector Copy Section

The sysctl collector can be enabled with --collector.sysctl. It supports exposing numeric sysctl values
as metrics using the --collector.sysctl.include flag and string values as info metrics by using the
--collector.sysctl.include-info flag. The flags can be repeated. For sysctl with multiple numeric values,
an optional mapping can be given to expose each value as its own metric. Otherwise an index label is used
to identify the different fields.

Examples

Numeric values

Single values

Using --collector.sysctl.include=vm.user_reserve_kbytes:
vm.user_reserve_kbytes = 131072 -> node_sysctl_vm_user_reserve_kbytes 131072

Multiple values

A sysctl can contain multiple values, for example:

net.ipv4.tcp_rmem = 4096	131072	6291456

Using --collector.sysctl.include=net.ipv4.tcp_rmem the collector will expose:

node_sysctl_net_ipv4_tcp_rmem{index="0"} 4096
node_sysctl_net_ipv4_tcp_rmem{index="1"} 131072
node_sysctl_net_ipv4_tcp_rmem{index="2"} 6291456

If the indexes have defined meaning like in this case, the values can be mapped to multiple metrics by appending the mapping to the --collector.sysctl.include flag:
Using --collector.sysctl.include=net.ipv4.tcp_rmem:min,default,max the collector will expose:

node_sysctl_net_ipv4_tcp_rmem_min 4096
node_sysctl_net_ipv4_tcp_rmem_default 131072
node_sysctl_net_ipv4_tcp_rmem_max 6291456

String values

String values need to be exposed as info metric. The user selects them by using the --collector.sysctl.include-info flag.

Single values

kernel.core_pattern = core -> node_sysctl_info{key="kernel.core_pattern_info", value="core"} 1

Multiple values

Given the following sysctl:

kernel.seccomp.actions_avail = kill_process kill_thread trap errno trace log allow

Setting --collector.sysctl.include-info=kernel.seccomp.actions_avail will yield:

node_sysctl_info{key="kernel.seccomp.actions_avail", index="0", value="kill_process"} 1
node_sysctl_info{key="kernel.seccomp.actions_avail", index="1", value="kill_thread"} 1
...

Textfile Collector Copy Section

The textfile collector is similar to the Pushgateway,
in that it allows exporting of statistics from batch jobs. It can also be used
to export static metrics, such as what role a machine has. The Pushgateway
should be used for service-level metrics. The textfile module is for metrics
that are tied to a machine.

To use it, set the --collector.textfile.directory flag on the node_exporter commandline. The
collector will parse all files in that directory matching the glob *.prom
using the text
format. Note: Timestamps are not supported.

To atomically push completion time for a cron job:

echo my_batch_job_completion_time $(date +%s) > /path/to/directory/my_batch_job.prom.$$
mv /path/to/directory/my_batch_job.prom.$$ /path/to/directory/my_batch_job.prom

To statically set roles for a machine using labels:

echo 'role{role="application_server"} 1' > /path/to/directory/role.prom.$$
mv /path/to/directory/role.prom.$$ /path/to/directory/role.prom

Filtering enabled collectors Copy Section

The node_exporter will expose all metrics from enabled collectors by default. This is the recommended way to collect metrics to avoid errors when comparing metrics of different families.

For advanced use the node_exporter can be passed an optional list of collectors to filter metrics. The parameters collect[] and exclude[] can be used multiple times (but cannot be combined). In Prometheus configuration you can use this syntax under the scrape config.

Collect only cpu and meminfo collector metrics:

params:
    collect[]:
      - cpu
      - meminfo

Collect all enabled collector metrics but exclude netdev:

params:
    exclude[]:
      - netdev

This can be useful for having different Prometheus servers collect specific metrics from nodes.

Development building and running Copy Chapter

Prerequisites:

• Go compiler
• RHEL/CentOS: glibc-static package.

Building:

git clone https://github.com/prometheus/node_exporter.git
cd node_exporter
make build
./node_exporter <flags>

To see all available configuration flags:

./node_exporter -h

Running tests Copy Chapter

make test

TLS endpoint Copy Chapter

EXPERIMENTAL

The exporter supports TLS via a new web configuration file.

./node_exporter --web.config.file=web-config.yml

See the exporter-toolkit web-configuration for more details.

File: docs/node-mixin/README.md Copy Chapter

Node Mixin

This is a work in progress. We aim for it to become a good role model for alerts
and dashboards eventually, but it is not quite there yet.

The Node Mixin is a set of configurable, reusable, and extensible alerts and
dashboards based on the metrics exported by the Node Exporter. The mixin creates
recording and alerting rules for Prometheus and suitable dashboard descriptions
for Grafana.

To use them, you need to have jsonnet (v0.16+) and jb installed. If you
have a working Go development environment, it's easiest to run the following:

go install github.com/google/go-jsonnet/cmd/jsonnet@latest
go install github.com/google/go-jsonnet/cmd/jsonnetfmt@latest
go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest

Next, install the dependencies by running the following command in this
directory:

jb install

You can then build the Prometheus rules files node_alerts.yaml and
node_rules.yaml:

make node_alerts.yaml node_rules.yaml

You can also build a directory dashboard_out with the JSON dashboard files
for Grafana:

make dashboards_out

Note that some of the generated dashboards require recording rules specified in
the previously generated node_rules.yaml.

For more advanced uses of mixins, see
https://github.com/monitoring-mixins/docs.

File: docs/TIME.md Copy Chapter

Monitoring time sync with node_exporter

`timex` collector Copy Chapter

This collector exports state of kernel time synchronization flag that should be
maintained by time-keeping daemon and is eventually raised by Linux kernel if
time-keeping daemon does not update it regularly.

Unfortunately some daemons do not handle this flag properly, e.g. chrony-1.30
from Debian/jessie clears STA_UNSYNC flag during daemon initialisation and
does not indicate clock synchronization status using this flag. Modern chrony
versions should work better. All chrony versions require rtcsync option to
maintain this flag. OpenNTPD does not touch this flag at all till
OpenNTPD-5.9p1.

On the other hand combination of sync_status and offset exported by timex
module is the way to monitor if systemd-timesyncd does its job.

`ntp` collector Copy Chapter

NOTE: This collector is deprecated and will be removed in the next major version release.

This collector is intended for usage with local NTP daemons including ntp.org, chrony, and OpenNTPD.

Note, some chrony packages have local stratum 10 configuration value making chrony a valid server when it is unsynchronised. This configuration makes one of the heuristics that derive node_ntp_sanity unreliable.

Note, OpenNTPD does not listen for SNTP queries by default. Add listen on 127.0.0.1 to the OpenNTPD configuration when using this collector with that package.

`node_ntp_stratum` Copy Section

This metric shows the stratum of the local NTP daemon.

Stratum 16 means that clock are unsynchronised. See also aforementioned note about default local stratum in chrony.

`node_ntp_leap` Copy Section

Raw leap flag value. 0 – OK, 1 – add leap second at UTC midnight, 2 – delete leap second at UTC midnight, 3 – unsynchronised.

OpenNTPD ignores leap seconds and never sets leap flag to 1 or 2.

`node_ntp_rtt` Copy Section

RTT (round-trip time) from node_exporter collector to local NTPD. This value is
used in sanity check as part of causality violation estimate.

`node_ntp_offset` Copy Section

Clock offset between local time and NTPD time.

ntp.org always sets NTPD time to local clock instead of relaying remote NTP
time, so this offset is irrelevant for this NTPD.

This value is used in sanity check as part of causality violation estimate.

`node_ntp_reference_timestamp_seconds` Copy Section

Reference Time. This field show time when the last adjustment was made, but
implementation details vary from "local wall-clock time" to "Reference Time
field in incoming SNTP packet".

time() - node_ntp_reference_timestamp_seconds and
node_time_seconds - node_ntp_reference_timestamp_seconds represent some estimate of
"freshness" of synchronization.

`node_ntp_root_delay` and `node_ntp_root_dispersion` Copy Section

These values are used to calculate synchronization distance that is limited by
collector.ntp.max-distance.

ntp.org adds known local offset to announced root dispersion and linearly
increases dispersion in case of NTP connectivity problems, OpenNTPD does not
account dispersion at all and always reports 0.

`node_ntp_sanity` Copy Section

Aggregate NTPD health including stratum, leap flag, sane freshness, root
distance being less than collector.ntp.max-distance and causality violation
being less than collector.ntp.local-offset-tolerance.

Causality violation is lower bound estimate of clock error done using SNTP,
it's calculated as positive portion of abs(node_ntp_offset) - node_ntp_rtt / 2.

File: docs/V0_16_UPGRADE_GUIDE.md Copy Chapter

Version 0.16.0 Upgrade Guide

The node_exporter 0.16.0 and newer renamed many metrics in order to conform with Prometheus naming best practices.

In order to allow easy upgrades, there are several options.

Update dashboards Copy Chapter

Grafana users can add multiple queries in order to display both the old and new data simultaneously.

Use recording rules Copy Chapter

We have provided a sample recording rule set that translates old metrics to new ones and the one that translates new metrics format to old one to create duplicate metrics (it translates "old" metrics format to new one). This has a minor disadvantage that it creates a lot of extra data, and re-aligns the timestamps of the data.

Run both old and new versions simultaneously. Copy Chapter

It's possible to run both the old and new exporter on different ports, and include an additional scrape job in Prometheus. It's recommended to enable only the collectors that have name changes that you care about.

File: examples/systemd/README.md Copy Chapter

Systemd Unit

If you are using distribution packages or the copr repository, you don't need to deal with these files!

The unit files (*.service and *.socket) in this directory are to be put into /etc/systemd/system.
It needs a user named node_exporter, whose shell should be /sbin/nologin and should not have any special privileges.
It needs a sysconfig file in /etc/sysconfig/node_exporter.
It needs a directory named /var/lib/node_exporter/textfile_collector, whose owner should be node_exporter:node_exporter.
A sample file can be found in sysconfig.node_exporter.

File: examples/launchctl/README.md Copy Chapter

MacOS LaunchDaemon

If you're installing through a package manager, you probably don't need to deal
with this file.

The plist file should be put in /Library/LaunchDaemons/ (user defined daemons), and the binary installed at
/usr/local/bin/node_exporter.

Ex. install globally by

sudo cp -n node_exporter /usr/local/bin/
sudo cp -n examples/launchctl/io.prometheus.node_exporter.plist /Library/LaunchDaemons/
sudo launchctl bootstrap system/ /Library/LaunchDaemons/io.prometheus.node_exporter.plist

# Optionally configure by dropping CLI arguments in a file
echo -- '--web.listen-address=:9101' | sudo tee /usr/local/etc/node_exporter.args

# Check it's running
sudo launchctl list | grep node_exporter

# See full process state
sudo launchctl print system/io.prometheus.node_exporter

# View logs
sudo tail /tmp/node_exporter.log

File: text_collector_examples/README.md Copy Chapter

Text collector example scripts

The scripts have been moved to
https://github.com/prometheus-community/node-exporter-textfile-collector-scripts

2. Official Technical Reference & Guides (prometheus/docs) Copy Chapter

File: README.md Copy Chapter

Prometheus landing page and documentation website

This repository contains both the content and the static-site generator code for the Prometheus documentation site and associated landing pages.

This is a Next.js-based website with some custom code to integrate documentation from other repositories and fetch information about available Prometheus component downloads from GitHub.

Contributing Changes Copy Chapter

See CONTRIBUTING.md for general instructions for new Prometheus contributors.

The documentation hosted on this website is pulled together from multiple repositories:

• The main documentation contents are located in the docs directory of this repository.
• Documentation concerning the Prometheus server is maintained in the Prometheus server repository and cloned into the website at build time.
• Documentation concerning the Alertmanager server is maintained in the Alertmanager server repository and cloned into the website at build time.

As a guideline, please keep the documentation generally applicable and avoid use-case-specific changes.

For guidelines around Markdown formatting and available frontmatter fields and features, please see the Markdown guide.

Building and running the website Copy Chapter

Prerequisites Copy Section

You'll need to have the following installed on your system:

• Node.js
• NPM

Environment variables Copy Section

To bypass anonymous user rate limits while downloading documentation from the other repositories, as well as fetching information about available downloads from the GitHub API, the pre-build scripts require a GitHub access token with read access to the repositories (no write access is required). You can create a personal access token in your GitHub account settings (https://github.com/settings/tokens/new).

Create a .env file in the root of the repository and add the following environment variable:

GITHUB_TOKEN=<your_github_token>

Installing npm dependencies Copy Section

To install all required npm package dependencies, run:

npm install

Building the website Copy Section

To build the website, run:

npm run build-all

This cleans any previous build artifacts, fetches the latest documentation from the Prometheus and Alertmanager repositories, fetches information about available downloads (for the Download page), builds the website, and then indexes it (for the built-in Pagefind-based search functionality).

The final output is a static website in the out directory.

You can also run each of these build steps separately:

• npm run clean - Cleans any build output and generated files from previous runs.
• npm run fetch-repo-docs - Fetches the latest documentation from the Prometheus and Alertmanager repositories.
• npm run fetch-downloads-info - Fetches information about available downloads (for the Download page).
• npm run build - Builds the website. When using npm, this automatically also runs the postbuild script, which generates Pagefind search indexes. If you are using pnpm, you will either need to run npm run postbuild manually, or set the enablePrePostScripts option in your pnpm-workspace.yaml file.

Serving the static build output Copy Section

To serve the static build output, run:

npx serve out

This will start a web server on port 3000. You can access the website at http://localhost:3000.

Running the website in development mode Copy Section

To run the website in development mode, run:

npm run dev

This will start a web server on port 3000. You can access the website at http://localhost:3000.

The website will automatically reload when you make changes to the source files.

NOTE: Site search is not available in development mode, as it requires building a Pagefind index on the static build output and then loading the generated /pagefind/pagefind.js file. This only happens when building the app for production via npm run build (part of npm run build-all).

Configuration Copy Chapter

You can configure some high-level settings for the documentation website in the docs-config.ts file in the root of the repository. This file configures:

• The base URL of the website.
• Which repositories to fetch documentation from.
• Which repositories to fetch download information from.
• Information about LTS (long-term-support) versions.

Automatic Deployment Copy Chapter

This site is automatically deployed using Netlify.

If you have the prerequisite access rights, you can view the Netlify settings here:

• GitHub webhook notifying Netlify of branch changes: https://github.com/prometheus/docs/settings/hooks
• Netlify project: https://app.netlify.com/sites/prometheus-docs

Changes to the main branch are deployed to the main site at https://prometheus.io.

Netlify also creates preview deploys for every pull request. To view these for a PR where all checks have passed:

1. In the CI section of the PR, click on "Show all checks".
2. On the "deploy/netlify" entry, click on "Details" to view the preview site for the PR.

You may have to wait a while for the "deploy/netlify" check to appear after creating or updating the PR, even if the other checks have already passed.

License Copy Chapter

Apache License 2.0, see LICENSE.

File: blog/README.md Copy Chapter

Prometheus Blog Content Guide

The Prometheus blog exists to share what’s happening in and around the Prometheus project.
It’s a space for updates, tutorials, deep dives, and stories from the community.
Our goal is to keep Prometheus users and contributors informed and engaged.

What belongs on the blog? Copy Chapter

We feature posts that help people learn, stay up to date, or feel more
connected to the project.

That could mean announcing new releases, explaining how a feature works,
or walking readers through a setup guide.

It might also mean sharing how your organization uses Prometheus,
talking about a recent event, or reflecting on your journey as a contributor.

What doesn’t belong are ads, marketing pitches, or reposts from company blogs.
We want posts that feel original, useful, and rooted in the spirit of open source collaboration.

How should posts be written? Copy Chapter

We understand that everyone has their own writing style, and we’d hate to cramp yours.
So, there are no rigid rules around style. But here a few formatting guidelines to keep the posts consistent:

• Headings: See this guide for headings.

• Links: Add them with descriptive text instead of just dropping the raw URL. Example:
  [Prometheus documentation](https://prometheus.io/docs/introduction/overview/).

• Images and diagrams: Always include alt text so the content is accessible to everyone.
  Store images in the <repo_root>/public/assets folder (instead of linking from external sites) and reference them in your post.

[!NOTE]
See the general Markdown Documentation Formatting Guide for more formatting rules.

The tone should be friendly and professional.
If you’re not sure what that looks like, take a look at some of the previously published posts
on the blog for inspiration.

Most posts fall somewhere between 800 words (like this short announcements)
and 1500 words (like this long deep dive).

If you include code snippets or configuration examples, please make sure they work as written.
Diagrams or screenshots are also great additions if they help readers grasp complex ideas more quickly.

When multiple people contribute to a post, we’ll make sure everyone is credited at the end.

[!IMPORTANT]
After your introductory section, add a `` marker on its own line.
This marker indicates where the preview cuts off on https://prometheus.io/blog/. See the example below.

Example structure:

---
title: "Getting Started with Prometheus"
author: Your Name (@yourgithubusername)
date: 2025-11-02
---

Prometheus makes it easy to collect and query metrics from your applications. In this post, we'll walk through
setting up your first metrics endpoint and understanding what Prometheus collects.



## Installing Prometheus

First, let's get Prometheus installed on your system...

How to contribute Copy Chapter

If you’d like to write for the blog, the process is simple:

0. Find a sponsor (a maintainer with expertise
   in your topic area), discuss your post idea with them, and see if they can sponsor your post.

[!IMPORTANT]
For Sponsors:
By agreeing to sponsor a post, you are committing to reviewing the post PR for technical accuracy and
merging it once you are satisfied it meets the required standard.

1. Create an issue in the prometheus/docs repo using the Blog Post Proposal template.
2. Wait for feedback from maintainers or editors to confirm it’s a good fit.
3. Draft your post in Markdown and submit it as a pull request.
4. Go through the review process, where we’ll help refine the draft together.
5. Once it’s approved, we’ll merge and publish it on the blog. 🎉
6. Share your post on your socials and feel free to tag us so we can boost it too.

Review process Copy Chapter

Every post should be reviewed before it goes live.
Whoever reviews the pull request will check for both technical accuracy and editorial quality.
This ensures that the content is both correct and accessible.
In some cases, these might be done by different people, but often one reviewer can cover both.

Why contribute? Copy Chapter

Contributing to the blog is a way to give back to the Prometheus community beyond code.
It helps others learn from your experience and brings visibility to your work.

Plus, non-code contributions like this count as real contributions. You’ll be joining
the long list of people who’ve shaped Prometheus in meaningful ways.

File: docs/visualization/browser.md Copy Chapter

title: Expression browser sort_rank: 1 Copy Chapter

The expression browser is available at /graph on the Prometheus server, allowing you
to enter any expression and see its result either in a table or graphed over time.

This is primarily useful for ad-hoc queries and debugging. For graphs, use
Grafana or Console templates.

File: docs/visualization/consoles.md Copy Chapter

title: Console templates sort_rank: 4 Copy Chapter

CAUTION: Starting with Prometheus 3.0, console templates and libraries are no longer bundled with Prometheus. If you wish to use console templates, you must provide your own templates and libraries by specifying the --web.console.templates and --web.console.libraries command-line flags. This documentation page is maintained for historical reference and to demonstrate the capabilities of console templates. Please be aware that any referenced console libraries from the Prometheus 2.x branch are no longer maintained and may contain known security vulnerabilities (CVEs).

Console templates allow for creation of arbitrary consoles using the Go
templating language. These are served
from the Prometheus server.

Console templates are the most powerful way to create templates that can be
easily managed in source control. There is a learning curve though, so users new
to this style of monitoring should try out
Grafana first.

Getting started Copy Chapter

Prometheus comes with an example set of consoles to get you going. These can be
found at /consoles/index.html.example on a running Prometheus and will
display Node Exporter consoles if Prometheus is scraping Node Exporters with a
job="node" label.

The example consoles have 5 parts:

1. A navigation bar on top
2. A menu on the left
3. Time controls on the bottom
4. The main content in the center, usually graphs
5. A table on the right

The navigation bar is for links to other systems, such as other Prometheis
¹,
documentation, and whatever else makes sense to you. The menu is for navigation
inside the same Prometheus server, which is very useful to be able to quickly
open a console in another tab to correlate information. Both are configured in
console_libraries/menu.lib.

The time controls allow changing of the duration and range of the graphs.
Console URLs can be shared and will show the same graphs for others.

The main content is usually graphs. There is a configurable JavaScript graphing
library provided that will handle requesting data from Prometheus, and rendering
it via Rickshaw.

Finally, the table on the right can be used to display statistics in a more
compact form than graphs.

Example Console Copy Chapter

This is a basic console. It shows the number of tasks, how many of them are up,
the average CPU usage, and the average memory usage in the right-hand-side
table. The main content has a queries-per-second graph.

{{template "head" .}}

{{template "prom_right_table_head"}}

  MyJob
  {{ template "prom_query_drilldown" (args "sum(up{job='myjob'})") }}
      / {{ template "prom_query_drilldown" (args "count(up{job='myjob'})") }}
  


  CPU
  {{ template "prom_query_drilldown" (args
      "avg by(job)(rate(process_cpu_seconds_total{job='myjob'}[5m]))"
      "s/s" "humanizeNoSmallPrefix") }}
  


  Memory
  {{ template "prom_query_drilldown" (args
       "avg by(job)(process_resident_memory_bytes{job='myjob'})"
       "B" "humanize1024") }}
  

{{template "prom_right_table_tail"}}


{{template "prom_content_head" .}}
<h1>MyJob</h1>

<h3>Queries</h3>

<script>
new PromConsole.Graph({
  node: document.querySelector("#queryGraph"),
  expr: "sum(rate(http_query_count{job='myjob'}[5m]))",
  name: "Queries",
  yAxisFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
  yHoverFormatter: PromConsole.NumberFormatter.humanizeNoSmallPrefix,
  yUnits: "/s",
  yTitle: "Queries"
})
</script>

{{template "prom_content_tail" .}}

{{template "tail"}}

The prom_right_table_head and prom_right_table_tail templates contain the
right-hand-side table. This is optional.

prom_query_drilldown is a template that will evaluate the expression passed to it, format it,
and link to the expression in the expression browser. The first
argument is the expression. The second argument is the unit to use. The third
argument is how to format the output. Only the first argument is required.

Valid output formats for the third argument to prom_query_drilldown:

• Not specified: Default Go display output.
• humanize: Display the result using metric prefixes.
• humanizeNoSmallPrefix: For absolute values greater than 1, display the
  result using metric prefixes. For
  absolute values less than 1, display 3 significant digits. This is useful
  to avoid units such as milliqueries per second that can be produced by
  humanize.
• humanize1024: Display the humanized result using a base of 1024 rather than 1000.
  This is usually used with B as the second argument to produce units such as KiB and MiB.
• printf.3g: Display 3 significant digits.

Custom formats can be defined. See
prom.lib for examples.

Graph Library Copy Chapter

The graph library is invoked as:

<script>
new PromConsole.Graph({
  node: document.querySelector("#queryGraph"),
  expr: "sum(rate(http_query_count{job='myjob'}[5m]))"
})
</script>

The head template loads the required Javascript and CSS.

Parameters to the graph library:

If both expr and name are lists, they must be of the same length. The name
will be applied to the plots for the corresponding expression.

Valid options for the yAxisFormatter and yHoverFormatter:

• PromConsole.NumberFormatter.humanize: Format using metric prefixes.
• PromConsole.NumberFormatter.humanizeNoSmallPrefix: For absolute values
  greater than 1, format using metric prefixes.
  For absolute values less than 1, format with 3 significant digits. This is
  useful to avoid units such as milliqueries per second that can be produced by
  PromConsole.NumberFormatter.humanize.
• PromConsole.NumberFormatter.humanize1024: Format the humanized result using a base of 1024 rather than 1000.

File: docs/visualization/grafana.md Copy Chapter

title: Grafana support for Prometheus nav_title: Grafana sort_rank: 2 Copy Chapter

Grafana is an open-source analytics and visualization platform used to monitor and analyze metrics from various data
sources. It allows users to create, explore, and share interactive dashboards, supporting integrations with databases
like Prometheus, InfluxDB, Elasticsearch, and more. Grafana is widely used for observability, providing alerting, plugin
extensibility, and a flexible query editor for real-time data visualization.

Note: The Grafana data source for Prometheus is included since Grafana 2.5.0 (2015-10-28).

The following shows an example Grafana dashboard which queries Prometheus for data:

Installing Copy Chapter

To install Grafana see the official Grafana
documentation.

Using Copy Chapter

By default, Grafana will be listening on
http://localhost:3000. The default login is "admin" /
"admin".

Creating a Prometheus data source Copy Section

To create a Prometheus data source in Grafana:

1. Click on the "cogwheel" in the sidebar to open the Configuration menu.
2. Click on "Data Sources".
3. Click on "Add data source".
4. Select "Prometheus" as the type.
5. Set the appropriate Prometheus server URL (for example, http://localhost:9090/)
6. Adjust other data source settings as desired (for example, choosing the right Access method).
7. Click "Save & Test" to save the new data source.

The following shows an example data source configuration:

Creating a Prometheus graph Copy Section

Follow the standard way of adding a new Grafana graph. Then:

1. Click the graph title, then click "Edit".
2. Under the "Metrics" tab, select your Prometheus data source (bottom right).
3. Enter any Prometheus expression into the "Query" field, while using the
   "Metric" field to lookup metrics via autocompletion.
4. To format the legend names of time series, use the "Legend format" input. For
   example, to show only the method and status labels of a returned query
   result, separated by a dash, you could use the legend format string
   {{method}} - {{status}}.
5. Tune other graph settings until you have a working graph.

The following shows an example Prometheus graph configuration:

In Grafana 7.2 and later, the $__rate_interval variable is
recommended
for use in the rateand increase functions.

Importing pre-built dashboards from Grafana.com Copy Section

Grafana.com maintains a collection of shared dashboards
which can be downloaded and used with standalone instances of Grafana. Use
the Grafana.com "Filter" option to browse dashboards for the "Prometheus"
data source only.

You must currently manually edit the downloaded JSON files and correct the
datasource: entries to reflect the Grafana data source name which you
chose for your Prometheus server. Use the "Dashboards" → "Home" → "Import"
option to import the edited dashboard file into your Grafana install.

File: docs/visualization/perses.md Copy Chapter

title: Perses support for Prometheus nav_title: Perses sort_rank: 3 Copy Chapter

Perses is an open-source dashboard and visualization platform designed for observability, with
native support for Prometheus as a data source.
It enables users to create, manage, and share dashboards for monitoring metrics and visualizing data.
Perses aims to provide a simple, flexible, and extensible alternative to other dashboarding tools, focusing on ease of
use, community-driven development, GitOps capabilities and dashboard as code approach.

Here is an example of a Perses dashboard querying Prometheus for data:

Installing Copy Chapter

To install Perses, see the official Perses documentation.

Using Copy Chapter

By default, Perses will be listening on port 8080. You can access the web UI at http://localhost:8080. There is no
login by default.

Creating a Prometheus data source Copy Section

To learn about how to set up a data source in Perses, please refer
to Perses documentation.
Once this connection to your Prometheus instance is configured, you are able to query it from the Dashboard and Explore
views.

Importing pre-built dashboards Copy Section

Perses is providing a set of pre-built dashboards that you can import into your instance. These dashboards are
maintained by the community and can be found in
the Perses dashboard repository

File: docs/tutorials/alerting_based_on_metrics.md Copy Chapter

title: Alerting based on metrics sort_rank: 5 Copy Chapter

In this tutorial we will create alerts on the ping_request_count metric that we instrumented earlier in the
Instrumenting HTTP server written in Go tutorial.

For the sake of this tutorial we will alert when the ping_request_count metric is greater than 5. Check out real world best practices to learn more about alerting principles.

Download the latest release of Alertmanager for your operating system from here.

Alertmanager supports various receivers like email, webhook, pagerduty, slack etc through which it can notify when an alert is firing. You can find the list of receivers and how to configure them here. We will use webhook as a receiver for this tutorial, head over to webhook.site and copy the webhook URL which we will use later to configure the Alertmanager.

First let's setup Alertmanager with the webhook receiver.

alertmanager.yml

global:
  resolve_timeout: 5m
route:
  receiver: webhook_receiver
receivers:
    - name: webhook_receiver
      webhook_configs:
        - url: ''
          send_resolved: false

Replace `` with the webhook that we copied earlier in the alertmanager.yml file and run the Alertmanager using the following command.

alertmanager --config.file=alertmanager.yml

Once the Alertmanager is up and running navigate to http://localhost:9093 and you should be able to access it.

Now that we have configured the Alertmanager with webhook receiver let's add the rules to the Prometheus config.

prometheus.yml

global:
 scrape_interval: 15s
 evaluation_interval: 10s
rule_files:
  - rules.yml
alerting:
  alertmanagers:
  - static_configs:
    - targets:
       - localhost:9093
scrape_configs:
 - job_name: prometheus
   static_configs:
       - targets: ["localhost:9090"]
 - job_name: simple_server
   static_configs:
       - targets: ["localhost:8090"]

Note that the evaluation_interval,rule_files and alerting sections were added to the Prometheus config. evaluation_interval defines the intervals at which the rules are evaluated, rule_files accepts an array of yaml files that defines the rules and the alerting section defines the Alertmanager configuration. As mentioned in the beginning of this tutorial we will create a basic rule where we want to
raise an alert when the ping_request_count value is greater than 5.

rules.yml

groups:
 - name: Count greater than 5
   rules:
   - alert: CountGreaterThan5
     expr: ping_request_count > 5
     for: 10s

Now let's run Prometheus using the following command.

prometheus --config.file=./prometheus.yml

Open http://localhost:9090/rules in your browser to see the rules. Next run the instrumented ping server and visit the http://localhost:8090/ping endpoint and refresh the page at least 6 times. You can check the ping count by navigating to the http://localhost:8090/metrics endpoint. To see the status of the alert visit http://localhost:9090/alerts. Once the condition ping_request_count > 5 is true for more than 10s the state will become FIRING. Now if you navigate back to your webhook.site URL you will see the alert message.

Similarly Alertmanager can be configured with other receivers to notify when an alert is firing.

File: docs/tutorials/getting_started.md Copy Chapter

title: Getting started with Prometheus sort_rank: 1 Copy Chapter

What is Prometheus ? Copy Chapter

Prometheus is a system monitoring and alerting system. It was opensourced by SoundCloud in 2012 and is the second project both to join and to graduate within Cloud Native Computing Foundation after Kubernetes. Prometheus stores all metrics data as time series, i.e metrics information is stored along with the timestamp at which it was recorded, optional key-value pairs called as labels can also be stored along with metrics.

What are metrics and why is it important? Copy Chapter

Metrics in layperson terms is a standard for measurement. What we want to measure depends from application to application. For a web server it can be request times, for a database it can be CPU usage or number of active connections etc.

Metrics play an important role in understanding why your application is working in a certain way. If you run a web application and someone comes up to you and says that the application is slow, you will need some information to find out what is happening with your application. For example the application can become slow when the number of requests are high. If you have the request count metric you can spot the reason and increase the number of servers to handle the heavy load. Whenever you are defining the metrics for your application you must put on your detective hat and ask this question what all information will be important for me to debug if any issue occurs in my application?

Basic Architecture of Prometheus Copy Chapter

The basic components of a Prometheus setup are:

• Prometheus Server (the server which scrapes and stores the metrics data).
• Targets to be scraped, for example an instrumented application that exposes its metrics, or an exporter that exposes metrics of another application.
• Alertmanager to raise alerts based on preset rules.

(Note: Apart from this Prometheus has push_gateway which is not covered here).

Let's consider a web server as an example application and we want to extract a certain metric like the number of API calls processed by the web server. So we add certain instrumentation code using the Prometheus client library and expose the metrics information. Now that our web server exposes its metrics we can configure Prometheus to scrape it. Now Prometheus is configured to fetch the metrics from the web server which is listening on xyz IP address port 7500 at a specific time interval, say, every minute.

At 11:00:00 when I make the server public for consumption, the application calculates the request count and exposes it, Prometheus simultaneously scrapes the count metric and stores the value as 0.

By 11:01:00 one request is processed. The instrumentation logic in the server increments the count to 1. When Prometheus scrapes the metric the value of count is 1 now.

By 11:02:00 two more requests are processed and the request count is 1+2 = 3 now. Similarly metrics are scraped and stored.

The user can control the frequency at which metrics are scraped by Prometheus.

(Note: This table is just a representation for understanding purposes. Prometheus doesn’t store the values in this exact format)

Prometheus also has an API which allows to query metrics which have been stored by scraping. This API is used to query the metrics, create dashboards/charts on it etc. PromQL is used to query these metrics.

A simple Line chart created on the Request Count metric will look like this

One can scrape multiple useful metrics to understand what is happening in the application and create multiple charts on them. Group the charts into a dashboard and use it to get an overview of the application.

Show me how it is done Copy Chapter

Let’s get our hands dirty and setup Prometheus. Prometheus is written using Go and all you need is the binary compiled for your operating system. Download the binary corresponding to your operating system from here and add the binary to your path.

Prometheus exposes its own metrics which can be consumed by itself or another Prometheus server.

Now that we have Prometheus installed, the next step is to run it. All that we need is just the binary and a configuration file. Prometheus uses yaml files for configuration.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]

In the above configuration file we have mentioned the scrape_interval, i.e how frequently we want Prometheus to scrape the metrics. We have added scrape_configs which has a name and target to scrape the metrics from. Prometheus by default listens on port 9090. So add it to targets.

prometheus --config.file=prometheus.yml

Now we have Prometheus up and running and scraping its own metrics every 15s. Prometheus has standard exporters available to export metrics. Next we will run a node exporter which is an exporter for machine metrics and scrape the same using Prometheus. (Download node metrics exporter.)

Run the node exporter in a terminal.

./node_exporter

Next, add node exporter to the list of scrape_configs:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ["localhost:9090"]
  - job_name: node_exporter
    static_configs:
      - targets: ["localhost:9100"]

In this tutorial we discussed what are metrics and why they are important, basic architecture of Prometheus and how to
run Prometheus.