# Technical Documentation: prometheus/client_golang > ℹ️ **Provenance:** Hybrid Fusion: `prometheus/client_golang` + `prometheus/docs` · [CodeWiki Reference](https://codewiki.google/github.com/prometheus/client_golang) · Recency: Active (< 180 days) ## 1. Project Overview & Quickstart (prometheus/client_golang) # Prometheus Go client library [](https://pkg.go.dev/github.com/prometheus/client_golang) [](https://securityscorecards.dev/viewer/?uri=github.com/prometheus/client_golang) [](https://slack.cncf.io/) This is the [Go](http://golang.org) client library for [Prometheus](http://prometheus.io). It has two separate parts, one for instrumenting application code, and one for creating clients that talk to the Prometheus HTTP API. ## Version Compatibility This library supports the two most recent major releases of Go. While it may function with older versions, we only provide fixes and support for the currently supported Go releases. > [!NOTE] > See our [Release Process](RELEASE.md#supported-go-versions) for details on compatibility and support policies. ## Important note about releases and stability This repository generally follows [Semantic Versioning](https://semver.org/). However, the API client in `prometheus/client_golang/api/…` is still considered experimental. Breaking changes of the API client will _not_ trigger a new major release. The same is true for selected other new features explicitly marked as **EXPERIMENTAL** in CHANGELOG.md. Features that require breaking changes in the stable parts of the repository are being batched up and tracked in the [v2 milestone](https://github.com/prometheus/client_golang/milestone/2), but plans for further development of v2 at the moment. > NOTE: The initial v2 attempt is in a [separate branch](https://github.com/prometheus/client_golang/tree/dev-v2). We also started experimenting on a new `prometheus.V2.*` APIs in [the 1.x's V2 struct](https://github.com/prometheus/client_golang/blob/main/prometheus/vnext.go#L23). Help wanted! ## Instrumenting applications [](https://pkg.go.dev/github.com/prometheus/client_golang/prometheus) The [`prometheus` directory](https://github.com/prometheus/client_golang/tree/main/prometheus) contains the instrumentation library. See the [guide](https://prometheus.io/docs/guides/go-application/) on the Prometheus website to learn more about instrumenting applications. The [`examples` directory](https://github.com/prometheus/client_golang/tree/main/examples) contains simple examples of instrumented code. ## Client for the Prometheus HTTP API [](https://pkg.go.dev/github.com/prometheus/client_golang/api) The [`api/prometheus` directory](https://github.com/prometheus/client_golang/tree/main/api/prometheus) contains the client for the [Prometheus HTTP API](http://prometheus.io/docs/querying/api/). It allows you to write Go applications that query time series data from a Prometheus server. It is still in alpha stage. ## Where is `model`, `extraction`, and `text`? The `model` packages has been moved to [`prometheus/common/model`](https://github.com/prometheus/common/tree/main/model). The `extraction` and `text` packages are now contained in [`prometheus/common/expfmt`](https://github.com/prometheus/common/tree/main/expfmt). ## Contributing and community See the [contributing guidelines](CONTRIBUTING.md) and the [Community section](http://prometheus.io/community/) of the homepage. `client_golang` community is also present on the CNCF Slack `#prometheus-client_golang`. ## 2. Official Technical Reference & Guides (prometheus/docs) ## File: README.md # Prometheus landing page and documentation website This repository contains both the content and the static-site generator code for the Prometheus documentation site and associated landing pages. This is a [Next.js](https://nextjs.org)-based website with some custom code to integrate documentation from other repositories and fetch information about available Prometheus component downloads from GitHub. ## Contributing Changes See [`CONTRIBUTING.md`](CONTRIBUTING.md) for general instructions for new Prometheus contributors. The documentation hosted on this website is pulled together from multiple repositories: * The main documentation contents are located in the [`docs`](docs) directory of this repository. * Documentation concerning the Prometheus server is [maintained in the Prometheus server repository](https://github.com/prometheus/prometheus/tree/main/docs) and cloned into the website at build time. * Documentation concerning the Alertmanager server is [maintained in the Alertmanager server repository](https://github.com/prometheus/alertmanager/tree/main/docs) and cloned into the website at build time. As a guideline, please keep the documentation generally applicable and avoid use-case-specific changes. For guidelines around Markdown formatting and available frontmatter fields and features, please see the [Markdown guide](markdown-guide.md). ## Building and running the website ### Prerequisites You'll need to have the following installed on your system: * [Node.js](https://nodejs.org/en/download/) * [NPM](https://www.npmjs.com/get-npm) ### Environment variables To bypass anonymous user rate limits while downloading documentation from the other repositories, as well as fetching information about available downloads from the GitHub API, the pre-build scripts require a GitHub access token with read access to the repositories (no write access is required). You can create a personal access token in your GitHub account settings (https://github.com/settings/tokens/new). Create a `.env` file in the root of the repository and add the following environment variable: ```env GITHUB_TOKEN= ``` ### Installing npm dependencies To install all required [npm](https://www.npmjs.com/) package dependencies, run: ```bash npm install ``` ### Building the website To build the website, run: ```bash npm run build-all ``` This cleans any previous build artifacts, fetches the latest documentation from the Prometheus and Alertmanager repositories, fetches information about available downloads (for the Download page), builds the website, and then indexes it (for the built-in [Pagefind](https://pagefind.app/)-based search functionality). The final output is a static website in the `out` directory. You can also run each of these build steps separately: * `npm run clean` - Cleans any build output and generated files from previous runs. * `npm run fetch-repo-docs` - Fetches the latest documentation from the Prometheus and Alertmanager repositories. * `npm run fetch-downloads-info` - Fetches information about available downloads (for the Download page). * `npm run build` - Builds the website. When using `npm`, this automatically also runs the `postbuild` script, which generates [Pagefind](https://pagefind.app/) search indexes. If you are using `pnpm`, you will either need to run `npm run postbuild` manually, or set the [`enablePrePostScripts` option](https://pnpm.io/cli/run#pnpm-workspaceyaml-settings) in your `pnpm-workspace.yaml` file. ### Serving the static build output To serve the static build output, run: ```bash npx serve out ``` This will start a web server on port 3000. You can access the website at [http://localhost:3000](http://localhost:3000). ### Running the website in development mode To run the website in development mode, run: ```bash npm run dev ``` This will start a web server on port 3000. You can access the website at [http://localhost:3000](http://localhost:3000). The website will automatically reload when you make changes to the source files. **NOTE:** Site search is not available in development mode, as it requires building a [Pagefind](https://pagefind.app/) index on the static build output and then loading the generated `/pagefind/pagefind.js` file. This only happens when building the app for production via `npm run build` (part of `npm run build-all`). ## Configuration You can configure some high-level settings for the documentation website in the [`docs-config.ts`](docs-config.ts) file in the root of the repository. This file configures: * The base URL of the website. * Which repositories to fetch documentation from. * Which repositories to fetch download information from. * Information about LTS (long-term-support) versions. ## Automatic Deployment This site is automatically deployed using [Netlify](https://www.netlify.com/). If you have the prerequisite access rights, you can view the Netlify settings here: * GitHub webhook notifying Netlify of branch changes: https://github.com/prometheus/docs/settings/hooks * Netlify project: https://app.netlify.com/sites/prometheus-docs Changes to the `main` branch are deployed to the main site at https://prometheus.io. Netlify also creates preview deploys for every pull request. To view these for a PR where all checks have passed: 1. In the CI section of the PR, click on "Show all checks". 2. On the "deploy/netlify" entry, click on "Details" to view the preview site for the PR. You may have to wait a while for the "deploy/netlify" check to appear after creating or updating the PR, even if the other checks have already passed. ## License Apache License 2.0, see [LICENSE](LICENSE). --- ## File: blog/README.md # Prometheus Blog Content Guide The Prometheus blog exists to share what’s happening in and around the Prometheus project. It’s a space for updates, tutorials, deep dives, and stories from the community. Our goal is to keep Prometheus users and contributors informed and engaged. ## What belongs on the blog? We feature posts that help people learn, stay up to date, or feel more connected to the project. That could mean announcing new releases, explaining how a feature works, or walking readers through a setup guide. It might also mean sharing how your organization uses Prometheus, talking about a recent event, or reflecting on your journey as a contributor. What doesn’t belong are ads, marketing pitches, or reposts from company blogs. We want posts that feel original, useful, and rooted in the spirit of open source collaboration. ## How should posts be written? We understand that everyone has their own writing style, and we’d hate to cramp yours. So, there are no rigid rules around style. But here a few formatting guidelines to keep the posts consistent: - **Headings:** See [this guide for headings](../markdown-guide.md#proper-usage-of-heading-levels). - **Links:** Add them with descriptive text instead of just dropping the raw URL. Example: `[Prometheus documentation](https://prometheus.io/docs/introduction/overview/)`. - **Images and diagrams:** Always include alt text so the content is accessible to everyone. Store images in the `/public/assets` folder (instead of linking from external sites) and reference them in your post. > [!NOTE] > See the general [Markdown Documentation Formatting Guide](../markdown-guide.md) for more formatting rules. The tone should be friendly and professional. If you’re not sure what that looks like, take a look at some of the [previously published posts](https://prometheus.io/blog/) on the blog for inspiration. Most posts fall somewhere between 800 words (like [this short announcements](https://prometheus.io/blog/2023/09/01/promcon2023-schedule/)) and 1500 words (like [this long deep dive](https://prometheus.io/blog/2021/11/16/agent/)). If you include code snippets or configuration examples, please make sure they work as written. Diagrams or screenshots are also great additions if they help readers grasp complex ideas more quickly. When multiple people contribute to a post, we’ll make sure everyone is credited at the end. > [!IMPORTANT] > After your introductory section, add a `` marker on its own line. This marker indicates where the preview cuts off on https://prometheus.io/blog/. See the example below. **Example structure:** ```markdown --- title: "Getting Started with Prometheus" author: Your Name (@yourgithubusername) date: 2025-11-02 --- Prometheus makes it easy to collect and query metrics from your applications. In this post, we'll walk through setting up your first metrics endpoint and understanding what Prometheus collects. ## Installing Prometheus First, let's get Prometheus installed on your system... ``` ## How to contribute If you’d like to write for the blog, the process is simple: 0. Find a sponsor (a [maintainer](https://github.com/prometheus/prometheus/blob/main/MAINTAINERS.md) with expertise in your topic area), discuss your post idea with them, and see if they can sponsor your post. > [!IMPORTANT] > **For Sponsors:** > By agreeing to sponsor a post, you are committing to reviewing the post PR for technical accuracy and > merging it once you are satisfied it meets the required standard. 1. Create an issue in the `prometheus/docs` repo using the [Blog Post Proposal template](https://github.com/prometheus/docs/issues/new?template=blog-post-proposal.yml). 2. Wait for feedback from maintainers or editors to confirm it’s a good fit. 3. Draft your post in Markdown and submit it as a pull request. 4. Go through the review process, where we’ll help refine the draft together. 5. Once it’s approved, we’ll merge and publish it on the blog. 🎉 6. Share your post on your socials and feel free to tag us so we can boost it too. ## Review process Every post should be reviewed before it goes live. Whoever reviews the pull request will check for both technical accuracy and editorial quality. This ensures that the content is both correct and accessible. In some cases, these might be done by different people, but often one reviewer can cover both. ## Why contribute? Contributing to the blog is a way to give back to the Prometheus community beyond code. It helps others learn from your experience and brings visibility to your work. Plus, non-code contributions like this count as real contributions. You’ll be joining the long list of people who’ve shaped Prometheus in meaningful ways. --- ## File: docs/visualization/browser.md --- title: Expression browser sort_rank: 1 --- The expression browser is available at `/graph` on the Prometheus server, allowing you to enter any expression and see its result either in a table or graphed over time. This is primarily useful for ad-hoc queries and debugging. For graphs, use [Grafana](/docs/visualization/grafana/) or [Console templates](/docs/visualization/consoles/). --- ## File: docs/visualization/consoles.md --- title: Console templates sort_rank: 4 --- CAUTION: Starting with Prometheus 3.0, console templates and libraries are no longer bundled with Prometheus. If you wish to use console templates, you must provide your own templates and libraries by specifying the `--web.console.templates` and `--web.console.libraries` command-line flags. This documentation page is maintained for historical reference and to demonstrate the capabilities of console templates. Please be aware that any referenced console libraries from the Prometheus 2.x branch are no longer maintained and may contain known security vulnerabilities (CVEs). Console templates allow for creation of arbitrary consoles using the [Go templating language](http://golang.org/pkg/text/template/). These are served from the Prometheus server. Console templates are the most powerful way to create templates that can be easily managed in source control. There is a learning curve though, so users new to this style of monitoring should try out [Grafana](/docs/visualization/grafana/) first. ## Getting started Prometheus comes with an example set of consoles to get you going. These can be found at `/consoles/index.html.example` on a running Prometheus and will display Node Exporter consoles if Prometheus is scraping Node Exporters with a `job="node"` label. The example consoles have 5 parts: 1. A navigation bar on top 1. A menu on the left 1. Time controls on the bottom 1. The main content in the center, usually graphs 1. A table on the right The navigation bar is for links to other systems, such as other Prometheis [1](/docs/introduction/faq/#what-is-the-plural-of-prometheus), documentation, and whatever else makes sense to you. The menu is for navigation inside the same Prometheus server, which is very useful to be able to quickly open a console in another tab to correlate information. Both are configured in `console_libraries/menu.lib`. The time controls allow changing of the duration and range of the graphs. Console URLs can be shared and will show the same graphs for others. The main content is usually graphs. There is a configurable JavaScript graphing library provided that will handle requesting data from Prometheus, and rendering it via [Rickshaw](https://shutterstock.github.io/rickshaw/). Finally, the table on the right can be used to display statistics in a more compact form than graphs. ## Example Console This is a basic console. It shows the number of tasks, how many of them are up, the average CPU usage, and the average memory usage in the right-hand-side table. The main content has a queries-per-second graph. ``` {{template "head" .}} {{template "prom_right_table_head"}} MyJob {{ template "prom_query_drilldown" (args "sum(up{job='myjob'})") }} / {{ template "prom_query_drilldown" (args "count(up{job='myjob'})") }} CPU {{ template "prom_query_drilldown" (args "avg by(job)(rate(process_cpu_seconds_total{job='myjob'}[5m]))" "s/s" "humanizeNoSmallPrefix") }} Memory {{ template "prom_query_drilldown" (args "avg by(job)(process_resident_memory_bytes{job='myjob'})" "B" "humanize1024") }} {{template "prom_right_table_tail"}} {{template "prom_content_head" .}}

MyJob

Queries

{{template "prom_content_tail" .}} {{template "tail"}} ``` The `prom_right_table_head` and `prom_right_table_tail` templates contain the right-hand-side table. This is optional. `prom_query_drilldown` is a template that will evaluate the expression passed to it, format it, and link to the expression in the [expression browser](/docs/visualization/browser/). The first argument is the expression. The second argument is the unit to use. The third argument is how to format the output. Only the first argument is required. Valid output formats for the third argument to `prom_query_drilldown`: * Not specified: Default Go display output. * `humanize`: Display the result using [metric prefixes](http://en.wikipedia.org/wiki/Metric_prefix). * `humanizeNoSmallPrefix`: For absolute values greater than 1, display the result using [metric prefixes](http://en.wikipedia.org/wiki/Metric_prefix). For absolute values less than 1, display 3 significant digits. This is useful to avoid units such as milliqueries per second that can be produced by `humanize`. * `humanize1024`: Display the humanized result using a base of 1024 rather than 1000. This is usually used with `B` as the second argument to produce units such as `KiB` and `MiB`. * `printf.3g`: Display 3 significant digits. Custom formats can be defined. See [prom.lib](https://github.com/prometheus/prometheus/blob/release-2.55/console_libraries/prom.lib) for examples. ## Graph Library The graph library is invoked as: ``` ``` The `head` template loads the required Javascript and CSS. Parameters to the graph library: | Name | Description | ------------- | ------------- | expr | Required. Expression to graph. Can be a list. | node | Required. DOM node to render into. | duration | Optional. Duration of the graph. Defaults to 1 hour. | endTime | Optional. Unixtime the graph ends at. Defaults to now. | width | Optional. Width of the graph, excluding titles. Defaults to auto-detection. | height | Optional. Height of the graph, excluding titles and legends. Defaults to 200 pixels. | min | Optional. Minimum x-axis value. Defaults to lowest data value. | max | Optional. Maximum y-axis value. Defaults to highest data value. | renderer | Optional. Type of graph. Options are `line` and `area` (stacked graph). Defaults to `line`. | name | Optional. Title of plots in legend and hover detail. If passed a string, `[[ label ]]` will be substituted with the label value. If passed a function, it will be passed a map of labels and should return the name as a string. Can be a list. | xTitle | Optional. Title of the x-axis. Defaults to `Time`. | yUnits | Optional. Units of the y-axis. Defaults to empty. | yTitle | Optional. Title of the y-axis. Defaults to empty. | yAxisFormatter | Optional. Number formatter for the y-axis. Defaults to `PromConsole.NumberFormatter.humanize`. | yHoverFormatter | Optional. Number formatter for the hover detail. Defaults to `PromConsole.NumberFormatter.humanizeExact`. | colorScheme | Optional. Color scheme to be used by the plots. Can be either a list of hex color codes or one of the [color scheme names](https://github.com/shutterstock/rickshaw/blob/master/src/js/Rickshaw.Fixtures.Color.js) supported by Rickshaw. Defaults to `'colorwheel'`. If both `expr` and `name` are lists, they must be of the same length. The name will be applied to the plots for the corresponding expression. Valid options for the `yAxisFormatter` and `yHoverFormatter`: * `PromConsole.NumberFormatter.humanize`: Format using [metric prefixes](http://en.wikipedia.org/wiki/Metric_prefix). * `PromConsole.NumberFormatter.humanizeNoSmallPrefix`: For absolute values greater than 1, format using [metric prefixes](http://en.wikipedia.org/wiki/Metric_prefix). For absolute values less than 1, format with 3 significant digits. This is useful to avoid units such as milliqueries per second that can be produced by `PromConsole.NumberFormatter.humanize`. * `PromConsole.NumberFormatter.humanize1024`: Format the humanized result using a base of 1024 rather than 1000. --- ## File: docs/visualization/grafana.md --- title: Grafana support for Prometheus nav_title: Grafana sort_rank: 2 --- [Grafana](http://grafana.com/) is an open-source analytics and visualization platform used to monitor and analyze metrics from various data sources. It allows users to create, explore, and share interactive dashboards, supporting integrations with databases like Prometheus, InfluxDB, Elasticsearch, and more. Grafana is widely used for observability, providing alerting, plugin extensibility, and a flexible query editor for real-time data visualization. Note: The Grafana data source for Prometheus is included since Grafana 2.5.0 (2015-10-28). The following shows an example Grafana dashboard which queries Prometheus for data: [](/assets/docs/grafana_prometheus.png) ## Installing To install Grafana see the [official Grafana documentation](https://grafana.com/grafana/download/). ## Using By default, Grafana will be listening on [http://localhost:3000](http://localhost:3000). The default login is "admin" / "admin". ### Creating a Prometheus data source To create a Prometheus data source in Grafana: 1. Click on the "cogwheel" in the sidebar to open the Configuration menu. 2. Click on "Data Sources". 3. Click on "Add data source". 4. Select "Prometheus" as the type. 5. Set the appropriate Prometheus server URL (for example, `http://localhost:9090/`) 6. Adjust other data source settings as desired (for example, choosing the right Access method). 7. Click "Save & Test" to save the new data source. The following shows an example data source configuration: [](/assets/docs/grafana_configuring_datasource.png) ### Creating a Prometheus graph Follow the standard way of adding a new Grafana graph. Then: 1. Click the graph title, then click "Edit". 2. Under the "Metrics" tab, select your Prometheus data source (bottom right). 3. Enter any Prometheus expression into the "Query" field, while using the "Metric" field to lookup metrics via autocompletion. 4. To format the legend names of time series, use the "Legend format" input. For example, to show only the `method` and `status` labels of a returned query result, separated by a dash, you could use the legend format string `{{method}} - {{status}}`. 5. Tune other graph settings until you have a working graph. The following shows an example Prometheus graph configuration: [](/assets/docs/grafana_qps_graph.png) In Grafana 7.2 and later, the `$__rate_interval` variable is [recommended](https://grafana.com/docs/grafana/latest/datasources/prometheus/#using-__rate_interval) for use in the `rate`and `increase` functions. ### Importing pre-built dashboards from Grafana.com Grafana.com maintains [a collection of shared dashboards](https://grafana.com/dashboards) which can be downloaded and used with standalone instances of Grafana. Use the Grafana.com "Filter" option to browse dashboards for the "Prometheus" data source only. You must currently manually edit the downloaded JSON files and correct the `datasource:` entries to reflect the Grafana data source name which you chose for your Prometheus server. Use the "Dashboards" → "Home" → "Import" option to import the edited dashboard file into your Grafana install. --- ## File: docs/visualization/perses.md --- title: Perses support for Prometheus nav_title: Perses sort_rank: 3 --- [Perses](https://perses.dev) is an open-source dashboard and visualization platform designed for observability, with native support for Prometheus as a data source. It enables users to create, manage, and share dashboards for monitoring metrics and visualizing data. Perses aims to provide a simple, flexible, and extensible alternative to other dashboarding tools, focusing on ease of use, community-driven development, GitOps capabilities and dashboard as code approach. Here is an example of a Perses dashboard querying Prometheus for data: [](/assets/docs/perses_prometheus.png) ## Installing To install Perses, see the official [Perses documentation](https://perses.dev/perses/docs/installation/in-a-container/). ## Using By default, Perses will be listening on port `8080`. You can access the web UI at `http://localhost:8080`. There is no login by default. ### Creating a Prometheus data source To learn about how to set up a data source in Perses, please refer to [Perses documentation](https://perses.dev/perses/docs/concepts/datasources). Once this connection to your Prometheus instance is configured, you are able to query it from the Dashboard and Explore views. ### Importing pre-built dashboards Perses is providing a set of pre-built dashboards that you can import into your instance. These dashboards are maintained by the community and can be found in the [Perses dashboard repository](https://github.com/perses/community-dashboards) --- ## File: docs/tutorials/alerting_based_on_metrics.md --- title: Alerting based on metrics sort_rank: 5 --- In this tutorial we will create alerts on the `ping_request_count` metric that we instrumented earlier in the [Instrumenting HTTP server written in Go](./instrumenting_http_server_in_go.md) tutorial. For the sake of this tutorial we will alert when the `ping_request_count` metric is greater than 5. Check out real world [best practices](../practices/alerting.md) to learn more about alerting principles. Download the latest release of Alertmanager for your operating system from [here](https://github.com/prometheus/alertmanager/releases). Alertmanager supports various receivers like `email`, `webhook`, `pagerduty`, `slack` etc through which it can notify when an alert is firing. You can find the list of receivers and how to configure them [here](/docs/alerting/latest/configuration/). We will use `webhook` as a receiver for this tutorial, head over to [webhook.site](https://webhook.site) and copy the webhook URL which we will use later to configure the Alertmanager. First let's setup Alertmanager with the webhook receiver. > alertmanager.yml ```yaml global: resolve_timeout: 5m route: receiver: webhook_receiver receivers: - name: webhook_receiver webhook_configs: - url: '' send_resolved: false ``` Replace `` with the webhook that we copied earlier in the alertmanager.yml file and run the Alertmanager using the following command. `alertmanager --config.file=alertmanager.yml` Once the Alertmanager is up and running navigate to [http://localhost:9093](http://localhost:9093) and you should be able to access it. Now that we have configured the Alertmanager with webhook receiver let's add the rules to the Prometheus config. > prometheus.yml ```yaml global: scrape_interval: 15s evaluation_interval: 10s rule_files: - rules.yml alerting: alertmanagers: - static_configs: - targets: - localhost:9093 scrape_configs: - job_name: prometheus static_configs: - targets: ["localhost:9090"] - job_name: simple_server static_configs: - targets: ["localhost:8090"] ``` Note that the `evaluation_interval`,`rule_files` and `alerting` sections were added to the Prometheus config. `evaluation_interval` defines the intervals at which the rules are evaluated, `rule_files` accepts an array of yaml files that defines the rules and the `alerting` section defines the Alertmanager configuration. As mentioned in the beginning of this tutorial we will create a basic rule where we want to raise an alert when the `ping_request_count` value is greater than 5. > rules.yml ```yaml groups: - name: Count greater than 5 rules: - alert: CountGreaterThan5 expr: ping_request_count > 5 for: 10s ``` Now let's run Prometheus using the following command. `prometheus --config.file=./prometheus.yml` Open [http://localhost:9090/rules](http://localhost:9090/rules) in your browser to see the rules. Next run the instrumented ping server and visit the [http://localhost:8090/ping](http://localhost:8090/ping) endpoint and refresh the page at least 6 times. You can check the ping count by navigating to the [http://localhost:8090/metrics](http://localhost:8090/metrics) endpoint. To see the status of the alert visit [http://localhost:9090/alerts](http://localhost:9090/alerts). Once the condition `ping_request_count > 5` is true for more than 10s the `state` will become `FIRING`. Now if you navigate back to your `webhook.site` URL you will see the alert message. Similarly Alertmanager can be configured with other receivers to notify when an alert is firing. --- ## File: docs/tutorials/getting_started.md --- title: Getting started with Prometheus sort_rank: 1 --- ## What is Prometheus ? Prometheus is a system monitoring and alerting system. It was opensourced by SoundCloud in 2012 and is the second project both to join and to graduate within Cloud Native Computing Foundation after Kubernetes. Prometheus stores all metrics data as time series, i.e metrics information is stored along with the timestamp at which it was recorded, optional key-value pairs called as labels can also be stored along with metrics. ## What are metrics and why is it important? Metrics in layperson terms is a standard for measurement. What we want to measure depends from application to application. For a web server it can be request times, for a database it can be CPU usage or number of active connections etc. Metrics play an important role in understanding why your application is working in a certain way. If you run a web application and someone comes up to you and says that the application is slow, you will need some information to find out what is happening with your application. For example the application can become slow when the number of requests are high. If you have the request count metric you can spot the reason and increase the number of servers to handle the heavy load. Whenever you are defining the metrics for your application you must put on your detective hat and ask this question **what all information will be important for me to debug if any issue occurs in my application?** ## Basic Architecture of Prometheus The basic components of a Prometheus setup are: - Prometheus Server (the server which scrapes and stores the metrics data). - Targets to be scraped, for example an instrumented application that exposes its metrics, or an exporter that exposes metrics of another application. - Alertmanager to raise alerts based on preset rules. (Note: Apart from this Prometheus has push_gateway which is not covered here). [](/assets/docs/tutorial/architecture.png) Let's consider a web server as an example application and we want to extract a certain metric like the number of API calls processed by the web server. So we add certain instrumentation code using the Prometheus client library and expose the metrics information. Now that our web server exposes its metrics we can configure Prometheus to scrape it. Now Prometheus is configured to fetch the metrics from the web server which is listening on xyz IP address port 7500 at a specific time interval, say, every minute. At 11:00:00 when I make the server public for consumption, the application calculates the request count and exposes it, Prometheus simultaneously scrapes the count metric and stores the value as 0. By 11:01:00 one request is processed. The instrumentation logic in the server increments the count to 1. When Prometheus scrapes the metric the value of count is 1 now. By 11:02:00 two more requests are processed and the request count is 1+2 = 3 now. Similarly metrics are scraped and stored. The user can control the frequency at which metrics are scraped by Prometheus. | Time Stamp | Request Count (metric) | | ---------- | ---------------------- | | 11:00:00 | 0 | | 11:01:00 | 1 | | 11:02:00 | 3 | (Note: This table is just a representation for understanding purposes. Prometheus doesn’t store the values in this exact format) Prometheus also has an API which allows to query metrics which have been stored by scraping. This API is used to query the metrics, create dashboards/charts on it etc. PromQL is used to query these metrics. A simple Line chart created on the Request Count metric will look like this [](/assets/docs/tutorial/sample_graph.png) One can scrape multiple useful metrics to understand what is happening in the application and create multiple charts on them. Group the charts into a dashboard and use it to get an overview of the application. ## Show me how it is done Let’s get our hands dirty and setup Prometheus. Prometheus is written using [Go](https://golang.org/) and all you need is the binary compiled for your operating system. Download the binary corresponding to your operating system from [here](https://prometheus.io/download/) and add the binary to your path. Prometheus exposes its own metrics which can be consumed by itself or another Prometheus server. Now that we have Prometheus installed, the next step is to run it. All that we need is just the binary and a configuration file. Prometheus uses yaml files for configuration. ```yaml global: scrape_interval: 15s scrape_configs: - job_name: prometheus static_configs: - targets: ["localhost:9090"] ``` In the above configuration file we have mentioned the `scrape_interval`, i.e how frequently we want Prometheus to scrape the metrics. We have added `scrape_configs` which has a name and target to scrape the metrics from. Prometheus by default listens on port 9090. So add it to targets. > prometheus --config.file=prometheus.yml Now we have Prometheus up and running and scraping its own metrics every 15s. Prometheus has standard exporters available to export metrics. Next we will run a node exporter which is an exporter for machine metrics and scrape the same using Prometheus. ([Download node metrics exporter.](https://prometheus.io/download/#node_exporter)) Run the node exporter in a terminal. `./node_exporter` [](/assets/docs/tutorial/node_exporter.png) Next, add node exporter to the list of scrape_configs: ```yaml global: scrape_interval: 15s scrape_configs: - job_name: prometheus static_configs: - targets: ["localhost:9090"] - job_name: node_exporter static_configs: - targets: ["localhost:9100"] ``` In this tutorial we discussed what are metrics and why they are important, basic architecture of Prometheus and how to run Prometheus. --- METRICS --- - Files Extracted: 9 - Estimated Token Budget: ~9186 tokens - Recency Window: Active (< 180 days) - Canonical Reference: https://codewiki.google/github.com/prometheus/client_golang