## 1. Project Overview & Quickstart (grafana/pyroscope) [](https://github.com/grafana/pyroscope/actions/workflows/ci.yml) [](https://goreportcard.com/report/github.com/grafana/pyroscope) [](LICENSE) [](https://app.fossa.com/projects/git%2Bgithub.com%2Fgrafana%2Fpyroscope?ref=badge_shield) [](https://github.com/grafana/pyroscope/releases) [](https://hub.docker.com/r/grafana/pyroscope) [](https://pkg.go.dev/github.com/grafana/pyroscope/v2) ## 🎉 **Announcement: Pyroscope 2.0 is here!** Pyroscope 2.0 makes the new **v2 architecture** the default. Profiles are written directly to object storage, removing the need for in-memory ingesters and local disks - simplifying operations and lowering resource usage at scale. Existing v1 deployments can opt in via a flag and migrate without data loss. Read the [2.0 release notes](https://grafana.com/docs/pyroscope/latest/release-notes/v2-0/) and the [v2 architecture overview](https://grafana.com/docs/pyroscope/latest/reference-pyroscope-v2-architecture/). Upgrading from v1? See the [migration guide](https://grafana.com/docs/pyroscope/latest/reference-pyroscope-v2-architecture/migrate-from-v1/). Want the full story? Watch the GrafanaCON 2026 talk [_Pyroscope 2.0: Continuous Profiling Architecture Deep Dive_](https://www.youtube.com/watch?v=vseSm-pzgmc). ## What is Grafana Pyroscope? Grafana Pyroscope is a continuous profiling platform designed to surface performance insights from your applications, helping you optimize resource usage such as CPU, memory, and I/O operations. With Pyroscope, you can both **proactively** and **reactively** address performance bottlenecks across your system. The typical use cases are: - **Proactive:** Reducing resource consumption, improving application performance, or preventing latency issues. - **Reactive:** Quickly resolving incidents with line-level detail and debugging active CPU, memory, or I/O bottlenecks. Pyroscope provides powerful tools to give you a comprehensive view of your application's behavior while allowing you to drill down into specific services for more targeted root cause analysis. ## How Does Pyroscope Work? Pyroscope consists of three main components: - **Pyroscope Server:** Stores and processes profiling data and serves queries. - **Clients and instrumentation:** Profiling data reaches the server in several ways: the [Pyroscope SDKs](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/) (push), [Grafana Alloy](https://grafana.com/docs/pyroscope/latest/configure-client/grafana-alloy/) (pull or push), or OTLP from OpenTelemetry-compatible sources such as the [OpenTelemetry eBPF profiler](https://grafana.com/docs/pyroscope/latest/configure-client/opentelemetry/ebpf-profiler/). - **Grafana Profiles Drilldown:** A queryless, intuitive UI for visualizing and analyzing profiling data (formerly Explore Profiles). Under the hood, Pyroscope v2 writes profiles straight to object storage—no ingesters, no local disk. The animations below trace the three parts of the architecture. For the details behind each component, see the [v2 architecture documentation](https://grafana.com/docs/pyroscope/latest/reference-pyroscope-v2-architecture/). **Write path** — profiles are routed by service and written straight to object storage: **Compaction** — compaction-workers merge small segments into larger blocks in the background: **Read path** — queries fan out across object storage to build flame graphs in Grafana Profiles Drilldown: --- ## [Pyroscope Live Demo](https://play.grafana.org/a/grafana-pyroscope-app/explore) [](https://play.grafana.org/a/grafana-pyroscope-app/explore) --- ## **Quick Start: Run the Pyroscope server locally** ### Docker ```sh docker run -it -p 4040:4040 grafana/pyroscope ``` ### Homebrew (macOS / Linux) ```sh brew install pyroscope-io/brew/pyroscope brew services start pyroscope ``` ### Binary Download the archive for your operating system and architecture from the [latest release](https://github.com/grafana/pyroscope/releases/latest), unpack it, and run the binary: ```sh tar xvf pyroscope_*.tar.gz ./pyroscope ``` Pyroscope listens on port `4040`. For Kubernetes/Helm, Linux packages, building from source, and full configuration options, see the [Get started guide](https://grafana.com/docs/pyroscope/latest/get-started/) and the [server documentation](https://grafana.com/docs/pyroscope/latest/configure-server/). ## **Quick Start: Visualize profiles with Grafana Profiles Drilldown** [Grafana Profiles Drilldown](https://grafana.com/docs/grafana/latest/visualizations/simplified-exploration/profiles/) (formerly Explore Profiles) is the primary, queryless way to visualize and analyze your profiling data. ### Grafana Cloud / OSS Profiles Drilldown is pre-installed and is the default way to explore your profiles – all you need to do is start sending data. ## Documentation For more information on how to use Pyroscope with other programming languages, install it on Linux, or use it in a production environment, check out our documentation: * [Getting Started](https://grafana.com/docs/pyroscope/latest/get-started/) * [Deployment Guide](https://grafana.com/docs/pyroscope/latest/deploy-kubernetes/) * [Pyroscope v2 Architecture](https://grafana.com/docs/pyroscope/latest/reference-pyroscope-v2-architecture/) ## Send data to the server You can send profiles to Pyroscope with the language SDKs, with [Grafana Alloy](https://grafana.com/docs/pyroscope/latest/configure-client/grafana-alloy/), or over OTLP from OpenTelemetry-compatible sources such as the [OpenTelemetry eBPF profiler](https://grafana.com/docs/pyroscope/latest/configure-client/opentelemetry/ebpf-profiler/). For more documentation on how to add the Pyroscope SDK to your code, see the [client documentation](https://grafana.com/docs/pyroscope/latest/configure-client/) on our website or find language-specific examples and documentation below: | **Golang** [Documentation](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/go_push/) [Examples](https://github.com/grafana/pyroscope/tree/main/examples/language-sdk-instrumentation/golang-push) | **Java** [Documentation](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/java/) [Examples](https://github.com/grafana/pyroscope/tree/main/examples/language-sdk-instrumentation/java/rideshare) | **Python** [Documentation](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/python/) [Examples](https://github.com/grafana/pyroscope/tree/main/examples/language-sdk-instrumentation/python) | **Ruby** [Documentation](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/ruby/) [Examples](https://github.com/grafana/pyroscope/tree/main/examples/language-sdk-instrumentation/ruby) | | --- | --- | --- | --- | | **Node.js** [Documentation](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/nodejs/) [Examples](https://github.com/grafana/pyroscope/tree/main/examples/language-sdk-instrumentation/nodejs/express) | **.NET** [Documentation](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/dotnet/) [Examples](https://github.com/grafana/pyroscope/tree/main/examples/language-sdk-instrumentation/dotnet) | **eBPF** [Documentation](https://grafana.com/docs/pyroscope/latest/configure-client/grafana-alloy/ebpf/) [Examples](https://github.com/grafana/pyroscope/tree/main/examples/grafana-alloy-auto-instrumentation/ebpf) | **Rust** [Documentation](https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/rust/) [Examples](https://github.com/grafana/pyroscope/tree/main/examples/language-sdk-instrumentation/rust/rideshare) | ## [Supported Languages][supported languages] Our documentation contains the most recent list of [supported languages] and also an overview over what [profiling types are supported per language][profile-types-languages]. Let us know what other integrations you want to see in [our issues](https://github.com/grafana/pyroscope/issues?q=is%3Aissue+is%3Aopen+label%3Anew-profilers) or in [our slack](https://slack.grafana.com). [supported languages]: https://grafana.com/docs/pyroscope/latest/configure-client/ [profile-types-languages]: https://grafana.com/docs/pyroscope/latest/configure-client/profile-types/ ## Credits Pyroscope is possible thanks to the excellent work of many people, including but not limited to: * Brendan Gregg — inventor of Flame Graphs * Julia Evans — creator of rbspy — sampling profiler for Ruby * Vladimir Agafonkin — creator of flamebearer — fast flame graph renderer * Ben Frederickson — creator of py-spy — sampling profiler for Python * Adam Saponara — creator of phpspy — sampling profiler for PHP * Alexei Starovoitov, Daniel Borkmann, and many others who made BPF based profiling in Linux kernel possible * Jamie Wong — creator of speedscope — interactive flame graph visualizer ## Contributing To start contributing, check out our [Contributing Guide](docs/internal/contributing/README.md) ### Thanks to the contributors of Pyroscope! [//]: contributor-faces [](https://github.com/simonswine) [](https://github.com/petethepig) [](https://github.com/cyriltovena) [](https://github.com/eh-am) [](https://github.com/kolesnikovae) [](https://github.com/korniltsev) [](https://github.com/Rperry2174) [](https://github.com/aleks-p) [](https://github.com/marcsanmi) [](https://github.com/aocenas) [](https://github.com/bryanhuhta) [](https://github.com/korniltsev-grafanista) [](https://github.com/jake-kramer) [](https://github.com/dogfrogfog) [](https://github.com/abeaumont) [](https://github.com/alsoba13) [](https://github.com/pavelpashkovsky) [](https://github.com/knylander-grafana) [](https://github.com/0xPoe) [](https://github.com/darrenjaneczek) [](https://github.com/LouisInFlow) [](https://github.com/oleg-kozlyuk-grafana) [](https://github.com/jdbaldry) [](https://github.com/shaleynikov) [](https://github.com/grafakus) [](https://github.com/09jvilla) [](https://github.com/joey-grafana) [](https://github.com/iOliverNguyen) [](https://github.com/Eve832) [](https://github.com/AdrK) [](https://github.com/alonlong) [](https://github.com/luweglarz) [](https://github.com/Loggy) [](https://github.com/robbymilo) [](https://github.com/RichiH) [](https://github.com/cjsampson) [](https://github.com/cristiangreco) [](https://github.com/callpraths) [](https://github.com/immanuwell) [](https://github.com/ekpatrice) [](https://github.com/ruslanpascoal2) [](https://github.com/StasDachinsky) [](https://github.com/gawicks) [](https://github.com/omarabid) [](https://github.com/scottzhlin) [](https://github.com/Skemba) [](https://github.com/cuishuang) [](https://github.com/wilfriedroset) [](https://github.com/korniltsev-grafanista-yolo-vibecoder239) [](https://github.com/rajat2004) [](https://github.com/nlamirault) [](https://github.com/alrs) [](https://github.com/fandreuz) [](https://github.com/cmonez) [](https://github.com/EgorMozheiko) [](https://github.com/cstyan) [](https://github.com/liaol) [](https://github.com/dancer1325) [](https://github.com/QuantumEnigmaa) [](https://github.com/Pranay0302) [](https://github.com/juliosaraiva) [](https://github.com/clovis1122) [](https://github.com/linthan) [//]: contributor-faces ## 2. Official Technical Reference & Guides (grafana/docs) # Prometheus Monitoring Mixins > NOTE: This project is *beta* stage. A mixin is a set of Grafana dashboards and Prometheus rules and alerts, packaged together in a reuseable and extensible bundle. Mixins are written in [jsonnet](https://jsonnet.org/), and are typically installed and updated with [jsonnet-bundler](https://github.com/jsonnet-bundler/jsonnet-bundler). For more information about mixins, see: * [Prometheus Monitoring Mixins Design Doc](https://docs.google.com/document/d/1A9xvzwqnFVSOZ5fD3blKODXfsat5fg6ZhnKu9LK3lB4/view). A [cached pdf](design.pdf) is included in this repo. * For more motivation, see "[The RED Method: How to instrument your services](https://kccncna17.sched.com/event/CU8K/the-red-method-how-to-instrument-your-services-b-tom-wilkie-kausal?iframe=no&w=100%&sidebar=yes&bg=no)" talk from CloudNativeCon Austin 2018. The KLUMPs system demo'd became the basis for the kubernetes-mixin. * "[Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts and Exporters](https://www.youtube.com/watch?v=b7-DtFfsL6E)" talk from CloudNativeCon Copenhagen 2018. * "[Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts and Exporters](https://promcon.io/2018-munich/talks/prometheus-monitoring-mixins/)" talk from PromCon 2018 (slightly updated). ## How to use mixins. Mixins are designed to be vendored into the repo with your infrastructure config. To do this, use [jsonnet-bundler](https://github.com/jsonnet-bundler/jsonnet-bundler): You then have three options for deploying your dashboards 1. Generate the config files and deploy them yourself. 1. Use ksonnet to deploy this mixin along with Prometheus and Grafana. 1. Use kube-prometheus to deploy this mixin. ## Generate config files You can manually generate the alerts, dashboards and rules files, but first you must install some tools: ``` $ go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest # macOS $ brew install jsonnet-bundler # Archlinux AUR $ yay -S jsonnet ``` Then, grab the mixin and its dependencies: ``` $ git clone https://github.com// $ cd $ jb install ``` Finally, build the mixin: ``` $ make prometheus_alerts.yaml $ make prometheus_rules.yaml $ make dashboards_out ``` The `prometheus_alerts.yaml` and `prometheus_rules.yaml` file then need to passed to your Prometheus server, and the files in `dashboards_out` need to be imported into you Grafana server. The exact details will depending on how you deploy your monitoring stack to Kubernetes. ## Using with prometheus-ksonnet Alternatively you can also use the mixin with [prometheus-ksonnet](https://github.com/grafana/jsonnet-libs/tree/master/prometheus-ksonnet), a [ksonnet](https://github.com/ksonnet/ksonnet) module to deploy a fully-fledged Prometheus-based monitoring system for Kubernetes: Make sure you have the ksonnet v0.8.0: ``` $ brew install https://raw.githubusercontent.com/ksonnet/homebrew-tap/82ef24cb7b454d1857db40e38671426c18cd8820/ks.rb $ brew pin ks $ ks version ksonnet version: v0.8.0 jsonnet version: v0.9.5 client-go version: v1.6.8-beta.0+$Format:%h$ ``` In your config repo, if you don't have a ksonnet application, make a new one (will copy credentials from current context): ``` $ ks init $ cd $ ks env add default ``` Grab the kubernetes-jsonnet module using and its dependencies, which include the kubernetes-mixin: ``` $ go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb $ jb init $ jb install github.com/kausalco/public/prometheus-ksonnet ``` Assuming you want to run in the default namespace ('environment' in ksonnet parlance), add the follow to the file `environments/default/main.jsonnet`: ``` local prometheus = import "prometheus-ksonnet/prometheus-ksonnet.libsonnet"; prometheus { _config+:: { namespace: "default", }, } ``` Apply your config: ``` $ ks apply default ``` ## Using kube-prometheus See the kube-prometheus docs for [instructions on how to use mixins with kube-prometheus](https://github.com/coreos/kube-prometheus#kube-prometheus). ## Customising the mixin Mixins typically allows you to override the selectors used for various jobs, to match those used in your Prometheus set. This example uses the [kubernetes-mixin](https://github.com/kubernetes-monitoring/kubernetes-mixin). In a new directory, add a file `mixin.libsonnet`: ``` local kubernetes = import "kubernetes-mixin/mixin.libsonnet"; kubernetes { _config+:: { kubeStateMetricsSelector: 'job="kube-state-metrics"', cadvisorSelector: 'job="kubernetes-cadvisor"', nodeExporterSelector: 'job="kubernetes-node-exporter"', kubeletSelector: 'job="kubernetes-kubelet"', }, } ``` Then, install the kubernetes-mixin: ``` $ jb init $ jb install github.com/kubernetes-monitoring/kubernetes-mixin ``` Generate the alerts, rules and dashboards: ``` $ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusAlerts)' > alerts.yml $ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusRules)' >files/rules.yml $ jsonnet -J vendor -m files/dashboards -e '(import "mixin.libsonnet").grafanaDashboards' ``` ## Guidelines for alert names, labels, and annotations Prometheus alerts deliberately allow users to define their own schema for names, labels, and annotations. The following is a style guide recommended for alerts in monitoring mixins. Following this guide helps creating useful notification templates for all mixins and customizing mixin alerts in a unified fashion. The alert **name** is a terse description of the alerting condition, using camel case, without whitespace, starting with a capital letter. The first component of the name should be shared between all alerts of a mixin (or between a group of related alerts within a larger mixin). Examples: `NodeFilesystemAlmostOutOfFiles` (from the [node-exporter mixin](https://github.com/prometheus/node_exporter/tree/master/docs/node-mixin), `PrometheusNotificationQueueRunningFull` (from the [Prometheus mixin](https://github.com/prometheus/prometheus/blob/master/documentation/prometheus-mixin)). To mark the severity of an alert, use a **label** called `severity` with one of the following label values: - `critical` for alerts that require immediate action. For a production system, those alerts will usually hit a pager. - `warning` for alerts that require action eventually but not urgently enough to wake someone up or require them to immediately interrupt what they are working on. A typical routing target for those alerts is some kind of ticket queueing or bug tracking system. - `info` for alerts that do not require any action by itself but mark something as “out of the ordinary”. Those alerts aren't usually routed anywhere, but can be inspected during troubleshooting. An alert can have the following **annotations**: - `summary` (mandatory): Essentially a more comprehensive and readable version of the alert name. Use a human-readable sentence, starting with a capital letter and ending with a period. Use a static string or, if dynamic expansion is needed, aim for expanding into the same string for alerts that are typically grouped together into one notification. In that way, it can be used as a common “headline” for all alerts in the notification template. Examples: `Filesystem has less than 3% inodes left.` (for the `NodeFilesystemAlmostOutOfFiles` alert mentioned above), `Prometheus alert notification queue predicted to run full in less than 30m.` (for the `PrometheusNotificationQueueRunningFull` alert mentioned above). - `description` (mandatory): A detailed description of a single alert, with most of the important information templated in. The description usually expands into a different string for every individual alert within a notification. A notification template can iterate through all the descriptions and format them into a list. Examples (again corresponding to the examples above): `Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left.`, `Alert notification queue of Prometheus %(prometheusName)s is running full.`. Note that we plan to add recommended optional annotations for a runbook link (presumably called `runbook_url`) and a dashboard link (`dashboard_url`). However, we still need to work out how to configure patterns for those URLs across mixins in a useful way.