parca Documentation

1. Project Overview & Quickstart (parca-dev/parca) Copy Chapter

Features Copy Chapter

• eBPF Profiler: A single profiler, using eBPF, automatically discovering targets from Kubernetes or systemd across the entire infrastructure with very low overhead. Supports C, C++, Rust, Go, and more!

• Open Standards: Both producing pprof formatted profiles with the eBPF based profiler, and ingesting any pprof formatted profiles allowing for wide language adoption and interoperability with existing tooling.

• Optimized Storage & Querying: Efficiently storing profiling data while retaining raw data and allowing slicing and dicing of data through a label-based search. Aggregate profiling data infrastructure wide, view single profiles in time or compare on any dimension.

Why? Copy Chapter

• Save Money: Many organizations have 20-30% of resources wasted with easily optimized code paths. The Parca Agent aims to lower the entry bar by requiring 0 instrumentation for the whole infrastructure. Deploy in your infrastructure and get started!
• Improve Performance: Using profiling data collected over time, Parca can with confidence and statistical significance determine hot paths to optimize. Additionally, it can show differences between any label dimension, such as deploys, versions, and regions.
• Understand Incidents: Profiling data provides unique insight and depth into what a process executed over time. Memory leaks, but also momentary spikes in CPU or I/O causing unexpected behavior, is traditionally difficult to troubleshoot are a breeze with continuous profiling.

Feedback & Support Copy Chapter

If you have any feedback, please open a discussion in the GitHub Discussions of this project.
We would love to learn what you think!

Installation & Documentation Copy Chapter

Check Parca's website for updated and in-depth installation guides and documentation!

parca.dev

Development Copy Chapter

You need to have Go, Node and Pnpm installed.

Clone the project

git clone https://github.com/parca-dev/parca.git

Go to the project directory

cd parca

Build the UI and compile the Go binaries

make build

Running the compiled Parca binary Copy Section

The binary was compiled to bin/parca .

./bin/parca

Now Parca is running locally and its web UI is available on http://localhost:7070/.

By default, Parca is scraping it's own pprof endpoints and you should see profiles show up over time.
The scrape configuration can be changed in the parca.yaml in the root of the repository.

Configuration Copy Section

Flags:

/* Detailed source-code truncated for AI context efficiency. */

Credits Copy Chapter

Parca was originally developed by Polar Signals. Read the announcement blog post: https://www.polarsignals.com/blog/posts/2021/10/08/introducing-parca-we-got-funded/

Contributing Copy Chapter

Check out our Contributing Guide to get started!
It explains how compile Parca, run it with Tilt as container in Kubernetes and send a Pull Request.

Contributors ✨ Copy Chapter

Thanks goes to these wonderful people (emoji key):

This project follows the all-contributors specification. Contributions of any kind welcome!

Tested by Meticulous Copy Chapter

We use Meticulous to automatically test our UI for any unwanted changes.

2. Official Technical Reference & Guides (parca-dev/docs) Copy Chapter

File: README.md Copy Chapter

Website

This website is built using Docusaurus 2, a modern static website generator.

Installation Copy Chapter

yarn install

Local Development Copy Chapter

yarn start

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

Build Copy Chapter

yarn build

This command generates static content into the build directory and can be served using any static contents hosting service.

Deployment Copy Chapter

GIT_USER=<Your GitHub username> USE_SSH=true yarn deploy

If you are using GitHub pages for hosting, this command is a convenient way to build the website and push to the gh-pages branch.

File: docs/agent-binary.mdx Copy Chapter

Parca Agent from Binary

import WithVersions from '@site/src/components/WithVersions';
import CodeBlock from '@theme/CodeBlock';
import BrowserWindow from '@site/src/components/BrowserWindow';

You can download the latest agent binary release for your architecture from our releases page.

{ versions =>

curl -sL https://github.com/parca-dev/parca-agent/releases/download/{versions.agent}/parca-agent_{versions.agent.substring(1)}_`uname -s_uname -m` -o parca-agent && {'\n'}
chmod +x parca-agent

}

We also produce Docker images that can be pulled:

{ versions =>

docker pull ghcr.io/parca-dev/parca-agent:{versions.agent}

}

To start Parca Agent, run the following command with the flag:

sudo parca-agent --log-level=debug --node=binary-test --remote-store-address=localhost:7070 --remote-store-insecure

:::info

The Agent needs to run as a privileged user to load necessary eBPF programs to Kernel. Please check our FAQ for further information.

:::

The command in the above configuration targets the Parca server that is presumably running on your system. And it uses the systemd service discovery to find the cgroups that have been running on your system.

:::caution

Note that Parca Agent's systemd metadata service discovery supports cgroup v1 and v2. This is only used to enrich the metadata of collected profiles.

:::

You should see with some log lines:

/* Detailed source-code truncated for AI context efficiency. */

Now we can view the active profilers by visiting http://localhost:7071:

And all the discovered processes:

Once Parca and Parca Agent are both running, you can navigate to the web interface on the browser.
You should shortly see the Select profile... dropdown menu populate with the profiles that Parca is retrieving from itself and receiving from the Agent.

Selecting parca_agent_cpu_sample_count as profile types and clicking the Search button will retrieve the profiles from Parca Agent for the time selection (default Last Hour).

File: docs/agent-server-snap-services.mdx Copy Chapter

Parca and Parca Agent as snap services

import BrowserWindow from "@site/src/components/BrowserWindow"

:::tip

You can configure both services quickly with a simple cloud-init file:

#cloud-config
snap:
  commands:
    - [install, parca]
    - [install, parca-agent, --classic]
    - [set, parca-agent, remote-store-insecure=true]
    - [set, parca-agent, remote-store-address=localhost:7070]
    - [start, parca]
    - [start, parca-agent]

:::

Setting up Parca Copy Chapter

You can install Parca using the snap package.

sudo snap install parca
sudo snap start parca

This will start the Parca server on port 7070 and configure it to retrieve profiles from itself every 1 second automatically.

Once Parca is running, you can navigate to the web interface on the browser.

Setting up Parca Agent Copy Chapter

You can install Parca Agent using the snap package.

sudo snap install --classic parca-agent
sudo snap set parca-agent remote-store-insecure=true
sudo snap set parca-agent remote-store-address=localhost:7070
sudo snap start parca-agent

The systemd service will be collecting profiles from docker.service, parca.service and parca-agent.service that have been running on your system.

File: docs/binary.mdx Copy Chapter

Parca from Binary

import WithVersions from "@site/src/components/WithVersions";
import CodeBlock from "@theme/CodeBlock";
import BrowserWindow from "@site/src/components/BrowserWindow";

You can download the latest binary release for your architecture from our releases page.

We also produce Docker images that can be pulled:

We provide an example configuration file that you can use to get Parca running, and the scrape configuration section should look familiar to anyone familiar with Prometheus.

object_storage:
  bucket:
    type: "FILESYSTEM"
    config:
      directory: "./tmp"

scrape_configs:
  - job_name: "default"
    scrape_interval: "2s"
    static_configs:
      - targets: ["127.0.0.1:7070"]

You can download from our repository:

To start Parca, pass this config file with the config-path flag:

parca --config-path="parca.yaml"

You should be greeted with the splash and some log lines:

ooooooooo.
`888   `Y88.
 888   .d88'  .oooo.   oooo d8b  .ooooo.   .oooo.
 888ooo88P'  `P  )88b  `888""8P d88' `"Y8 `P  )88b
 888          .oP"888   888     888        .oP"888
 888         d8(  888   888     888   .o8 d8(  888
o888o        `Y888""8o d888b    `Y8bod8P' `Y888""8o



{"caller":"log.go:124","level":"info","msg":"loading bucket configuration","name":"parca","ts":"2021-09-16T01:13:18.874904009Z"}
{"caller":"log.go:124","level":"debug","msg":"Starting provider","name":"parca","provider":"static/0","subs":"[default]","ts":"2021-09-16T01:13:18.876115335Z"}
{"addr":":7070","caller":"server.go:84","level":"info","msg":"starting server","name":"parca","ts":"2021-09-16T01:13:18.876178645Z"}
{"caller":"log.go:124","level":"debug","msg":"Discoverer channel closed","name":"parca","provider":"static/0","ts":"2021-09-16T01:13:18.876192954Z"}

This will start the Parca server on port 7070 and configure it to retrieve profiles from itself every 1 second automatically.

Other flags can be found using the help flag.

/* Detailed source-code truncated for AI context efficiency. */

Once Parca is running, you can navigate to the web interface on the browser.

You should shortly see the Select profile... dropdown menu populate with the profiles that Parca is retrieving from itself.
Selecting one of these profile types and clicking the Search button will retrieve the profiles of that type for the time selection (default Last Hour)

This should result in a time series based on the profile that is interactable. Clicking anywhere on the line graph should then bring up an Flame Graph for the profile that you've selected.

You can then interact with the Flame Graph to better understand how Parca is behaving.

File: docs/binary-filtering.mdx Copy Chapter

Binary Filtering

import BrowserWindow from "@site/src/components/BrowserWindow";

Parca allows filtering profiles by their associated binary. This is useful to narrow down the scope of a profile to a specific binary or set of binaries.

How to use it Copy Section

To use the binary filter feature, you need to be using a color profile that allows for binary filtering. Click on the "Preferences" button to load the preferences modal. In the preferences modal, you will see a section called "Flamegraph Color Profile". Ensure any of the color profiles except the "Neutral" option is selected. The "Neutral" option does not support binary filtering.

To see the list of the binaries in the profile, click on the "Show Legend" button to show the legend. The legend shows the list of binaries in the profile. You can now click on any binary name to filter the profile by the binary.

The example above specifically works for when you want to only see data in the Flame
Graph for a particular binary. So how about when you want to see data in the Flame
Graph excluding a particular binary? You can do this by right-clicking on the frame
in the Flame Graph and then click on the "Hide Binary" option. This will hide the
data for the binary you right-clicked on.

This is especially useful for Python workloads that call out to native extensions, so just viewing interpreted frames is not quite right, but just hiding the Python runtime. This means other native extensions like libcuda or pytorch would still be visible.

In addition to that, you can hide multiple binaries at a time! Using the same Flame Graph as above, we also want to hide the libc.so.6 frames.

To get the Flame Graph back to its initial state, there's a "Reset Legend" button to reset the Flame Graph.

File: docs/commercial.md Copy Chapter

Commercial Solutions

Polar Signals, the original creator of the Parca project, is the only commercial provider of Parca-compatible APIs as part of the Polar Signals Cloud product.

Polar Signals also offers paid support for Parca open-source users.

File: docs/community.md Copy Chapter

Community

Code of Conduct Copy Chapter

Parca follows CNCF Code of Conduct. That includes everything conducted under the umbrella of the project, including but not limited to the Office hours, interactions on GitHub, mailing lists, and Slack.

Please contact the Parca maintainers at [email protected] to report any Code of Conduct violations.

Contact Copy Chapter

You can reach out to the rest of the Parca community including the core developers on our Discord.

Contributing Copy Chapter

If you'd like to contribute, we'd love to have you do so! For major changes, please open an issue first to discuss what you would like to change. If you are not entirely sure about this, you can discuss this on the Parca Discord server as well. If there is an existing issue you would like to work on, please comment on the issue.

For detailed instructions, please refer to the CONTRIBUTING.md files in parca and parca-agent respectively.

We use RFCs to document all things architecture and design for the Parca project. You can find an index of the RFCs here.

Parca uses GitHub to manage reviews, issues and pull requests. Contributions to Parca documentation and this
website can be made here.

File: docs/concepts.mdx Copy Chapter

Concepts

import BrowserWindow from "@site/src/components/BrowserWindow";

Data model Copy Chapter

Parca's multi-dimensional data-model originates from the Prometheus project. It allows for concise and easy to use label-selectors to slice and dice the data when querying.

Stable links Copy Chapter

Parca has a strong philosophy on having stable links. Observability projects work best when they can be shared with teams and so Parca does its best to represent the state of the frontend via the URL to be shared with other people via chat such as Discord, Slack or issue trackers like GitHub or GitLab.

Pprof Copy Chapter

Parca ingests profiles using the open-source pprof profile.proto format. More can be read about pprof and the format here.

How is this different than the pprof utility?

The pprof utility offers a robust set of tools for exploring profiles. Parca specifically tries to solve the storage and retrieval of these profiles, and allows you to explore these profiles across time.

Flame Graphs Copy Chapter

Often to better understand how your application is behaving you need to compare two profiles. Parca allows you to make selections of two points in time that you want to compare and will generate an Flame Graph of the difference between the two selected profiles.

If a span is larger in the second profile, you'll see that span colored in red, and will be darker shade the larger the difference is. This allows one to quickly visually identify where a program is growing in usage scope.

You'll see in the image below with the profile selections reversed from the previous example, that the spans that are now smaller in the second selection are colored green. This allows you to quickly visually identify where improvements to usage are being made.

If a span remains blue, it means there was no significant change between the two compared profiles for that section of code.

Cumulative and Diff values Copy Chapter

When comparing two profiles, it's not unusual to see a Cumulative value and Diff value in the tooltip that displays when you hover over the Flame Graph.

As an example, a stack trace like main() > foo() > bar() where main calls foo that calls bar. In one profile, we observe this stack trace running at 500ns, so its cumulative value will be 500 and then later in another profile, we see this stack trace but it was running for only 300ns, thus it's cumulative value will be 300.

However, if we compare the latest stack trace that was running at 300ns to the former one running 500ns, we see that the diff is 300-500 = -200. It means that compared to the first profile we've seen this same stack trace 200ns less.

File: docs/configuration.md Copy Chapter

Configuration

Parca obtains configuration information via a YAML file.

The overall structure of a config file for Parca is as follows:

object_storage:
  bucket:
    type: "FILESYSTEM"
    config:
      directory: "./tmp"

scrape_configs:
    # Section name, mandatory
  - job_name: "pr"
    # Set of query parameters with which the target is scraped.
    params: 
      # Name of keys with value(s)
      - key_1: ['value1', 'value2']
    # How frequently to scrape the targets of this scrape config.
    scrape_interval: "30s"
    # Timeout for scraping targets of this config.
    scrape_timeout: "40s"
    # URL scheme with which to fetch metrics from targets.
    scheme: https
    # Config for adding custom scrape endpoints.
    profiling_config:
      pprof_config:
        config_1: # Name of the profile.
          enabled: true
          path: /path/to/scrape
          # Add a query parameter '?seconds=<scrape-interval>' to the profile.
          delta: true
        config_2:
          enabled: true
          path: /another/path
      # String prefix to add to all paths specified in the pprof config.
      path_prefix: /debug
    # 
    static_configs:
     - targets: [ '127.0.0.1:7070' ]
    # Configuration for relabeling of target label sets.
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

See Prometheus documentation for details about relabelling.

S3 configuration Copy Chapter

Parca supports using an S3-compatible bucket for storing profile data.
The s3 configuration uses the Thanos config format.

For example:

object_storage:
  bucket:
    type: S3
    config:
      bucket: "bucky",
      endpoint: "some.s3.endpoint:8000",
      access_key: "bar",
      secret_key: "foo",
      insecure: true

File: docs/debuginfo-cli.mdx Copy Chapter

Debuginfo CLI

import WithVersions from '@site/src/components/WithVersions';
import CodeBlock from '@theme/CodeBlock';
import BrowserWindow from '@site/src/components/BrowserWindow';

The parca-debuginfo CLI is an auxiliary tool that helps you to upload debug information for your binaries.
It is useful when you do not want to deploy binaries with debug information to your production environments, yet you want to utilize Parca Agent.
By using the parca-debuginfo tool, you can extract and upload the debug information from your binaries to your Parca server and then strip the debug information from the binaries before you deploy them.

Read the docs on symbolization to understand the process.

You can download the latest parca-debuginfo binary release for your architecture from our releases page.

If you want to move the CLI to a path in your $PATH you can move the binary:

mv parca-debuginfo /usr/local/bin/parca-debuginfo

The parca-debuginfo tool has to commands:

Usage: parca-debuginfo <command>

Flags:
  -h, --help                Show context-sensitive help.
      --log-level="info"    Log level.

Commands:
  upload --store-address=STRING  ...
    Upload debug information files.

  extract  ...
    Extract debug information.

  buildid 
    Extract buildid.

Run "parca-debuginfo <command> --help" for more information on a command.

Upload command to directly extract and upload the debug information to a Parca server:

Usage: main upload --store-address=STRING  ...

Upload debug information files.

Arguments:
   ...    Paths to upload.

Flags:
  -h, --help                        Show context-sensitive help.
      --log-level="info"            Log level.

      --store-address=STRING        gRPC address to sends symbols to.
      --bearer-token=STRING         Bearer token to authenticate with store.
      --bearer-token-file=STRING    File to read bearer token from to authenticate with store.
      --insecure                    Send gRPC requests via plaintext instead of TLS.
      --insecure-skip-verify        Skip TLS certificate verification.

parca-debuginfo upload --store-address=localhost:7070 --insecure /path/to/your/binary

level=info ts=2021-10-07T12:28:30.710169775Z caller=debuginfo.go:109 msg="debug information uploaded successfully" buildid=324367736569664b3941566c467a727543434d622f576479554f6838735973614362395a71307441742f61754d4a7357616c32695a734e576b77337256472f3836554d764a506b7846656d333936726d417653 file=/path/to/your/binary
level=info ts=2021-10-07T12:28:30.710224803Z caller=main.go:154 msg=done!

And another command Extract to extract the debug information for local investigations.

Usage: main extract  ...

Extract debug information.

Arguments:
   ...    Paths to extract debug information.

Flags:
  -h, --help                Show context-sensitive help.
      --log-level="info"    Log level.

      --output-dir="out"    Output directory path to use for extracted debug information files.

parca-debuginfo extract --insecure /path/to/your/binary

level=info ts=2021-10-07T12:30:27.549546915Z caller=main.go:138 msg="debug information extracted" file=out/324367736569664b3941566c467a727543434d622f576479554f6838735973614362395a71307441742f61754d4a7357616c32695a734e576b77337256472f3836554d764a506b7846656d333936726d417653
level=info ts=2021-10-07T12:30:27.549605633Z caller=main.go:154 msg=done!

By default, you can find extracted information in out directory where you executed the tool:

ls out
324367736569664b3941566c467a727543434d622f576479554f6838735973614362395a71307441742f61754d4a7357616c32695a734e576b77337256472f3836554d764a506b7846656d333936726d417653

If all you want to do is extract the buildid of a binary, you can use the buildid command:

Usage: main buildid 

Extract buildid.

Arguments:
      Paths to extract buildid.

Flags:
  -h, --help                Show context-sensitive help.
      --log-level="info"    Log level.

For example for your own binary use:

parca-debuginfo buildid /path/to/your/binary

Build ID: 324367736569664b3941566c467a727543434d622f576479554f6838735973614362395a71307441742f61754d4a7357616c32695a734e576b77337256472f3836554d764a506b7846656d333936726d417653
level=info msg=done!