LICENSE Documentation

Official TypeScript developer resources and documentation for dundee/LICENSE. Verified technical context for AI.

0 stars TypeScript TypeScript 1 file ยท ~17,408 tokens
RAW Doc

1. Project Overview & Quickstart (dundee/LICENSE)

LICENSE

Open-source repository dundee/LICENSE

Repository Details

Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.

2. Official Technical Reference & Guides (dundee/docs)

File: README.md

Prometheus Documentation

This repository contains both the content and the static-site generator code for the
Prometheus documentation site.

Contributing Changes

See CONTRIBUTING.md for general instructions for new Prometheus contributors.

The main documentation contents of this website are located in the content/docs directory.

Documentation concerning the Prometheus server is maintained in the Prometheus server repository and cloned into the website at build time.

As a guideline, please keep the documentation generally applicable and avoid use-case-specific changes.

Prerequisites

You need to have a working Ruby environment set up (including bundler)
and then install the necessary gems:

bash
cd docs
make bundle

Building

To generate the static site, run:

bash
make build

The resulting static site will be stored in the output directory.

Optionally, you can use an API token to avoid rate limits on the API. You can get an API token from https://github.com/settings/tokens/new.

bash
export GITHUB_AUTHENTICATION='-u user:token'

Development Server

To run a local server that displays the generated site, run:

bash
# Rebuild the site whenever relevant files change:
make guard
# Start the local development server in a separate shell:
make serve

You should now be able to view the generated site at
http://localhost:3000/.

Automatic Deployment

This site is automatically deployed using Netlify.

If you have the prerequisite access rights, you can view the Netlify settings here:

Changes to the master branch are deployed to the main site at https://prometheus.io.

Netlify also creates preview deploys for every pull request. To view these for a PR where all checks have passed:

  1. In the CI section of the PR, click on "Show all checks".
  2. On the "deploy/netlify" entry, click on "Details" to view the preview site for the PR.

You may have to wait a while for the "deploy/netlify" check to appear after creating or updating the PR, even if the other checks have already passed.

License

Apache License 2.0, see LICENSE.

File: content/blog/2015-04-24-prometheus-monitring-spreads-through-the-internet.md

title: Prometheus Monitoring Spreads through the Internet created_at: 2015-04-24 kind: article author_name: Brian Brazil

It has been almost three months since we publicly announced Prometheus version
0.10.0, and we're now at version 0.13.1.

SoundCloud's announcement blog post
remains the best overview of the key components of Prometheus, but there has
been a lot of other online activity around Prometheus. This post will let you
catch up on anything you missed.

In the future, we will use this blog to publish more articles and announcements
to help you get the most out of Prometheus.

Using Prometheus

Posts on how to use Prometheus comprise the majority of online content. Here
are the ones we're aware of with the part of the ecosystem they cover:

Articles

These articles look at how Prometheus fits into the broader picture of keeping services up and running:

Philosophy

Monitoring isn't just about the technical details. How it affects the design of
your systems, operations, and human factors are important too:

The comments on the Hacker News post about Prometheus are also insightful.

Non-English

Several posts have appeared in languages beyond English:

Closing

Finally, I'd like to share how to run Prometheus on a Raspberry Pi.

File: content/blog/2015-06-01-advanced-service-discovery.md

title: Advanced Service Discovery in Prometheus 0.14.0 created_at: 2015-06-01 kind: article author_name: Fabian Reinartz, Julius Volz

This week we released Prometheus v0.14.0 โ€” a version with many long-awaited additions
and improvements.

On the user side, Prometheus now supports new service discovery mechanisms. In
addition to DNS-SRV records, it now supports Consul
out of the box, and a file-based interface allows you to connect your own
discovery mechanisms. Over time, we plan to add other common service discovery
mechanisms to Prometheus.

Aside from many smaller fixes and improvements, you can now also reload your configuration during
runtime by sending a SIGHUP to the Prometheus process. For a full list of changes, check the
changelog for this release.

In this blog post, we will take a closer look at the built-in service discovery mechanisms and provide
some practical examples. As an additional resource, see
Prometheus's configuration documentation.

Prometheus and targets

For a proper understanding of this blog post, we first need to take a look at how
Prometheus labels targets.

There are various places in the configuration file where target labels may be
set. They are applied in the following order, with later stages overwriting any
labels set by an earlier stage:

  1. Global labels, which are assigned to every target scraped by the Prometheus instance.
  2. The job label, which is configured as a default value for each scrape configuration.
  3. Labels that are set per target group within a scrape configuration.
  4. Advanced label manipulation via relabeling.

Each stage overwrites any colliding labels from the earlier stages. Eventually, we have a flat
set of labels that describe a single target. Those labels are then attached to every time series that
is scraped from this target.

Note: Internally, even the address of a target is stored in a special
__address__ label. This can be useful during advanced label manipulation
(relabeling), as we will see later. Labels starting with __ do not appear in
the final time series.

Scrape configurations and relabeling

Aside from moving from an ASCII protocol buffer format to YAML, a fundamental change to
Prometheus's configuration is the change from per-job configurations to more generalized scrape
configurations. While the two are almost equivalent for simple setups, scrape configurations
allow for greater flexibility in more advanced use cases.

Each scrape configuration defines a job name which serves as a default value for the
job label. The job label can then be redefined for entire target groups or individual targets.
For example, we can define two target groups, each of which defines targets for one job.
To scrape them with the same parameters, we can configure them as follows:

text
scrape_configs:
- job_name: 'overwritten-default'

  scrape_interval: 10s
  scrape_timeout:  5s

  target_groups:
  - targets: ['10.1.200.130:5051', '10.1.200.134:5051']
    labels:
      job: 'job1'

  - targets: ['10.1.200.130:6220', '10.1.200.134:6221']
    labels:
      job: 'job2'

Through a mechanism named relabeling,
any label can be removed, created, or modified on a per-target level. This
enables fine-grained labeling that can also take into account metadata coming
from the service discovery. Relabeling is the last stage of label assignment
and overwrites any labels previously set.

Relabeling works as follows:

  • A list of source labels is defined.
  • For each target, the values of those labels are concatenated with a separator.
  • A regular expression is matched against the resulting string.
  • A new value based on those matches is assigned to another label.

Multiple relabeling rules can be defined for each scrape configuration. A simple one
that squashes two labels into one, looks as follows:

text
relabel_configs:
- source_labels: ['label_a', 'label_b']
  separator:     ';'
  regex:         '(.*);(.*)'
  replacement:   '${1}-${2}'
  target_label:  'label_c'

This rule transforms a target with the label set:

text
{
  "job": "job1",
  "label_a": "foo",
  "label_b": "bar"
}

...into a target with the label set:

text
{
  "job": "job1",
  "label_a": "foo",
  "label_b": "bar",
  "label_c": "foo-bar"
}

You could then also remove the source labels in an additional relabeling step.

You can read more about relabeling and how you can use it to filter targets in the
configuration documentation.

Over the next sections, we will see how you can leverage relabeling when using service discovery.

Discovery with DNS-SRV records

Since the beginning, Prometheus has supported target discovery via DNS-SRV records.
The respective configuration looked like this:

text
job {
  name: "api-server"
  sd_name: "telemetry.eu-west.api.srv.example.org"
  metrics_path: "/metrics"
}

Prometheus 0.14.0 allows you to specify multiple SRV records to be queried in a
single scrape configuration, and also provides service-discovery-specific meta
information that is helpful during the relabeling phase.

When querying the the DNS-SRV records, a label named __meta_dns_name is
attached to each target. Its value is set to the SRV record name for which it was
returned. If we have structured SRV record names like telemetry.<zone>.<job>.srv.example.org,
we can extract relevant labels from it those names:

text
scrape_configs:
- job_name: 'myjob'

  dns_sd_configs:
  - names:
    - 'telemetry.eu-west.api.srv.example.org'
    - 'telemetry.us-west.api.srv.example.org'
    - 'telemetry.eu-west.auth.srv.example.org'
    - 'telemetry.us-east.auth.srv.example.org'

  relabel_configs:
  - source_labels: ['__meta_dns_name']
    regex:         'telemetry\.(.+?)\..+?\.srv\.example\.org'
    target_label:  'zone'
    replacement:   '$1'
  - source_labels: ['__meta_dns_name']
    regex:         'telemetry\..+?\.(.+?)\.srv\.example\.org'
    target_label:  'job'
    replacement:   '$1'

This will attach the zone and job label to each target based on the SRV record
it came from.

Discovery with Consul

Service discovery via Consul is now supported natively. It can be configured by defining
access parameters for our Consul agent and a list of Consul services for which we want
to query targets.

The tags of each Consul node are concatenated by a configurable separator and exposed
through the __meta_consul_tags label. Various other Consul-specific meta
labels are also provided.

Scraping all instances for a list of given services can be achieved with a simple
consul_sd_config and relabeling rules:

text
scrape_configs:
- job_name: 'overwritten-default'

  consul_sd_configs:
  - server:   '127.0.0.1:5361'
    services: ['auth', 'api', 'load-balancer', 'postgres']

  relabel_configs:
  - source_labels: ['__meta_consul_service']
    regex:         '(.*)'
    target_label:  'job'
    replacement:   '$1'
  - source_labels: ['__meta_consul_node']
    regex:         '(.*)'
    target_label:  'instance'
    replacement:   '$1'
  - source_labels: ['__meta_consul_tags']
    regex:         ',(production|canary),'
    target_label:  'group'
    replacement:   '$1'

This discovers the given services from the local Consul agent.
As a result, we get metrics for four jobs (auth, api, load-balancer, and postgres). If a node
has the production or canary Consul tag, a respective group label is assigned to the target.
Each target's instance label is set to the node name provided by Consul.

A full documentation of all configuration parameters for service discovery via Consul
can be found on the Prometheus website.

Custom service discovery

Finally, we added a file-based interface to integrate your custom service discovery or other common mechanisms
that are not yet supported out of the box.

With this mechanism, Prometheus watches a set of directories or files which define target groups.
Whenever any of those files changes, a list of target groups is read from the files and scrape targets
are extracted.
It's now our job to write a small bridge program that runs as Prometheus's side-kick.
It retrieves changes from an arbitrary service discovery mechanism and writes the target information
to the watched files as lists of target groups.

These files can either be in YAML:

text
- targets: ['10.11.150.1:7870', '10.11.150.4:7870']
  labels:
    job: 'mysql'

- targets: ['10.11.122.11:6001', '10.11.122.15:6002']
  labels:
    job: 'postgres'

...or in JSON format:

text
[
  {
    "targets": ["10.11.150.1:7870", "10.11.150.4:7870"],
    "labels": {
      "job": "mysql"
    }
  },
  {
    "targets": ["10.11.122.11:6001", "10.11.122.15:6002"],
    "labels": {
      "job": "postgres"
    }
  }
]

We now configure Prometheus to watch the tgroups/ directory in its working directory
for all .json files:

text
scrape_configs:
- job_name: 'overwritten-default'

  file_sd_configs:
  - names: ['tgroups/*.json']

What's missing now is a program that writes files to this directory. For the sake of this example,
let's assume we have all our instances for different jobs in a single denormalized MySQL table.
(Hint: you probably don't want to do service discovery this way.)

Every 30 seconds, we read all instances from the MySQL table and write the
resulting target groups into a JSON file. Note that we do not have to keep
state whether or not any targets or their labels have changed. Prometheus will
automatically detect changes and applies them to targets without interrupting
their scrape cycles.

text
import os, time, json

from itertools import groupby
from MySQLdb import connect


def refresh(cur):
    # Fetch all rows.
    cur.execute("SELECT address, job, zone FROM instances")

    tgs = []
    # Group all instances by their job and zone values.
    for key, vals in groupby(cur.fetchall(), key=lambda r: (r[1], r[2])):
        tgs.append({
            'labels': dict(zip(['job', 'zone'], key)),
            'targets': [t[0] for t in vals],
        })

    # Persist the target groups to disk as JSON file.
    with open('tgroups/target_groups.json.new', 'w') as f:
        json.dump(tgs, f)
        f.flush()
        os.fsync(f.fileno())

    os.rename('tgroups/target_groups.json.new', 'tgroups/target_groups.json')


if __name__ == '__main__':
    while True:
        with connect('localhost', 'root', '', 'test') as cur:
            refresh(cur)
        time.sleep(30)

While Prometheus will not apply any malformed changes to files, it is considered best practice to
update your files atomically via renaming, as we do in our example.
It is also recommended to split larger amounts of target groups into several files based on
logical grouping.

Conclusion

With DNS-SRV records and Consul, two major service discovery methods are now
natively supported by Prometheus. We've seen that relabeling is a powerful
approach to make use of metadata provided by service discovery mechanisms.

Make sure to take a look at the new configuration documentation
to upgrade your Prometheus setup to the new release and find out about other configuration options,
such as basic HTTP authentication and target filtering via relabeling.

We provide a migration tool that upgrades
your existing configuration files to the new YAML format.
For smaller configurations we recommend a manual upgrade to get familiar with the new format and
to preserve comments.

File: content/blog/2015-06-18-practical-anomaly-detection.md

title: Practical Anomaly Detection created_at: 2015-06-18 kind: article author_name: Brian Brazil

In his Open Letter To Monitoring/Metrics/Alerting Companies,
John Allspaw asserts that attempting "to detect anomalies perfectly, at the right time, is not possible".

I have seen several attempts by talented engineers to build systems to
automatically detect and diagnose problems based on time series data. While it
is certainly possible to get a demonstration working, the data always turned
out to be too noisy to make this approach work for anything but the simplest of
real-world systems.

All hope is not lost though. There are many common anomalies which you can
detect and handle with custom-built rules. The Prometheus query
language gives you the tools to discover
these anomalies while avoiding false positives.

Building a query

A common problem within a service is when a small number of servers are not
performing as well as the rest, such as responding with increased latency.

Let us say that we have a metric instance:latency_seconds:mean5m representing the
average query latency for each instance of a service, calculated via a
recording rule from a
Summary metric.

A simple way to start would be to look for instances with a latency
more than two standard deviations above the mean:

text
instance:latency_seconds:mean5m
> on (job) group_left()
  (
      avg by (job)(instance:latency_seconds:mean5m)
    + on (job)
      2 * stddev by (job)(instance:latency_seconds:mean5m)
  )

You try this out and discover that there are false positives when
the latencies are very tightly clustered. So you add a requirement
that the instance latency also has to be 20% above the average:

text
(
      instance:latency_seconds:mean5m
    > on (job) group_left()
      (
          avg by (job)(instance:latency_seconds:mean5m)
        + on (job)
          2 * stddev by (job)(instance:latency_seconds:mean5m)
      )
  )
> on (job) group_left()
  1.2 * avg by (job)(instance:latency_seconds:mean5m)

Finally, you find that false positives tend to happen at low traffic levels.
You add a requirement for there to be enough traffic for 1 query per second to
be going to each instance. You create an alert definition for all of this:

yaml
groups:
- name: Practical Anomaly Detection
  rules:
  - alert: InstanceLatencyOutlier
    expr: >
      (
            (
                instance:latency_seconds:mean5m
              > on (job) group_left()
                (
                    avg by (job)(instance:latency_seconds:mean5m)
                  + on (job)
                    2 * stddev by (job)(instance:latency_seconds:mean5m)
                )
            )
          > on (job) group_left()
            1.2 * avg by (job)(instance:latency_seconds:mean5m)
        and on (job)
            avg by (job)(instance:latency_seconds_count:rate5m)
          >
            1
      )
    for: 30m

Automatic actions

The above alert can feed into the
Alertmanager, and from there to
your chat, ticketing, or paging systems. After a while you might discover that the
usual cause of the alert is something that there is not a proper fix for, but there is an
automated action such as a restart, reboot, or machine replacement that resolves
the issue.

Rather than having humans handle this repetitive task, one option is to
get the Alertmanager to send the alert to a web service that will perform
the action with appropriate throttling and safety features.

The generic webhook
sends alert notifications to an HTTP endpoint of your choice. A simple Alertmanager
configuration that uses it could look like this:

text
# A simple notification configuration which only sends alert notifications to
# an external webhook.
receivers:
- name: restart_webhook
  webhook_configs:
    url: "http://example.org/my/hook"

route:
  receiver: restart_webhook

Summary

The Prometheus query language allows for rich processing of your monitoring
data. This lets you to create alerts with good signal-to-noise ratios, and the
Alertmanager's generic webhook support can trigger automatic remediations.
This all combines to enable oncall engineers to focus on problems where they can
have the most impact.

When defining alerts for your services, see also our alerting best practices.

File: content/blog/2015-06-24-monitoring-dreamhack.md

title: Monitoring DreamHack - the World's Largest Digital Festival created_at: 2015-06-24 kind: article author_name: Christian Svensson (DreamHack Network Team)

Editor's note: This article is a guest post written by a Prometheus user.

If you are operating the network for 10,000's of demanding gamers, you need to
really know what is going on inside your network. Oh, and everything needs to be
built from scratch in just five days.

If you have never heard about DreamHack before, here
is the pitch: Bring 20,000 people together and have the majority of them bring
their own computer. Mix in professional gaming (eSports), programming contests,
and live music concerts. The result is the world's largest festival dedicated
solely to everything digital.

To make such an event possible, there needs to be a lot of infrastructure in
place. Ordinary infrastructures of this size take months to build, but the crew
at DreamHack builds everything from scratch in just five days. This of course
includes stuff like configuring network switches, but also building the
electricity distribution, setting up stores for food and drinks, and even
building the actual tables.

The team that builds and operates everything related to the network is
officially called the Network team, but we usually refer to ourselves as tech
or dhtech. This post is going to focus on the work of dhtech and how we used
Prometheus during DreamHack Summer 2015 to try to kick our monitoring up another
notch.

The equipment

Turns out that to build a highly performant network for 10,000+
computers, you need at least the same number of network ports. In our case these
come in the form of ~400 Cisco 2950 switches. We call these the access switches.
These are everywhere in the venue where participants will be seated with their
computers.


Dutifully standing in line, the access switches are ready to greet the
DreamHackers with high-speed connectivity.

Obviously just connecting all these computers to a switch is not enough. That
switch needs to be connected to the other switches as well. This is where the
distribution switches (or dist switches) come into play. These are switches that
take the hundreds of links from all access switches and aggregate them into
more manageable 10-Gbit/s high-capacity fibre. The dist switches are then
further aggregated into our core, where the traffic is routed to its
destination.

On top of all of this, we operate our own WiFi networks, DNS/DHCP servers, and
other infrastructure. When completed, our core looks something like the image
below.


The planning map for the distribution and core layers. The core is
clearly visible in "Hall D"

All in all this is becoming a lengthy list of stuff to monitor, so let's get to
the reason you're here: How do we make sure we know what's going on?

Introducing: dhmon

dhmon is the collective name of the systems that not only
monitor the network, but also allow other teams to collect metrics on whatever
they want.

Since the network needs to be built in five days, it's essential that the
monitoring systems are easy to set up and keep in sync if we need to do last
minute infrastructural changes (like adding or removing devices). When we start
to build the network, we need monitoring as soon as possible to be able to
discover any problems with the equipment or other issues we hadn't foreseen.

In the past we have tried to use a mix of commonly available software such as
Cacti, SNMPc, and Opsview among others. While these have worked they have focused on
being closed systems and only provided the bare minimum. A few years back a few
people from the team said "Enough, we can do better ourselves!" and started
writing a custom monitoring solution.

At the time the options were limited. Over the years the system went from using
Graphite (scalability issues), a custom Cassandra store (high complexity), and
InfluxDB (immature software) to finally land on using Prometheus. I first
learned about Prometheus back in 2014 when I met Julius Volz and I had been
eager to try it ever since. This summer we finally replaced the custom
InfluxDB-based metrics store that we had written with Prometheus. Spoiler: We're
not going back.

The architecture

The monitoring solution consists of three layers:
collection, storage, presentation. Our most critical collectors are
snmpcollector (SNMP) and ipplan-pinger (ICMP), closely followed by dhcpinfo
(DHCP lease stats). We also have some scripts that dump stats about other
systems into node_exporter's
textfile collector.


The current architecture plan of dhmon as of Summer 2015

We use Prometheus as a central timeseries storage and querying engine, but we
also use Redis and memcached to export snapshot views of binary information that
we collect but cannot store in Prometheus in any sensible way, or when we need
to access very fresh data.

One such case is in our presentation layer. We use our dhmap web application to
get an overview of the overall health of the access switches. In order to be
effective at resolving errors, we need a latency of ~10 seconds from data
collection to presentation. Our goal is to have fixed the problem before the
customer notices, or at least before they have walked over to the support people
to report an issue. For this reason, we have been using memcached since the
beginning to access the latest snapshot of the network.

We continued to use memcached this year for our low-latency data, while using
Prometheus for everything that's historical or not as latency-sensitive. This
decision was made simply because we were unsure how Prometheus would perform at
very short sampling intervals. In the end, we found no reason for why we can't
use Prometheus for this data as well - we will definitely try to replace our
memcached with Prometheus at the next DreamHack.


The overview of our access layer visualized by dhmon

Prometheus setup

The block that so far has been referred to as Prometheus
really consists of three products:
Prometheus,
PromDash, and
Alertmanager. The setup is fairly
basic and all three components are running on the same host. Everything is
served by an Apache web server that just acts as a reverse proxy.

text
ProxyPass /prometheus http://localhost:9090/prometheus
ProxyPass /alertmanager http://localhost:9093/alertmanager
ProxyPass /dash http://localhost:3000/dash

Exploring the network

Prometheus has a powerful querying engine that allows
you to do pretty cool things with the streaming information collected from all
over your network. However, sometimes the queries need to process too much data
to finish within a reasonable amount of time. This happened to us when we wanted
to graph the top 5 utilized links out of ~18,000 in total. While the query
worked, it would take roughly the amount of time we set our timeout limit to,
meaning it was both slow and flaky. We decided to use Prometheus' recording
rules for precomputing heavy queries.

text
precomputed_link_utilization_percent = rate(ifHCOutOctets{layer!='access'}[10m])*8/1000/1000
                                         / on (device,interface,alias)
                                       ifHighSpeed{layer!='access'}

After this, running topk(5, precomputed_link_utilization_percent) was
blazingly fast.

Being reactive: alerting

So at this stage we had something we could query for
the state of the network. Since we are humans, we don't want to spend our time
running queries all the time to see if things are still running as they should,
so obviously we need alerting.

For example: we know that all our access switches use GigabitEthernet0/2 as an
uplink. Sometimes when the network cables have been in storage for too long they
oxidize and are not able to negotiate the full 1000 Mbps that we want.

The negotiated speed of a network port can be found in the SNMP OID
IF-MIB::ifHighSpeed. People familiar with SNMP will however recognize that
this OID is indexed by an arbitrary interface index. To make any sense of this
index, we need to cross-reference it with data from SNMP OID IF-MIB::ifDescr
to retrieve the actual interface name.

Fortunately, our snmpcollector supports this kind of cross-referencing while
generating Prometheus metrics. This allows us in a simple way to not only query
data, but also define useful alerts. In our setup we configured the SNMP
collection to annotate any metric under the IF-MIB::ifTable and
IF-MIB::ifXTable OIDs with ifDescr. This will come in handy now when we need
to specify that we are only interested in the GigabitEthernet0/2 port and no
other interface.

Let's have a look at what such an alert definition looks like.

text
ALERT BadUplinkOnAccessSwitch
  IF ifHighSpeed{layer='access', interface='GigabitEthernet0/2'} < 1000 FOR 2m
  SUMMARY "Interface linking at {{$value}} Mbps"
  DESCRIPTION "Interface {{$labels.interface}} on {{$labels.device}} linking at {{$value}} Mbps"

Done! Now we will get an alert if a switch's uplink suddenly links at a
non-optimal speed.

Let's also look at how an alert for an almost full DHCP scope looks like:

text
ALERT DhcpScopeAlmostFull
  IF ceil((dhcp_leases_current_count / dhcp_leases_max_count)*100) > 90 FOR 2m
  SUMMARY "DHCP scope {{$labels.network}} is almost full"
  DESCRIPTION "DHCP scope {{$labels.network}} is {{$value}}% full"

We found the syntax to define alerts easy to read and understand even if you had
no previous experience with Prometheus or time series databases.


Oops! Turns out we have some bad uplinks, better run out and fix
it!

Being proactive: dashboards

While alerting is an essential part of
monitoring, sometimes you just want to have a good overview of the health of
your network. To achieve this we used PromDash.
Every time someone asked us something about the network, we crafted a query to
get the answer and saved it as a dashboard widget. The most interesting ones
were then added to an overview dashboard that we proudly displayed.


The DreamHack Overview dashboard powered by PromDash

The future

While changing an integral part of any system is a complex job and
we're happy that we managed to integrate Prometheus in just one event, there are
without a doubt a lot of areas to improve. Some areas are pretty basic: using
more precomputed metrics to improve performance, adding more alerts, and tuning
the ones we have. Another area is to make it easier for operators: creating an
alert dashboard suitable for our network operations center (NOC), figuring out
if we want to page the people on-call, or just let the NOC escalate alerts.

Some bigger features we're planning on adding: syslog analysis (we have a lot of
syslog!), alerts from our intrusion detection systems, integrating with our
Puppet setup, and also integrating more across the different teams at DreamHack.
We managed to create a proof-of-concept where we got data from one of the
electrical current sensors into our monitoring, making it easy to see if a
device is faulty or if it simply doesn't have any electricity anymore. We're
also working on integrating with the point-of-sale systems that are used in the
stores at the event. Who doesn't want to graph the sales of ice cream?

Finally, not all services that the team operates are on-site, and some even run
24/7 after the event. We want to monitor these services with Prometheus as well,
and in the long run when Prometheus gets support for federation, utilize the
off-site Prometheus to replicate the metrics from the event Prometheus.

Closing words

We're really excited about Prometheus and how easy it makes
setting up scalable monitoring and alerting from scratch.

A huge shout-out to everyone that helped us in #prometheus on
FreeNode during the event. Special thanks to Brian
Brazil, Fabian Reinartz and Julius Volz. Thanks for helping us even in the cases
where it was obvious that we hadn't read the documentation thoroughly enough.

Finally, dhmon is all open-source, so head over to https://github.com/dhtech/
and have a look if you're interested. If you feel like you would like to be a
part of this, just head over to #dreamhack on
QuakeNet and have a chat with us. Who knows, maybe
you will help us build the next DreamHack?

File: content/blog/2015-08-17-service-discovery-with-etcd.md

title: Custom service discovery with etcd created_at: 2015-08-17 kind: article author_name: Fabian Reinartz

In a previous post we
introduced numerous new ways of doing service discovery in Prometheus.
Since then a lot has happened. We improved the internal implementation and
received fantastic contributions from our community, adding support for
service discovery with Kubernetes and Marathon. They will become available
with the release of version 0.16.

We also touched on the topic of custom service discovery.

Not every type of service discovery is generic enough to be directly included
in Prometheus. Chances are your organisation has a proprietary
system in place and you just have to make it work with Prometheus.
This does not mean that you cannot enjoy the benefits of automatically
discovering new monitoring targets.

In this post we will implement a small utility program that connects a custom
service discovery approach based on etcd, the
highly consistent distributed key-value store, to Prometheus.

Targets in etcd and Prometheus

Our fictional service discovery system stores services and their
instances under a well-defined key schema:

text
/services/<service_name>/ =

Prometheus should now automatically add and remove targets for all existing
services as they come and go.
We can integrate with Prometheus's file-based service discovery, which
monitors a set of files that describe targets as lists of target groups in
JSON format.

A single target group consists of a list of addresses associated with a set of
labels. Those labels are attached to all time series retrieved from those
targets.
One example target group extracted from our service discovery in etcd could
look like this:

text
{
  "targets": ["10.0.33.1:54423", "10.0.34.12:32535"],
  "labels": {
    "job": "node_exporter"
  }
}

The program

What we need is a small program that connects to the etcd cluster and performs
a lookup of all services found in the /services path and writes them out into
a file of target groups.

Let's get started with some plumbing. Our tool has two flags: the etcd server
to connect to and the file to which the target groups are written. Internally,
the services are represented as a map from service names to instances.
Instances are a map from the instance identifier in the etcd path to its
address.

text
const servicesPrefix = "/services"

type (
  instances map[string]string
  services  map[string]instances
)

var (
  etcdServer = flag.String("server", "http://127.0.0.1:4001", "etcd server to connect to")
  targetFile = flag.String("target-file", "tgroups.json", "the file that contains the target groups")
)

Our main function parses the flags and initializes our object holding the
current services. We then connect to the etcd server and do a recursive read
of the /services path.
We receive the subtree for the given path as a result and call srvs.handle,
which recursively performs the srvs.update method for each node in the
subtree. The update method modifies the state of our srvs object to be
aligned with the state of our subtree in etcd.
Finally, we call srvs.persist which transforms the srvs object into a list
of target groups and writes them out to the file specified by the
-target-file flag.

text
func main() {
  flag.Parse()

  var (
    client  = etcd.NewClient([]string{*etcdServer})
    srvs    = services{}
  )

  // Retrieve the subtree of the /services path.
  res, err := client.Get(servicesPrefix, false, true)
  if err != nil {
    log.Fatalf("Error on initial retrieval: %s", err)
  }
  srvs.handle(res.Node, srvs.update)
  srvs.persist()
}

Let's assume we have this as a working implementation. We could now run this
tool every 30 seconds to have a mostly accurate view of the current targets in
our service discovery.

But can we do better?

The answer is yes. etcd provides watches, which let us listen for updates on
any path and its sub-paths. With that, we are informed about changes
immediately and can apply them immediately. We also don't have to work through
the whole /services subtree again and again, which can become important for
a large number of services and instances.

We extend our main function as follows:

text
func main() {
  // ...

  updates := make(chan *etcd.Response)

  // Start recursively watching for updates.
  go func() {
    _, err := client.Watch(servicesPrefix, 0, true, updates, nil)
    if err != nil {
      log.Errorln(err)
    }
  }()

  // Apply updates sent on the channel.
  for res := range updates {
    log.Infoln(res.Action, res.Node.Key, res.Node.Value)

    handler := srvs.update
    if res.Action == "delete" {
      handler = srvs.delete
    }
    srvs.handle(res.Node, handler)
    srvs.persist()
  }
}

We start a goroutine that recursively watches for changes to entries in
/services. It blocks forever and sends all changes to the updates channel.
We then read the updates from the channel and apply it as before. In case an
instance or entire service disappears however, we call srvs.handle using the
srvs.delete method instead.

We finish each update by another call to srvs.persist to write out the
changes to the file Prometheus is watching.

Modification methods

So far so good โ€“ conceptually this works. What remains are the update and
delete handler methods as well as the persist method.

update and delete are invoked by the handle method which simply calls
them for each node in a subtree, given that the path is valid:

text
var pathPat = regexp.MustCompile(`/services/([^/]+)(?:/(\d+))?`)

func (srvs services) handle(node *etcd.Node, handler func(*etcd.Node)) {
  if pathPat.MatchString(node.Key) {
    handler(node)
  } else {
    log.Warnf("unhandled key %q", node.Key)
  }

  if node.Dir {
    for _, n := range node.Nodes {
      srvs.handle(n, handler)
    }
  }
}

`update`

The update methods alters the state of our services object
based on the node which was updated in etcd.

text
func (srvs services) update(node *etcd.Node) {
  match := pathPat.FindStringSubmatch(node.Key)
  // Creating a new job directory does not require any action.
  if match[2] == "" {
    return
  }
  srv := match[1]
  instanceID := match[2]

  // We received an update for an instance.
  insts, ok := srvs[srv]
  if !ok {
    insts = instances{}
    srvs[srv] = insts
  }
  insts[instanceID] = node.Value
}

`delete`

The delete methods removes instances or entire jobs from our services
object depending on which node was deleted from etcd.

text
func (srvs services) delete(node *etcd.Node) {
  match := pathPat.FindStringSubmatch(node.Key)
  srv := match[1]
  instanceID := match[2]

  // Deletion of an entire service.
  if instanceID == "" {
    delete(srvs, srv)
    return
  }

  // Delete a single instance from the service.
  delete(srvs[srv], instanceID)
}

`persist`

The persist method transforms the state of our services object into a list of TargetGroups. It then writes this list into the -target-file in JSON
format.

text
type TargetGroup struct {
  Targets []string          `json:"targets,omitempty"`
  Labels  map[string]string `json:"labels,omitempty"`
}

func (srvs services) persist() {
  var tgroups []*TargetGroup
  // Write files for current services.
  for job, instances := range srvs {
    var targets []string
    for _, addr := range instances {
      targets = append(targets, addr)
    }

    tgroups = append(tgroups, &TargetGroup{
      Targets: targets,
      Labels:  map[string]string{"job": job},
    })
  }

  content, err := json.Marshal(tgroups)
  if err != nil {
    log.Errorln(err)
    return
  }

  f, err := create(*targetFile)
  if err != nil {
    log.Errorln(err)
    return
  }
  defer f.Close()

  if _, err := f.Write(content); err != nil {
    log.Errorln(err)
  }
}

Taking it live

All done, so how do we run this?

We simply start our tool with a configured output file:

text
./etcd_sd -target-file /etc/prometheus/tgroups.json

Then we configure Prometheus with file based service discovery
using the same file. The simplest possible configuration looks like this:

text
scrape_configs:
- job_name: 'default' # Will be overwritten by job label of target groups.
  file_sd_configs:
  - names: ['/etc/prometheus/tgroups.json']

And that's it. Now our Prometheus stays in sync with services and their
instances entering and leaving our service discovery with etcd.

Conclusion

If Prometheus does not ship with native support for the service discovery of
your organisation, don't despair. Using a small utility program you can easily
bridge the gap and profit from seamless updates to the monitored targets.
Thus, you can remove changes to the monitoring configuration from your
deployment equation.

A big thanks to our contributors Jimmy Dyson
and Robert Jacob for adding native support
for Kubernetes and Marathon.
Also check out Keegan C Smith's take on EC2 service discovery based on files.

You can find the full source of this blog post on GitHub.

File: content/blog/2016-01-26-one-year-of-open-prometheus-development.md

title: One Year of Open Prometheus Development created_at: 2016-01-26 kind: article author_name: Julius Volz

The beginning

A year ago today, we officially announced Prometheus to the wider world. This
is a great opportunity for us to look back and share some of the wonderful
things that have happened to the project since then. But first, let's start at
the beginning.

Although we had already started Prometheus as an open-source project on GitHub in
2012, we didn't make noise about it at first. We wanted to give the project
time to mature and be able to experiment without friction. Prometheus was
gradually introduced for production monitoring at
SoundCloud in 2013 and then saw more and more
usage within the company, as well as some early adoption by our friends at
Docker and Boxever in 2014. Over the years, Prometheus was growing more and
more mature and although it was already solving people's monitoring problems,
it was still unknown to the wider public.

Going public

Everything changed for us a year ago, in January of 2015. After more than two
years of development and internal usage, we felt that Prometheus was ready for
a wider audience and decided to go fully public with our official announcement
blog post,
a website, and a series of
related
posts.
We already received a good deal of attention during the first week after the
announcement, but nothing could prepare us for what happened a week later:
someone unknown to us (hello there,
jjwiseman!) had submitted
the Prometheus website to Hacker News and somehow their
post had made it all the way to the top.

This is when things started going slightly crazy in a good way. We saw a sharp
rise in contributors, mailing list questions, GitHub issues, IRC visitors,
requests for conference and meetup talks, and increasing buzz on the net in
general. Since the beginning, we have been very lucky about the quality of our
newly expanded community: The kind of people who were attracted to Prometheus
also turned out to be very competent, constructive, and high-quality
contributors and users. The ideal open-source scenario of receiving a lot of
value back from the community was a reality pretty much from day one.

What does all that Hacker News buzz look like in terms of GitHub stars? Try and
see if you can find the exact moment in this graph (ironically, a Gnuplot and
not Prometheus graph) when we went out of "dark mode" and got hit by Hacker
News:

This attention also put us in the 4th place of GitHub's trending repositories
worldwide:

After the first wave

After those first weeks, the initial onslaught of incoming communication cooled
down a bit, but we were and still are receiving constantly growing adoption.

To give you an idea of the ecosystem, we now have:

  • 33 repositories in our GitHub organization
  • ~4800 total GitHub stars
  • 200+ contributors
  • 2300+ pull requests (60+ open)
  • 1100+ issues (300+ open)
  • 150+ people in our IRC channel (#prometheus on FreeNode)
  • 250+ people on the mailing list who have created 300+ threads
  • 20+ Prometheus-related talks and workshops
  • 100+ articles and blog posts

Besides countless smaller features and bug fixes to existing projects, the
community has contributed many projects of their own to the Prometheus
ecosystem. Most of them are exporters that translate metrics from existing
systems into Prometheus's data model, but there have also been important
additions to Prometheus itself, such as service discovery mechanisms for
Kubernetes,
Marathon and
EC2.

Shortly after making more noise about Prometheus, we also found one contributor
(Fabian) so outstanding that he ended up joining
SoundCloud to work on Prometheus. He has since become the most active developer
on the project and we have him to thank for major new features such as
generalized service discovery support, runtime-reloadable configurations, new
powerful query language features, a custom-built query parser, and so much
more. He is currently working on the new beta rewrite of the
Alertmanager.

Finally, we have been honored to be recognized and adopted by major players in
the industry. Google is now instrumenting its open-source
container management system Kubernetes natively with
Prometheus metrics. CoreOS is picking it up for
etcd's monitoring as well. DigitalOcean is betting on Prometheus for their
internal monitoring. By now, the list of companies using Prometheus in one way
or another has become too long to mention all of them:
Google,
CoreOS, Docker,
Boxever,
DigitalOcean, Financial Times,
Improbable, KPMG, and many more.
Even the world's largest digital festival,
DreamHack, has used
Prometheus to keep
tabs on their network infrastructure in 2015, and
FOSDEM will do so in 2016.

The widely popular dashboard builder Grafana also added
native Prometheus backend support in version
2.5. Since
people all around the world are already using and loving Grafana, we are going
to focus on improving Grafana's Prometheus integration and will invest
less energy in our own dashboard builder
PromDash in the future.

With the Prometheus ecosystem continuing to grow, the first users have started
asking about commercial support. While Prometheus will always remain an
independent open source project, one of our core contributors (Brian
Brazil) has recently founded his own company,
Robust Perception, which provides support
and consulting services around Prometheus and monitoring in general.

On a lighter note, 2015 has also been the year in which Brian proved Prometheus's query
language to be Turing complete by implementing
Conway's Game of Life in PromQL.

The road ahead

Both personally and technically, we are really excited about what has happened
last year in Prometheus-land. We love the opportunity to provide the world with
a powerful new approach to monitoring, especially one that is much better
suited towards modern cloud- and container-based infrastructures than
traditional solutions. We are also very grateful to all contributors and
hope to continuously improve Prometheus for everyone.

Although Prometheus is relatively mature by now, we have a list of major goals
we want to tackle in 2016. The highlights will be polishing the new
Alertmanager rewrite, supporting full read and write integration for external
long-term storage, as well as eventually releasing a stable 1.0 version of the
Prometheus server itself.

Stay tuned!

File: content/blog/2016-03-03-custom-alertmanager-templates.md

title: Custom Alertmanager Templates created_at: 2016-03-03 kind: article author_name: Fabian Reinartz

The Alertmanager handles alerts sent by Prometheus servers and sends
notifications about them to different receivers based on their labels.

A receiver can be one of many different integrations such as PagerDuty, Slack,
email, or a custom integration via the generic webhook interface (for example JIRA).

Templates

The messages sent to receivers are constructed via templates.
Alertmanager comes with default templates but also allows defining custom
ones.

In this blog post, we will walk through a simple customization of Slack
notifications.

We use this simple Alertmanager configuration that sends all alerts to Slack:

yaml
global:
  slack_api_url: '<slack_webhook_url>'

route:
  receiver: 'slack-notifications'
  # All alerts in a notification have the same value for these labels.
  group_by: [alertname, datacenter, app]

receivers:
- name: 'slack-notifications'
  slack_configs:
  - channel: '#alerts'

By default, a Slack message sent by Alertmanager looks like this:

It shows us that there is one firing alert, followed by the label values of
the alert grouping (alertname, datacenter, app) and further label values the
alerts have in common (critical).

Customize

If you have alerts, you should also have documentation on how to handle them โ€“
a runbook. A good approach to that is having a wiki that has a section for
each app you are running with a page for each alert.

Suppose we have such a wiki running at https://internal.myorg.net/wiki/alerts.
Now we want links to these runbooks shown in our Slack messages.

In our template, we need access to the "alertname" and the "app" label. Since
these are labels we group alerts by, they are available in the GroupLabels
map of our templating data.

We can directly add custom templating to our Alertmanager's Slack configuration
that is used for the text section of our Slack message.
The templating language is the one provided
by the Go programming language.

yaml
global:
  slack_api_url: '<slack_webhook_url>'

route:
- receiver: 'slack-notifications'
  group_by: [alertname, datacenter, app]

receivers:
- name: 'slack-notifications'
  slack_configs:
  - channel: '#alerts'
    # Template for the text field in Slack messages.
    text: 'https://internal.myorg.net/wiki/alerts/{{ .GroupLabels.app }}/{{ .GroupLabels.alertname }}'

We reload our Alertmanager by sending a SIGHUP or restart it to load the
changed configuration. Done.

Our Slack notifications now look like this:

Template files

Alternatively, we can also provide a file containing named templates, which
are then loaded by Alertmanager. This is especially helpful for more complex
templates that span many lines.

We create a file /etc/alertmanager/templates/myorg.tmpl and create a
template in it named "slack.myorg.text":

text
{{ define "slack.myorg.text" }}https://internal.myorg.net/wiki/alerts/{{ .GroupLabels.app }}/{{ .GroupLabels.alertname }}{{ end}}

Our configuration now loads the template with the given name for the "text"
field and we provide a path to our custom template file:

yaml
global:
  slack_api_url: '<slack_webhook_url>'

route:
- receiver: 'slack-notifications'
  group_by: [alertname, datacenter, app]

receivers:
- name: 'slack-notifications'
  slack_configs:
  - channel: '#alerts'
    text: '{{ template "slack.myorg.text" . }}'

templates:
- '/etc/alertmanager/templates/myorg.tmpl'

We reload our Alertmanager by sending a SIGHUP or restart it to load the
changed configuration and the new template file. Done.

To test and iterate on your Prometheus Alertmanager notification templates for Slack you can use the following tool.

File: content/blog/2016-03-23-interview-with-life360.md

title: Interview with Life360 created_at: 2016-03-23 kind: article author_name: Brian Brazil

This is the first in a series of interviews with users of Prometheus, allowing
them to share their experiences of evaluating and using Prometheus. Our first
interview is with Daniel from Life360.

Can you tell us about yourself and what Life360 does?

Iโ€™m Daniel Ben Yosef, a.k.a, dby, and Iโ€™m an Infrastructure Engineer for
Life360, and before that, Iโ€™ve held systems
engineering roles for the past 9 years.

Life360 creates technology that helps families stay connected, weโ€™re the Family
Network app for families. Weโ€™re quite busy handling these families - at peak
we serve 700k requests per minute for 70 million registered families.

We manage around 20 services in production, mostly handling location requests
from mobile clients (Android, iOS, and Windows Phone), spanning over 150+
instances at peak. Redundancy and high-availability are our goals and we strive
to maintain 100% uptime whenever possible because families trust us to be
available.

We hold user data in both our MySQL multi-master cluster and in our 12-node
Cassandra ring which holds around 4TB of data at any given time. We have
services written in Go, Python, PHP, as well as plans to introduce Java to our
stack. We use Consul for service discovery, and of course our Prometheus setup
is integrated with it.

What was your pre-Prometheus monitoring experience?

Our monitoring setup, before we switched to Prometheus, included many
components such as:

  • Copperegg (now Idera)
  • Graphite + Statsd + Grafana
  • Sensu
  • AWS Cloudwatch

We primarily use MySQL, NSQ and HAProxy and we found that all of the monitoring
solutions mentioned above were very partial, and required a lot of
customization to actually get all working together.

Why did you decide to look at Prometheus?

We had a few reasons for switching to Prometheus, one of which is that we
simply needed better monitoring.

Prometheus has been known to us for a while, and we have been tracking it and
reading about the active development, and at a point (a few months back) we
decided to start evaluating it for production use.

The PoC results were incredible. The monitoring coverage of MySQL was amazing,
and we also loved the JMX monitoring for Cassandra, which had been sorely
lacking in the past.

How did you transition?

We started with a relatively small box (4GB of memory) as an initial point. It
was effective for a small number of services, but not for our full monitoring
needs.

We also initially deployed with Docker, but slowly transitioned to its own box
on an r3.2xl instance (60GB ram), and that holds all of our service monitoring
needs with 30 days of in-memory data.

We slowly started introducing all of our hosts with the Node Exporter and built
Grafana graphs, up to the point where we had total service coverage.

We were also currently looking at InfluxDB for long term storage, but due to
recent developments,
this may no longer be a viable option.

We then added exporters for MySQL, Node, Cloudwatch, HAProxy, JMX, NSQ (with a
bit of our own code), Redis and Blackbox (with our own contribution to add
authentication headers).

What improvements have you seen since switching?

The visibility and instrumentation gain was the first thing we saw. Right
before switching, we started experiencing Graphiteโ€™s scalability issues, and
having an in-place replacement for Graphite so stakeholders can continue to use
Grafana as a monitoring tool was extremely valuable to us. Nowadays, we are
focusing on taking all that data and use it to detect anomalies, which will
eventually become alerts in the Alert Manager.

What do you think the future holds for Life360 and Prometheus?

We currently have one of our projects instrumented directly with a Prometheus
client, a Python-based service. As we build out new services, Prometheus is
becoming our go-to for instrumentation, and will help us gain extremely
meaningful alerts and stats about our infrastructure.

We look forward to growing with the project and keep contributing.

Thank you Daniel! The source for Life360's dashboards is shared on Github.

File: content/blog/2016-05-01-interview-with-showmax.md

title: Interview with ShowMax created_at: 2016-05-01 kind: article author_name: Brian Brazil

This is the second in a series of interviews with users of Prometheus, allowing
them to share their experiences of evaluating and using Prometheus.

Can you tell us about yourself and what ShowMax does?

Iโ€™m Antonin Kral, and Iโ€™m leading research and architecture for
ShowMax. Before that, Iโ€™ve held architectural and CTO
roles for the past 12 years.

ShowMax is a subscription video on demand service that launched in South Africa
in 2015. Weโ€™ve got an extensive content catalogue with more than 20,000
episodes of TV shows and movies. Our service is currently available in 65
countries worldwide. While better known rivals are skirmishing in America and
Europe, ShowMax is battling a more difficult problem: how do you binge-watch
in a barely connected village in sub-Saharan Africa? Already 35% of video
around the world is streamed, but there are still so many places the revolution
has left untouched.

We are managing about 50 services running mostly on private clusters built
around CoreOS. They are primarily handling API requests from our clients
(Android, iOS, AppleTV, JavaScript, Samsung TV, LG TV etc), while some of them
are used internally. One of the biggest internal pipelines is video encoding
which can occupy 400+ physical servers when handling large ingestion batches.

The majority of our back-end services are written in Ruby, Go or Python. We use
EventMachine when writing apps in Ruby (Goliath on MRI, Puma on JRuby). Go is
typically used in apps that require large throughput and donโ€™t have so much
business logic. Weโ€™re very happy with Falcon for services written in Python.
Data is stored in PostgreSQL and ElasticSearch clusters. We use etcd and custom
tooling for configuring Varnishes for routing requests.

What was your pre-Prometheus monitoring experience?

The primary use-cases for monitoring systems are:

  • Active monitoring and probing (via Icinga)
  • Metrics acquisition and creation of alerts based on these metrics (now Prometheus)
  • Log acquisition from backend services
  • Event and log acquisition from apps

The last two use-cases are handled via our logging infrastructure. It consists
of a collector running in the service container, which is listening on local
Unix socket. The socket is used by apps to send messages to the outside world.
Messages are transferred via RabbitMQ servers to consumers. Consumers are
custom written or hekad based. One of the main message flows is going towards
the service ElasticSearch cluster, which makes logs accessible for Kibana and
ad-hoc searches. We also save all processed events to GlusterFS for archival
purposes and/or further processing.

We used to run two metric acquisition pipelines in parallel. The first is based
on Collectd + StatsD + Graphite + Grafana and the other using Collectd +
OpenTSDB. We have struggled considerably with both pipelines. We had to deal
with either the I/O hungriness of Graphite, or the complexity and inadequate
tooling around OpenTSDB.

Why did you decide to look at Prometheus?

After learning from our problems with the previous monitoring system, we looked
for a replacement. Only a few solutions made it to our shortlist. Prometheus
was one of the first, as Jiri Brunclik, our head of Operations at the time, had
received a personal recommendation about the system from former colleagues at
Google.

The proof of concept went great. We got a working system very quickly. We also
evaluated InfluxDB as a main system as well as a long-term storage for
Prometheus. But due to recent developments, this may no longer be a viable
option for us.

How did you transition?

We initially started with LXC containers on one of our service servers, but
quickly moved towards a dedicated server from Hetzner, where we host the
majority of our services. Weโ€™re using PX70-SSD, which is Intelยฎ Xeonยฎ E3-1270
v3 Quad-Core Haswell with 32GB RAM, so we have plenty of power to run
Prometheus. SSDs allow us to have retention set to 120 days. Our logging
infrastructure is built around getting logs locally (receiving them on Unix
socket) and then pushing them towards the various workers.

Having this infrastructure available made pushing metrics a logical choice
(especially in pre-Prometheus times). On the other side, Prometheus is
primarily designed around the paradigm of scraping metrics. We wanted to stay
consistent and push all metrics towards Prometheus initially. We have created a
Go daemon called prometheus-pusher. Itโ€™s responsible for scraping metrics from
local exporters and pushing them towards the Pushgateway. Pushing metrics has
some positive aspects (e.g. simplified service discovery) but also quite a few
drawbacks (e.g. making it hard to distinguish between a network partition vs. a
crashed service). We made Prometheus-pusher available on
GitHub, so you can try it
yourself.

The next step was for us to figure out what to use for managing dashboards and
graphs. We liked the Grafana integration, but didnโ€™t really like how Grafana
manages dashboard configurations. We are running Grafana in a Docker
container, so any changes should be kept out of the container. Another problem
was the lack of change tracking in Grafana.

We have thus decided to write a generator which takes YAML maintained within
git and generates JSON configs for Grafana dashboards. It is furthermore able to
deploy dashboards to Grafana started in a fresh container without the need for
persisting changes made into the container. This provides you with automation,
repeatability, and auditing.

We are pleased to announce that this tool is also now available under an Apache
2.0 license on GitHub.

What improvements have you seen since switching?

An improvement which we saw immediately was the stability of Prometheus. We
were fighting with stability and scalability of Graphite prior to this, so
getting that sorted was a great win for us. Furthermore the speed and stability
of Prometheus made access to metrics very easy for developers. Prometheus is
really helping us to embrace the DevOps culture.

Tomas Cerevka, one of our backend developers, was testing a new version of the
service using JRuby. He needed a quick peek into the heap consumption of that
particular service. He was able to get that information in a snap. For us,
this speed is essential.

What do you think the future holds for ShowMax and Prometheus?

Prometheus has become an integral part of monitoring in ShowMax and it is going
to be with us for the foreseeable future. We have replaced our whole metric
storage with Prometheus, but the ingestion chain remains push based. We are
thus thinking about following Prometheus best practices and switching to a pull
model.

Weโ€™ve also already played with alerts. We want to spend more time on this topic
and come up with increasingly sophisticated alert rules.

Explore Related Technologies & Topics

View All 9,111+ Repositories โ†’

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

โšก MCP Servers Model Context Protocol โ†’ ๐Ÿง  Agent Skills Prompt & rule files โ†’ ๐Ÿ”Œ OpenAPI Specs REST & Swagger schemas โ†’