1. Project Overview & Quickstart (grafana/loki)
File: README.md
Loki: like Prometheus, but for logs.
Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus.
It is designed to be very cost effective and easy to operate.
It does not index the contents of the logs, but rather a set of labels for each log stream.
Compared to other log aggregation systems, Loki:
- does not do full text indexing on logs. By storing compressed, unstructured logs and only indexing metadata, Loki is simpler to operate and cheaper to run.
- indexes and groups log streams using the same labels youβre already using with Prometheus, enabling you to seamlessly switch between metrics and logs using the same labels that youβre already using with Prometheus.
- is an especially good fit for storing Kubernetes Pod logs. Metadata such as Pod labels is automatically scraped and indexed.
- has native support in Grafana (needs Grafana v6.0).
A Loki-based logging stack consists of 3 components:
- Alloy is agent, responsible for gathering logs and sending them to Loki.
- Loki is the main service, responsible for storing logs and processing queries.
- Grafana for querying and displaying the logs.
Note that Alloy replaced Promtail in the stack, because Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy.
Loki is like Prometheus, but for logs: we prefer a multidimensional label-based approach to indexing, and want a single-binary, easy to operate system with no dependencies.
Loki differs from Prometheus by focusing on logs instead of metrics, and delivering logs via push, instead of pull.
Getting started
β οΈ Helm Chart Migration
Effective March 16, 2026, the Grafana Loki Helm chart will be forked to a new repository grafana-community/helm-charts. The chart in the Loki repository will continue to be maintained for GEL users only. See #20705 for details.
Upgrading
Documentation
- Latest release
- Upcoming release, at the tip of the main branch
Commonly used sections:
- API documentation for getting logs into Loki.
- Labels
- Operations
- Docker Driver Client is a Docker plugin to send logs directly to Loki from Docker containers.
- LogCLI provides a command-line interface for querying logs.
- Loki Canary monitors your Loki installation for missing logs.
- Troubleshooting presents help dealing with error messages.
- Loki in Grafana describes how to set up a Loki datasource in Grafana.
Getting Help
If you have any questions or feedback regarding Loki:
- Search existing thread in the Grafana Labs community forum for Loki: https://community.grafana.com
- Ask a question on the Loki Slack channel. To invite yourself to the Grafana Slack, visit https://slack.grafana.com/ and join the #loki channel.
- File an issue for bugs, issues and feature suggestions.
- Send an email to [email protected], or use the web interface.
- UI issues should be filed directly in Grafana.
Your feedback is always welcome.
Further Reading
- The original design doc for Loki is a good source for discussion of the motivation and design decisions.
- Callum Styan's March 2019 DevOpsDays Vancouver talk "Grafana Loki: Log Aggregation for Incident Investigations".
- Grafana Labs blog post "How We Designed Loki to Work Easily Both as Microservices and as Monoliths".
- Tom Wilkie's early-2019 CNCF Paris/FOSDEM talk "Grafana Loki: like Prometheus, but for logs" (slides, video).
- David Kaltschmidt's KubeCon 2018 talk "On the OSS Path to Full Observability with Grafana" (slides, video) on how Loki fits into a cloud-native environment.
- Goutham Veeramachaneni's blog post "Loki: Prometheus-inspired, open source logging for cloud natives" on details of the Loki architecture.
- David Kaltschmidt's blog post "Closer look at Grafana's user interface for Loki" on the ideas that went into the logging user interface.
Contributing
Refer to CONTRIBUTING.md
Building from source
Loki can be run in a single host, no-dependencies mode using the following commands.
You need an up-to-date version of Go, we recommend using the version found in our Makefile
# Checkout source code
$ git clone https://github.com/grafana/loki
$ cd loki
# Build binary
$ go build ./cmd/loki
# Run executable
$ ./loki -config.file=./cmd/loki/loki-local-config.yamlAlternatively, on Unix systems you can use make to build the binary, which adds additional arguments to the go build command.
# Build binary
$ make loki
# Run executable
$ ./cmd/loki/loki -config.file=./cmd/loki/loki-local-config.yamlTo run multiple Loki tenants locally, ensure that auth_enabled is set to true and provide a runtime config with any tenant specific overrides.
# Build binary
$ make loki
# Run executable
./loki -config.file=./cmd/loki/loki-local-multi-tenant-config.yaml -runtime-config.file=./cmd/loki/loki-overrides.yamlAdopters
Please see ADOPTERS.md for some of the organizations using Loki today.
If you would like to add your organization to the list, please open a PR to add it to the list.
License
Grafana Loki is distributed under AGPL-3.0-only. For Apache-2.0 exceptions, see LICENSING.md.
File: docs/README.md
Loki Documentation
This directory contains the source code for the Loki documentation.
Some key things to know about the Loki documentation source:
- The docs are written in markdown, specifically the CommonMark flavor of markdown.
- The Grafana docs team uses Hugo to generate the documentation.
- While you can view the documentation in GitHub, GitHub does not render the images or links correctly and cannot render the Hugo specific shortcodes. To read the Loki documentation, see the Documentation Site.
- If you have a trivial fix or improvement, go ahead and create a pull request.
- If you plan to do something more involved, for example creating a new topic, discuss your ideas on the relevant GitHub issue.
- Pull requests are merged to main, and published to Upcoming release. If your change needs to be published to the Latest release before the next Loki release (that is, it needs to be published immediately), add the appropriate backport label to your PR.
Contributing
We're glad you're here to help make the Loki documentation even better for the Loki community.
Issues and contributions are always welcome! Don't feel shy about contributing. All input is welcome. No fix is too small.
If the documentation confuses you or you think something is missing in the docs, create an issue.
If you find something that you think you can fix, please go ahead and contribute a pull request (PR). You don't need to ask permission.
The Loki documentation is written using the CommonMark flavor of markdown, including some extended features. For more information about markdown, you can see the CommonMark specification, and a quick reference guide for CommonMark.
If you have a GitHub account and you're just making a small fix, for example fixing a typo or updating an example, you can edit the topic in GitHub.
- Find the topic in the Loki repo.
- Click the pencil icon.
- Enter your changes.
- Click Commit changes. GitHub creates a pull request for you.
- The Loki team uses Conventional Commits for commit messages. Make sure your commit messages for doc updates start with
doc:. - If this is your first contribution to the Loki repository, you will need to sign the Contributor License Agreement (CLA) before your PR can be accepted.
NOTE: A member of the Loki repo maintainers must approve and run the continuous integration (CI) workflows for community contributions. - Add the
type/docslabel to identify your PR as a docs contribution. This helps the documentation team track our work. - If your contribution needs to be added to the current release or previous releases, apply the appropriate
backportlabel. You can find more information about backporting in the Writers' toolkit.
For larger contributions, for example documenting a new feature or adding a new topic, consider running the project locally to see how the changes look like before making a pull request.
The docs team has created a Writers' Toolkit that documents how we write documentation at Grafana Labs. Writers' Toolkit contains information about how we structure documentation at Grafana, including templates for different types of topics, information about Hugo shortcodes that extend markdown to add additional features, and information about linters and other tools that we use to write documentation. Writers' Toolkit also includes our Style Guide.
Note that in Hugo the structure of the documentation is based on the folder structure of the documentation repository. The URL structure is generated based on the folder structure and file names. Try to avoid moving or renaming files, as this will break cross-references to those files. If you must move or rename files, run make docs as described below to find and fix broken links before you submit your pull request.
Shared content
NOTE: As of Loki/GEL 3.0, there are shared files between the Loki docs and the GEL docs. The Grafana Enterprise Logs documentation will pull in content from the Loki repo when publishing the GEL docs. Files that are shared between the two doc sets will contain a comment indicating that the content is shared.
For more information about shared content, see the reuse content section of the Writers' Toolkit.
For more information about building and testing documentation, see the build and review section of the Writers' Toolkit.
Lambda-Promtail documentation
As of June 2025, the code for the Lambda-promtail client has moved from the Loki repository to a separate lambda-promtail repository.
As of October 2025, the documentation for the Lambda-promtail client has also moved to the lambda-promtail repository. You can find it under docs/sources.
Testing documentation
Loki uses the static site generator Hugo to generate the documentation. The Loki repository uses a continuous integration (CI) action to sync documentation to the Grafana website. The CI is triggered on every merge to main in the docs subfolder.
You can preview the documentation in GitHub, but GitHub does not render images or any of the Hugo shortcodes. However, you can preview the documentation locally after installing Docker or Podman.
To get a local preview of the documentation:
- Navigate to the directory with the documentation makefile,
/loki/docs. - Run the command
make docs. This uses thegrafana/docsimage which internally uses Hugo to generate the static site. - Open http://localhost:3002/docs/loki/latest/ to review your changes.
Note that
make docsuses a lot of memory. If it crashes, increase the memory allocated to Docker and try again.
For more information about building and testing documentation, see build and review section of the Writers' Toolkit
2. Official Technical Reference & Guides (grafana/docs)
Prometheus Monitoring Mixins
NOTE: This project is beta stage.
A mixin is a set of Grafana dashboards and Prometheus rules and alerts, packaged together in a reuseable and extensible bundle.
Mixins are written in jsonnet, and are typically installed and updated with jsonnet-bundler.
For more information about mixins, see:
- Prometheus Monitoring Mixins Design Doc. A cached pdf is included in this repo.
- For more motivation, see
"The RED Method: How to instrument your services" talk from CloudNativeCon Austin 2018. The KLUMPs system demo'd became the basis for the kubernetes-mixin. - "Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts and Exporters" talk from CloudNativeCon Copenhagen 2018.
- "Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts and Exporters" talk from PromCon 2018 (slightly updated).
How to use mixins.
Mixins are designed to be vendored into the repo with your infrastructure config.
To do this, use jsonnet-bundler:
You then have three options for deploying your dashboards
- Generate the config files and deploy them yourself.
- Use ksonnet to deploy this mixin along with Prometheus and Grafana.
- Use kube-prometheus to deploy this mixin.
Generate config files
You can manually generate the alerts, dashboards and rules files, but first you
must install some tools:
$ go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest
# macOS
$ brew install jsonnet-bundler
# Archlinux AUR
$ yay -S jsonnetThen, grab the mixin and its dependencies:
$ git clone https://github.com/<mixin org>/<mixin repo>
$ cd <mixin repo>
$ jb installFinally, build the mixin:
$ make prometheus_alerts.yaml
$ make prometheus_rules.yaml
$ make dashboards_outThe prometheus_alerts.yaml and prometheus_rules.yaml file then need to passed
to your Prometheus server, and the files in dashboards_out need to be imported
into you Grafana server. The exact details will depending on how you deploy your
monitoring stack to Kubernetes.
Using with prometheus-ksonnet
Alternatively you can also use the mixin with
prometheus-ksonnet,
a ksonnet module to deploy a fully-fledged
Prometheus-based monitoring system for Kubernetes:
Make sure you have the ksonnet v0.8.0:
$ brew install https://raw.githubusercontent.com/ksonnet/homebrew-tap/82ef24cb7b454d1857db40e38671426c18cd8820/ks.rb
$ brew pin ks
$ ks version
ksonnet version: v0.8.0
jsonnet version: v0.9.5
client-go version: v1.6.8-beta.0+$Format:%h$In your config repo, if you don't have a ksonnet application, make a new one (will copy credentials from current context):
$ ks init <application name>
$ cd <application name>
$ ks env add defaultGrab the kubernetes-jsonnet module using and its dependencies, which include
the kubernetes-mixin:
$ go get github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb
$ jb init
$ jb install github.com/kausalco/public/prometheus-ksonnetAssuming you want to run in the default namespace ('environment' in ksonnet parlance), add the follow to the file environments/default/main.jsonnet:
local prometheus = import "prometheus-ksonnet/prometheus-ksonnet.libsonnet";
prometheus {
_config+:: {
namespace: "default",
},
}Apply your config:
$ ks apply defaultUsing kube-prometheus
See the kube-prometheus docs for instructions on how to use mixins with kube-prometheus.
Customising the mixin
Mixins typically allows you to override the selectors used for various jobs,
to match those used in your Prometheus set.
This example uses the kubernetes-mixin.
In a new directory, add a file mixin.libsonnet:
local kubernetes = import "kubernetes-mixin/mixin.libsonnet";
kubernetes {
_config+:: {
kubeStateMetricsSelector: 'job="kube-state-metrics"',
cadvisorSelector: 'job="kubernetes-cadvisor"',
nodeExporterSelector: 'job="kubernetes-node-exporter"',
kubeletSelector: 'job="kubernetes-kubelet"',
},
}Then, install the kubernetes-mixin:
$ jb init
$ jb install github.com/kubernetes-monitoring/kubernetes-mixinGenerate the alerts, rules and dashboards:
$ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusAlerts)' > alerts.yml
$ jsonnet -J vendor -S -e 'std.manifestYamlDoc((import "mixin.libsonnet").prometheusRules)' >files/rules.yml
$ jsonnet -J vendor -m files/dashboards -e '(import "mixin.libsonnet").grafanaDashboards'Guidelines for alert names, labels, and annotations
Prometheus alerts deliberately allow users to define their own schema for
names, labels, and annotations. The following is a style guide recommended for
alerts in monitoring mixins. Following this guide helps creating useful
notification templates for all mixins and customizing mixin alerts in a unified
fashion.
The alert name is a terse description of the alerting condition, using
camel case, without whitespace, starting with a capital letter. The first
component of the name should be shared between all alerts of a mixin (or
between a group of related alerts within a larger mixin). Examples:NodeFilesystemAlmostOutOfFiles (from the node-exporter
mixin,PrometheusNotificationQueueRunningFull (from the Prometheus
mixin).
To mark the severity of an alert, use a label called severity with one of
the following label values:
criticalfor alerts that require immediate action. For a production system,
those alerts will usually hit a pager.warningfor alerts that require action eventually but not urgently enough
to wake someone up or require them to immediately interrupt what they are
working on. A typical routing target for those alerts is some kind of ticket
queueing or bug tracking system.infofor alerts that do not require any action by itself but mark something
as βout of the ordinaryβ. Those alerts aren't usually routed anywhere, but
can be inspected during troubleshooting.
An alert can have the following annotations:
summary(mandatory): Essentially a more comprehensive and readable version
of the alert name. Use a human-readable sentence, starting with a capital
letter and ending with a period. Use a static string or, if dynamic expansion
is needed, aim for expanding into the same string for alerts that are
typically grouped together into one notification. In that way, it can be used
as a common βheadlineβ for all alerts in the notification template. Examples:Filesystem has less than 3% inodes left.(for theNodeFilesystemAlmostOutOfFilesalert mentioned above),Prometheus alert notification queue predicted to run full in less than 30m.(for thePrometheusNotificationQueueRunningFullalert mentioned above).description(mandatory): A detailed description of a single alert, with
most of the important information templated in. The description usually
expands into a different string for every individual alert within a
notification. A notification template can iterate through all the
descriptions and format them into a list. Examples (again corresponding to
the examples above):Filesystem on {{ $labels.device }} at {{ $labels.instance }} has only {{ printf "%.2f" $value }}% available inodes left.,Alert notification queue of Prometheus %(prometheusName)s is running full..
Note that we plan to add recommended optional annotations for a runbook link
(presumably called runbook_url) and a dashboard link
(dashboard_url). However, we still need to work out how to configure patterns
for those URLs across mixins in a useful way.