badger Documentation

Go database & document-database repository by @dgraph-io. AI agent rules & system prompts with 16k stars.

15,742 stars Go Go 1 file · ~6,561 tokens #database#document-database#go#golang#key-value#library#ssd
RAW Doc

File: README.md

BadgerDB






BadgerDB is an embeddable, persistent and fast key-value (KV) database written in pure Go. It is the
underlying database for Dgraph, a fast, distributed graph
database. It's meant to be a performant alternative to non-Go-based key-value stores like RocksDB.

Project Status

Badger is stable and is being used to serve data sets worth hundreds of terabytes. Badger supports
concurrent ACID transactions with serializable snapshot isolation (SSI) guarantees. A Jepsen-style
bank test runs nightly for 8h, with --race flag and ensures the maintenance of transactional
guarantees. Badger has also been tested to work with filesystem level anomalies, to ensure
persistence and consistency. Badger is being used by a number of projects which includes Dgraph,
Jaeger Tracing, UsenetExpress, and many more.

The list of projects using Badger can be found here.

Please consult the Changelog for more detailed information on releases.

Note: Badger is built with go 1.23 and we refrain from bumping this version to minimize downstream
effects of those using Badger in applications built with older versions of Go.

Table of Contents

Getting Started

Installing

To start using Badger, install Go 1.23 or above. Badger v3 and above needs go modules. From your
project, run the following command

sh
go get github.com/dgraph-io/badger/v4

This will retrieve the library.

Installing Badger Command Line Tool

Badger provides a CLI tool which can perform certain operations like offline backup/restore. To
install the Badger CLI, retrieve the repository and checkout the desired version. Then run

sh
cd badger
go install .

This will install the badger command line utility into your $GOBIN path.

Badger Documentation

Badger Documentation is available at https://badger.dgraph.io

Resources

Blog Posts

  1. Introducing Badger: A fast key-value store written natively in Go
  2. Make Badger crash resilient with ALICE
  3. Badger vs LMDB vs BoltDB: Benchmarking key-value databases in Go
  4. Concurrent ACID Transactions in Badger

Design

Badger was written with these design goals in mind:

  • Write a key-value database in pure Go.
  • Use latest research to build the fastest KV database for data sets spanning terabytes.
  • Optimize for SSDs.

Badger’s design is based on a paper titled WiscKey: Separating Keys from Values in SSD-conscious
Storage.

Comparisons

Feature Badger RocksDB BoltDB
Design LSM tree with value log LSM tree only B+ tree
High Read throughput Yes No Yes
High Write throughput Yes Yes No
Designed for SSDs Yes (with latest research 1) Not specifically 2 No
Embeddable Yes Yes Yes
Sorted KV access Yes Yes Yes
Pure Go (no Cgo) Yes No Yes
Transactions Yes, ACID, concurrent with SSI3 Yes (but non-ACID) Yes, ACID
Snapshots Yes Yes Yes
TTL support Yes Yes No
3D access (key-value-version) Yes4 No No

1 The WISCKEY paper (on which Badger is based) saw big wins with separating
values from keys, significantly reducing the write amplification compared to a typical LSM tree.

2 RocksDB is an SSD optimized version of LevelDB, which was designed specifically for
rotating disks. As such RocksDB's design isn't aimed at SSDs.

3 SSI: Serializable Snapshot Isolation. For more details, see the blog post
Concurrent ACID Transactions in Badger

4 Badger provides direct access to value versions via its Iterator API. Users can also
specify how many versions to keep per key via Options.

Benchmarks

We have run comprehensive benchmarks against RocksDB, Bolt and LMDB. The benchmarking code, and the
detailed logs for the benchmarks can be found in the badger-bench repo. More explanation,
including graphs can be found the blog posts (linked above).

Projects Using Badger

Below is a list of known projects that use Badger:

  • Dgraph - Distributed graph database.
  • Jaeger - Distributed tracing platform.
  • go-ipfs - Go client for the InterPlanetary File System (IPFS),
    a new hypermedia distribution protocol.
  • Riot - An open-source, distributed search engine.
  • emitter - Scalable, low latency, distributed pub/sub
    broker with message storage, uses MQTT, gossip and badger.
  • OctoSQL - Query tool that allows you to join, analyse and
    transform data from multiple databases using SQL.
  • Dkron - Distributed, fault tolerant job scheduling system.
  • smallstep/certificates - Step-ca is an online
    certificate authority for secure, automated certificate management.
  • Sandglass - distributed, horizontally scalable,
    persistent, time sorted message queue.
  • TalariaDB - Grab's Distributed, low latency time-series
    database.
  • Sloop - Salesforce's Kubernetes History Visualization
    Project.
  • Usenet Express - Serving over 300TB of data with Badger.
  • gorush - A push notification server written in Go.
  • 0-stor - Single device object store.
  • Dispatch Protocol - Blockchain protocol for distributed
    application data analytics.
  • GarageMQ - AMQP server written in Go.
  • RedixDB - A real-time persistent key-value store with the
    same redis protocol.
  • BBVA - Raft backend implementation using BadgerDB for
    Hashicorp raft.
  • Fantom - aBFT Consensus platform for
    distributed applications.
  • decred - An open, progressive, and self-funding
    cryptocurrency with a system of community-based governance integrated into its blockchain.
  • OpenNetSys - Create useful dApps in any software language.
  • HoneyTrap - An extensible and opensource system for
    running, monitoring and managing honeypots.
  • Insolar - Enterprise-ready blockchain platform.
  • IoTeX - The next generation of the decentralized
    network for IoT powered by scalability- and privacy-centric blockchains.
  • go-sessions - The sessions manager for Go net/http and
    fasthttp.
  • Babble - BFT Consensus platform for distributed
    applications.
  • Tormenta - Embedded object-persistence layer / simple JSON
    database for Go projects.
  • BadgerHold - An embeddable NoSQL store for querying Go
    types built on Badger
  • Goblero - Pure Go embedded persistent job queue backed by
    BadgerDB
  • Surfline - Serving global wave and weather forecast data with Badger.
  • Cete - Simple and highly available distributed key-value store
    built on Badger. Makes it easy bringing up a cluster of Badger with Raft consensus algorithm by
    hashicorp/raft.
  • Volument - A new take on website analytics backed by Badger.
  • KVdb - Hosted key-value store and serverless platform built on top of Badger.
  • Terminotes - Self hosted notes storage and search
    server - storage powered by BadgerDB
  • Pyroscope - Open source continuous profiling platform
    built with BadgerDB
  • Veri - A distributed feature store optimized for Search and
    Recommendation tasks.
  • bIter - A library and Iterator interface for working with
    the badger.Iterator, simplifying from-to, and prefix mechanics.
  • ld - (Lean Database) A very simple gRPC-only key-value
    database, exposing BadgerDB with key-range scanning semantics.
  • Souin - A RFC compliant HTTP cache with lot of other features
    based on Badger for the storage. Compatible with all existing reverse-proxies.
  • Xuperchain - A highly flexible blockchain architecture
    with great transaction performance.
  • m2 - A simple http key/value store based on the raft protocol.
  • chaindb - A blockchain storage layer used by
    Gossamer, a Go client for the
    Polkadot Network.
  • vxdb - Simple schema-less Key-Value NoSQL database with
    simplest API interface.
  • Opacity - Backend implementation for the Opacity
    storage project
  • Vephar - A minimal key/value store using hashicorp-raft
    for cluster coordination and Badger for data storage.
  • gowarcserver - Open-source server for warc files. Can be
    used in conjunction with pywb
  • flow-go - A fast, secure, and developer-friendly blockchain
    built to support the next generation of games, apps and the digital assets that power them.
  • Wrgl - A data version control system that works like Git but specialized to
    store and diff CSV.
  • Loggie - A lightweight, cloud-native data transfer agent
    and aggregator.
  • raft-badger - raft-badger implements LogStore and
    StableStore Interface of hashcorp/raft. it is used to store raft log and metadata of
    hashcorp/raft.
  • DVID - A dataservice for branched versioning of a variety
    of data types. Originally created for large-scale brain reconstructions in Connectomics.
  • KVS - A library for making it easy to persist, load and query
    full structs into BadgerDB, using an ownership hierarchy model.
  • LLS - LLS is an efficient URL Shortener that can be used to
    shorten links and track link usage. Support for BadgerDB and MongoDB. Improved performance by more
    than 30% when using BadgerDB
  • lakeFS - lakeFS is an open-source data version control that
    transforms your object storage to Git-like repositories. lakeFS uses BadgerDB for its underlying
    local metadata KV store implementation
  • Goptivum - Goptivum is a better frontend and API for the
    Vulcan Optivum schedule program
  • ActionManager - A dynamic entity manager based on rjsf schema
    and badger db
  • MightyMap - Mightymap: Conveys both robustness
    and high capability, fitting for a powerful concurrent map.
  • FlowG - A low-code log processing facility
  • Bluefin - Bluefin is a TUNA Proof of Work miner for the
    Fortuna smart contract on the Cardano blockchain
  • cDNSd - A Cardano blockchain backed DNS server daemon
  • Dingo - A Cardano blockchain data node

If you are using Badger in a project please send a pull request to add it to the list.

Platform Compatibility

Badger uses OS-specific implementations for directory locking and fsync operations. On
POSIX-compliant systems (Linux, macOS, BSD), these work as expected.

For non-POSIX platforms, be aware of potential limitations:

Platform File Notes
AIX dir_aix.go Directory fsync not supported; durability on crash may be affected
Windows dir_windows.go Uses different locking mechanism
Plan9 dir_plan9.go No file locking support
WASM/JS dir_other.go No file locking support

If you encounter issues on these platforms, review the corresponding dir_*.go source file for
implementation details.

Contributing

If you're interested in contributing to Badger see CONTRIBUTING.

Contact

File: skl/README.md

This is much better than skiplist and slist.

sh
BenchmarkReadWrite/frac_0-8            3000000         537 ns/op
BenchmarkReadWrite/frac_1-8            3000000         503 ns/op
BenchmarkReadWrite/frac_2-8            3000000         492 ns/op
BenchmarkReadWrite/frac_3-8            3000000         475 ns/op
BenchmarkReadWrite/frac_4-8            3000000         440 ns/op
BenchmarkReadWrite/frac_5-8            5000000         442 ns/op
BenchmarkReadWrite/frac_6-8            5000000         380 ns/op
BenchmarkReadWrite/frac_7-8            5000000         338 ns/op
BenchmarkReadWrite/frac_8-8            5000000         294 ns/op
BenchmarkReadWrite/frac_9-8            10000000        268 ns/op
BenchmarkReadWrite/frac_10-8           100000000       26.3 ns/op

And even better than a simple map with read-write lock:

sh
BenchmarkReadWriteMap/frac_0-8         2000000         774 ns/op
BenchmarkReadWriteMap/frac_1-8         2000000         647 ns/op
BenchmarkReadWriteMap/frac_2-8         3000000         605 ns/op
BenchmarkReadWriteMap/frac_3-8         3000000         603 ns/op
BenchmarkReadWriteMap/frac_4-8         3000000         556 ns/op
BenchmarkReadWriteMap/frac_5-8         3000000         472 ns/op
BenchmarkReadWriteMap/frac_6-8         3000000         476 ns/op
BenchmarkReadWriteMap/frac_7-8         3000000         457 ns/op
BenchmarkReadWriteMap/frac_8-8         5000000         444 ns/op
BenchmarkReadWriteMap/frac_9-8         5000000         361 ns/op
BenchmarkReadWriteMap/frac_10-8        10000000        212 ns/op

Node Pooling

Command used

sh
rm -Rf tmp && /usr/bin/time -l ./populate -keys_mil 10

For pprof results, we run without using /usr/bin/time. There are four runs below.

Results seem to vary quite a bit between runs.

Before node pooling

sh
1311.53MB of 1338.69MB total (97.97%)
Dropped 30 nodes (cum <= 6.69MB)
Showing top 10 nodes out of 37 (cum >= 12.50MB)
      flat  flat%   sum%        cum   cum%
  523.04MB 39.07% 39.07%   523.04MB 39.07%  github.com/dgraph-io/badger/skl.(*Skiplist).Put
  184.51MB 13.78% 52.85%   184.51MB 13.78%  runtime.stringtoslicebyte
  166.01MB 12.40% 65.25%   689.04MB 51.47%  github.com/dgraph-io/badger/mem.(*Table).Put
     165MB 12.33% 77.58%      165MB 12.33%  runtime.convT2E
  116.92MB  8.73% 86.31%   116.92MB  8.73%  bytes.makeSlice
   62.50MB  4.67% 90.98%    62.50MB  4.67%  main.newValue
   34.50MB  2.58% 93.56%    34.50MB  2.58%  github.com/dgraph-io/badger/table.(*BlockIterator).parseKV
   25.50MB  1.90% 95.46%   100.06MB  7.47%  github.com/dgraph-io/badger/y.(*MergeIterator).Next
   21.06MB  1.57% 97.04%    21.06MB  1.57%  github.com/dgraph-io/badger/table.(*Table).read
   12.50MB  0.93% 97.97%    12.50MB  0.93%  github.com/dgraph-io/badger/table.header.Encode

      128.31 real       329.37 user        17.11 sys
3355660288  maximum resident set size
         0  average shared memory size
         0  average unshared data size
         0  average unshared stack size
   2203080  page reclaims
       764  page faults
         0  swaps
       275  block input operations
        76  block output operations
         0  messages sent
         0  messages received
         0  signals received
     49173  voluntary context switches
    599922  involuntary context switches

After node pooling

sh
1963.13MB of 2026.09MB total (96.89%)
Dropped 29 nodes (cum <= 10.13MB)
Showing top 10 nodes out of 41 (cum >= 185.62MB)
      flat  flat%   sum%        cum   cum%
  658.05MB 32.48% 32.48%   658.05MB 32.48%  github.com/dgraph-io/badger/skl.glob..func1
  297.51MB 14.68% 47.16%   297.51MB 14.68%  runtime.convT2E
  257.51MB 12.71% 59.87%   257.51MB 12.71%  runtime.stringtoslicebyte
  249.01MB 12.29% 72.16%  1007.06MB 49.70%  github.com/dgraph-io/badger/mem.(*Table).Put
  142.43MB  7.03% 79.19%   142.43MB  7.03%  bytes.makeSlice
     100MB  4.94% 84.13%   758.05MB 37.41%  github.com/dgraph-io/badger/skl.newNode
   99.50MB  4.91% 89.04%    99.50MB  4.91%  main.newValue
      75MB  3.70% 92.74%       75MB  3.70%  github.com/dgraph-io/badger/table.(*BlockIterator).parseKV
   44.62MB  2.20% 94.94%    44.62MB  2.20%  github.com/dgraph-io/badger/table.(*Table).read
   39.50MB  1.95% 96.89%   185.62MB  9.16%  github.com/dgraph-io/badger/y.(*MergeIterator).Next

      135.58 real       374.29 user        17.65 sys
3740614656  maximum resident set size
         0  average shared memory size
         0  average unshared data size
         0  average unshared stack size
   2276566  page reclaims
       770  page faults
         0  swaps
       128  block input operations
        90  block output operations
         0  messages sent
         0  messages received
         0  signals received
     46434  voluntary context switches
    597049  involuntary context switches

File: table/README.md

Size of table is 123,217,667 bytes for all benchmarks.

BenchmarkRead

sh
$ go test -bench ^BenchmarkRead$ -run ^$ -count 3
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/badger/table
BenchmarkRead-16              10       154074944 ns/op
BenchmarkRead-16              10       154340411 ns/op
BenchmarkRead-16              10       151914489 ns/op
PASS
ok       github.com/dgraph-io/badger/table       22.467s

Size of table is 123,217,667 bytes, which is ~118MB.

The rate is ~762MB/s using LoadToRAM (when table is in RAM).

To read a 64MB table, this would take ~0.084s, which is negligible.

BenchmarkReadAndBuild

sh
$ go test -bench BenchmarkReadAndBuild -run ^$ -count 3
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/badger/table
BenchmarkReadAndBuild-16              1       1026755231 ns/op
BenchmarkReadAndBuild-16              1       1009543316 ns/op
BenchmarkReadAndBuild-16              1       1039920546 ns/op
PASS
ok       github.com/dgraph-io/badger/table       12.081s

The rate is ~123MB/s. To build a 64MB table, this would take ~0.56s. Note that this does NOT include
the flushing of the table to disk. All we are doing above is reading one table (which is in RAM) and
write one table in memory.

The table building takes 0.56-0.084s ~ 0.4823s.

BenchmarkReadMerged

Below, we merge 5 tables. The total size remains unchanged at ~122M.

sh
$ go test -bench ReadMerged -run ^$ -count 3
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/badger/table
BenchmarkReadMerged-16              2       977588975 ns/op
BenchmarkReadMerged-16              2       982140738 ns/op
BenchmarkReadMerged-16              2       962046017 ns/op
PASS
ok       github.com/dgraph-io/badger/table       27.433s

The rate is ~120MB/s. To read a 64MB table using merge iterator, this would take ~0.53s.

BenchmarkRandomRead

sh
go test -bench BenchmarkRandomRead$ -run ^$ -count 3
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/badger/table
BenchmarkRandomRead-16              500000       2645 ns/op
BenchmarkRandomRead-16              500000       2648 ns/op
BenchmarkRandomRead-16              500000       2614 ns/op
PASS
ok       github.com/dgraph-io/badger/table       50.850s

For random read benchmarking, we are randomly reading a key and verifying its value.

DB Open benchmark

  1. Create badger DB with 2 billion key-value pairs (about 380GB of data)

    sh
    badger fill -m 2000 --dir="/tmp/data" --sorted

  2. Clear buffers and swap memory

    sh
    free -mh && sync && echo 3 | sudo tee /proc/sys/vm/drop_caches && sudo swapoff -a && sudo swapon -a && free -mh

    Also flush disk buffers

    sh
    blockdev --flushbufs /dev/nvme0n1p4

  3. Run the benchmark

    sh
    go test -run=^$ github.com/dgraph-io/badger -bench ^BenchmarkDBOpen$ -benchdir="/tmp/data" -v

badger 2019/06/04 17:15:56 INFO: 126 tables out of 1028 opened in 3.017s
badger 2019/06/04 17:15:59 INFO: 257 tables out of 1028 opened in 6.014s
badger 2019/06/04 17:16:02 INFO: 387 tables out of 1028 opened in 9.017s
badger 2019/06/04 17:16:05 INFO: 516 tables out of 1028 opened in 12.025s
badger 2019/06/04 17:16:08 INFO: 645 tables out of 1028 opened in 15.013s
badger 2019/06/04 17:16:11 INFO: 775 tables out of 1028 opened in 18.008s
badger 2019/06/04 17:16:14 INFO: 906 tables out of 1028 opened in 21.003s
badger 2019/06/04 17:16:17 INFO: All 1028 tables opened in 23.851s
badger 2019/06/04 17:16:17 INFO: Replaying file id: 1998 at offset: 332000
badger 2019/06/04 17:16:17 INFO: Replay took: 9.81µs
goos: linux
goarch: amd64
pkg: github.com/dgraph-io/badger
BenchmarkDBOpen-16              1       23930082140 ns/op
PASS
ok       github.com/dgraph-io/badger       24.076s

    It takes about 23.851s to open a DB with 2 billion sorted key-value entries.

File: docs/index.md

Badger

What is Badger?

BadgerDB is an embeddable, persistent, and fast key-value (KV) database written in pure Go. It is
the underlying database for Dgraph, a fast, distributed graph
database. It is meant to be an efficient alternative to non-Go-based key-value stores like RocksDB.

Changelog

We keep the repo Changelog up to date
with each release.

Quickstart

Design

Encryption at rest

Troubleshooting

File: docs/README.md

Badger Docs

If you are looking for Badger's documentation, you might find https://dgraph.io/docs/badger much more readable.

Getting Started

We use Hugo for our documentation.

Running locally

  1. Download and install the latest patch of Hugo version v0.69.x from here.
  2. Run hugo server within the docs folder.
  3. Visit http://localhost:1313 to see the documentation site.

Contributing

If you're interested in contributing to Badger, please review our guidelines.

Contact

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#database Browse database catalog → #document-database Browse document-database catalog → #go Browse go catalog → #golang Browse golang catalog → #key-value Browse key-value catalog → #library Browse library catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →