rocksdb Documentation

File: README.md Copy Chapter

RocksDB: A Persistent Key-Value Store for Flash and RAM Storage Copy Chapter

RocksDB is developed and maintained by Facebook Database Engineering Team.
It is built on earlier work on LevelDB by Sanjay Ghemawat ([email protected])
and Jeff Dean ([email protected])

This code is a library that forms the core building block for a fast
key-value server, especially suited for storing data on flash drives.
It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs
between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF)
and Space-Amplification-Factor (SAF). It has multi-threaded compactions,
making it especially suitable for storing multiple terabytes of data in a
single database.

Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples

See the github wiki for more explanation.

The public interface is in include/. Callers should not include or
rely on the details of any other header files in this package. Those
internal APIs may be changed without warning.

Questions and discussions are welcome on the RocksDB Developers Public Facebook group and email list on Google Groups.

License Copy Chapter

RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses.

File: docs/_docs/faq.md Copy Chapter

docid: support-faq title: FAQ layout: docs permalink: /docs/support/faq.html Copy Chapter

Here is an ever-growing list of frequently asked questions around RocksDB

What is RocksDB? Copy Chapter

RocksDB is an embeddable persistent key-value store for fast storage. RocksDB can also be the foundation for a client-server database but our current focus is on embedded workloads.

RocksDB builds on LevelDB to be scalable to run on servers with many CPU cores, to efficiently use fast storage, to support IO-bound, in-memory and write-once workloads, and to be flexible to allow for innovation.

For the latest details, watch Mark Callaghan’s and Igor Canadi’s talk at CMU on 10/2015. Dhruba Borthakur’s introductory talk from the Data @ Scale 2013 conference provides some perspective about how RocksDB has evolved.

How does performance compare? Copy Chapter

We benchmarked LevelDB and found that it was unsuitable for our server workloads. The benchmark results look awesome at first sight, but we quickly realized that those results were for a database whose size was smaller than the size of RAM on the test machine – where the entire database could fit in the OS page cache. When we performed the same benchmarks on a database that was at least 5 times larger than main memory, the performance results were dismal.

By contrast, we’ve published the RocksDB benchmark results for server side workloads on Flash. We also measured the performance of LevelDB on these server-workload benchmarks and found that RocksDB solidly outperforms LevelDB for these IO bound workloads. We found that LevelDB’s single-threaded compaction process was insufficient to drive server workloads. We saw frequent write-stalls with LevelDB that caused 99-percentile latency to be tremendously large. We found that mmap-ing a file into the OS cache introduced performance bottlenecks for reads. We could not make LevelDB consume all the IOs offered by the underlying Flash storage.

What is RocksDB suitable for? Copy Chapter

RocksDB can be used by applications that need low latency database accesses. Possibilities include:

• A user-facing application that stores the viewing history and state of users of a website.
• A spam detection application that needs fast access to big data sets.
• A graph-search query that needs to scan a data set in realtime.
• A cache data from Hadoop, thereby allowing applications to query Hadoop data in realtime.
• A message-queue that supports a high number of inserts and deletes.

How big is RocksDB adoption? Copy Chapter

RocksDB is an embedded storage engine that is used in a number of backend systems at Facebook. In the Facebook newsfeed’s backend, it replaced another internal storage engine called Centrifuge and is one of the many components used. ZippyDB, a distributed key value store service used by Facebook products relies RocksDB. Details on ZippyDB are in Muthu Annamalai’s talk at Data@Scale in Seattle. Dragon, a distributed graph query engine part of the social graph infrastructure, is using RocksDB to store data. Parse has been running MongoDB on RocksDB in production since early 2015.

RocksDB is proving to be a useful component for a lot of other groups in the industry. For a list of projects currently using RocksDB, take a look at our USERS.md list on github.

How good is RocksDB as a database storage engine? Copy Chapter

Our engineering team at Facebook firmly believes that RocksDB has great potential as storage engine for databases. It has been proven in production with MongoDB: MongoRocks is the RocksDB based storage engine for MongoDB.

MyRocks is the RocksDB based storage engine for MySQL. Using RocksDB we have managed to achieve 2x better compression and 10x less write amplification for our benchmarks compared to our existing MySQL setup. Given our current results, work is currently underway to develop MyRocks into a production ready solution for web-scale MySQL workloads. Follow along on GitHub!

Why is RocksDB open sourced? Copy Chapter

We are open sourcing this project on GitHub because we think it will be useful beyond Facebook. We are hoping that software programmers and database developers will use, enhance, and customize RocksDB for their use-cases. We would also like to engage with the academic community on topics related to efficiency for modern database algorithms.

File: docs/_docs/getting-started.md Copy Chapter

docid: getting-started title: Getting started layout: docs permalink: /docs/getting-started.html Copy Chapter

Overview Copy Chapter

The RocksDB library provides a persistent key value store. Keys and values are arbitrary byte arrays. The keys are ordered within the key value store according to a user-specified comparator function.

The library is maintained by the Facebook Database Engineering Team, and is based on LevelDB, by Sanjay Ghemawat and Jeff Dean at Google.

This overview gives some simple examples of how RocksDB is used. For the story of why RocksDB was created in the first place, see Dhruba Borthakur’s introductory talk from the Data @ Scale 2013 conference.

Opening A Database Copy Chapter

A rocksdb database has a name which corresponds to a file system directory. All of the contents of database are stored in this directory. The following example shows how to open a database, creating it if necessary:

#include <assert>
#include "rocksdb/db.h"

rocksdb::DB* db;
rocksdb::Options options;
options.create_if_missing = true;
rocksdb::Status status =
  rocksdb::DB::Open(options, "/tmp/testdb", &db);
assert(status.ok());
...

If you want to raise an error if the database already exists, add the following line before the rocksdb::DB::Open call:

options.error_if_exists = true;

Status Copy Chapter

You may have noticed the rocksdb::Status type above. Values of this type are returned by most functions in RocksDB that may encounter
an error. You can check if such a result is ok, and also print an associated error message:

rocksdb::Status s = ...;
if (!s.ok()) cerr << s.ToString() << endl;

Closing A Database Copy Chapter

When you are done with a database, just delete the database object. For example:

/* open the db as described above */
/* do something with db */
delete db;

Reads And Writes Copy Chapter

The database provides Put, Delete, and Get methods to modify/query the database. For example, the following code moves the value stored under key1 to key2.

std::string value;
rocksdb::Status s = db->Get(rocksdb::ReadOptions(), key1, &value);
if (s.ok()) s = db->Put(rocksdb::WriteOptions(), key2, value);
if (s.ok()) s = db->Delete(rocksdb::WriteOptions(), key1);

Further documentation Copy Chapter

These are just simple examples of how RocksDB is used. The full documentation is currently on the GitHub wiki.

Here are some specific details about the RocksDB implementation:

• RocksDB Overview
• Immutable BlockBased Table file format
• Log file format

File: docs/_top-level/support.md Copy Chapter

layout: top-level title: Support id: support category: support Copy Chapter

Need help? Copy Chapter

Do not hesitate to ask questions if you are having trouble with RocksDB.

GitHub issues Copy Section

Use GitHub issues to report bugs, issues and feature requests for the RocksDB codebase.

Facebook Group Copy Section

Use the RocksDB Facebook group for general questions and discussion about RocksDB.

FAQ Copy Section

Check out a list of commonly asked questions about RocksDB.

File: docs/doc-type-examples/2016-04-07-blog-post-example.md Copy Chapter

title: Blog Post Example layout: post author: exampleauthor category: blog Copy Chapter

Any local blog posts would go in the _posts directory.

This is an example blog post introduction, try to keep it short and about a paragraph long, to encourage people to click through to read the entire post.

Everything below the `` tag will only show on the actual blog post page, not on the /blog/ index.

Author is defined in _data/authors.yml

No posts? Copy Chapter

If you have no blog for your site, you can remove the entire _posts folder. Otherwise add markdown files in here. See CONTRIBUTING.md for details.

File: docs/doc-type-examples/docs-hello-world.md Copy Chapter

docid: hello-world title: Hello, World! layout: docs permalink: /docs/hello-world.html Copy Chapter

Any local docs would go in the _docs directory.

No documentation? Copy Chapter

If you have no documentation for your site, you can remove the entire _docs folder. Otherwise add markdown files in here. See CONTRIBUTING.md for details.

File: docs/doc-type-examples/top-level-example.md Copy Chapter

layout: top-level title: Support Example id: top-level-example category: top-level Copy Chapter

This is a static page disconnected from the blog or docs collections that can be added at a top-level (i.e., the same level as index.md).

File: docs/CONTRIBUTING.md Copy Chapter

This provides guidance on how to contribute various content to rocksdb.org.

Getting started Copy Chapter

You should only have to do these one time.

• Rename this file to CONTRIBUTING.md.
• Rename EXAMPLE-README-FOR-RUNNING-DOCS.md to README.md (replacing the existing README.md that came with the template).
• Rename EXAMPLE-LICENSE to LICENSE.
• Review the template information.
• Review ./_config.yml.
• Make sure you update title, description, tagline and gacode (Google Analytics) in ./_config.yml.

Basic Structure Copy Chapter

Most content is written in markdown. You name the file something.md, then have a header that looks like this:

---
docid: getting-started
title: Getting started with ProjectName
layout: docs
permalink: /docs/getting-started.html
---

Customize these values for each document, blog post, etc.

The filename of the .md file doesn't actually matter; what is important is the docid being unique and the permalink correct and unique too).

Landing page Copy Chapter

Modify index.md with your new or updated content.

If you want a GridBlock as part of your content, you can do so directly with HTML:

<h3>Your Features</h3>
      
        <li>The [Example](http://example.org/)</li>
        <li>[Another Example](http://example.com)</li>
      
    
  

  
    
      <h3>More information</h3>
      
         Stuff here

or with a combination of changing ./_data/features.yml and adding some Liquid to index.md, such as:

{% include content/gridblocks.html data_source=site.data.features imagealign="bottom"%}

Blog Copy Chapter

To modify a blog post, edit the appopriate markdown file in ./_posts/.

Adding a new blog post is a four-step process.

Some posts have a permalink and comments in the blog post YAML header. You will not need these for new blog posts. These are an artifact of migrating the blog from Wordpress to gh-pages.

1. Create your blog post in ./_posts/ in markdown (file extension .md or .markdown). See current posts in that folder or ./doc-type-examples/2016-04-07-blog-post-example.md for an example of the YAML format. If the ./_posts directory does not exist, create it.

• You can add a `` tag in the middle of your post such that you show only the excerpt above that tag in the main /blog index on your page.

1. If you have not authored a blog post before, modify the ./_data/authors.yml file with the author id you used in your blog post, along with your full name and Facebook ID to get your profile picture.
2. Run the site locally to test your changes. It will be at http://127.0.0.1/blog/your-new-blog-post-title.html
3. Push your changes to GitHub.

Docs Copy Chapter

To modify docs, edit the appropriate markdown file in ./_docs/.

To add docs to the site....

1. Add your markdown file to the ./_docs/ folder. See ./doc-type-examples/docs-hello-world.md for an example of the YAML header format. If the ./_docs/ directory does not exist, create it.

• You can use folders in the ./_docs/ directory to organize your content if you want.

1. Update _data/nav_docs.yml to add your new document to the navigation bar. Use the docid you put in your doc markdown in as the id in the _data/nav_docs.yml file.
2. Run the site locally to test your changes. It will be at http://127.0.0.1/docs/your-new-doc-permalink.html
3. Push your changes to GitHub.

Header Bar Copy Chapter

To modify the header bar, change ./_data/nav.yml.

Top Level Page Copy Chapter

To modify a top-level page, edit the appropriate markdown file in ./top-level/

If you want a top-level page (e.g., http://your-site.com/top-level.html) -- not in /blog/ or /docs/....

1. Create a markdown file in the root ./top-level/. See ./doc-type-examples/top-level-example.md for more information.

2. If you want a visible link to that file, update _data/nav.yml to add a link to your new top-level document in the header bar.

   This is not necessary if you just want to have a page that is linked to from another page, but not exposed as direct link to the user.

3. Run the site locally to test your changes. It will be at http://127.0.0.1/your-top-level-page-permalink.html

4. Push your changes to GitHub.

Other Changes Copy Chapter

• CSS: ./css/main.css or ./_sass/*.scss.
• Images: ./static/images/[docs | posts]/....
• Main Blog post HTML: ./_includes/post.html
• Main Docs HTML: ./_includes/doc.html

File: docs/index.md Copy Chapter

layout: home title: RocksDB | A persistent key-value store id: home Copy Chapter

Features Copy Chapter

{% include content/gridblocks.html data_source=site.data.features align="center" %}

File: docs/README.md Copy Chapter

User Documentation for rocksdb.org Copy Chapter

This directory will contain the user and feature documentation for RocksDB. The documentation will be hosted on GitHub pages.

Contributing Copy Section

See CONTRIBUTING.md for details on how to add or modify content.

Run the Site Locally Copy Section

The requirements for running a GitHub pages site locally is described in GitHub help. The steps below summarize these steps.

If you have run the site before, you can start with step 1 and then move on to step 5.

1. Ensure that you are in the /docs directory in your local RocksDB clone (i.e., the same directory where this README.md exists). The below RubyGems commands, etc. must be run from there.

2. Make sure you have Ruby and RubyGems installed.

   Ruby >= 2.2 is required for the gems. On the latest versions of Mac OS X, Ruby 2.0 is the
   default. Use brew install ruby (or your preferred upgrade mechanism) to install a newer
   version of Ruby for your Mac OS X system.

3. Make sure you have Bundler installed.

   text Copy

   # may require sudo
   gem install bundler

4. Install the project's dependencies

   text Copy

   # run this in the 'docs' directory
   bundle install

   If you get an error when installing nokogiri, you may be running into the problem described
   in this nokogiri issue. You can
   either brew uninstall xz (and then brew install xz after the bundle is installed) or
   xcode-select --install (although this may not work if you have already installed command
   line tools).

5. Run Jekyll's server.

   • On first runs or for structural changes to the documentation (e.g., new sidebar menu item), do a full build.
   text Copy

   bundle exec jekyll serve

   • For content changes only, you can use --incremental for faster builds.
   text Copy

   bundle exec jekyll serve --incremental

   We use bundle exec instead of running straight jekyll because bundle exec will always use the version of Jekyll from our Gemfile. Just running jekyll will use the system version and may not necessarily be compatible.

   • To run using an actual IP address, you can use --host=0.0.0.0
   text Copy

   bundle exec jekyll serve --host=0.0.0.0

   This will allow you to use the IP address associated with your machine in the URL. That way you could share it with other people.

   e.g., on a Mac, you can your IP address with something like ifconfig | grep "inet " | grep -v 127.0.0.1.

6. Either of commands in the previous step will serve up the site on your local device at http://127.0.0.1:4000/ or http://localhost:4000.

Updating the Bundle Copy Section

The site depends on Github Pages and the installed bundle is based on the github-pages gem.
Occasionally that gem might get updated with new or changed functionality. If that is the case,
you can run:

bundle update

to get the latest packages for the installation.