## File: README.md ## RocksDB: A Persistent Key-Value Store for Flash and RAM Storage [](https://circleci.com/gh/facebook/rocksdb) RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on [LevelDB](https://github.com/google/leveldb) by Sanjay Ghemawat (sanjay@google.com) and Jeff Dean (jeff@google.com) This code is a library that forms the core building block for a fast key-value server, especially suited for storing data on flash drives. It has a Log-Structured-Merge-Database (LSM) design with flexible tradeoffs between Write-Amplification-Factor (WAF), Read-Amplification-Factor (RAF) and Space-Amplification-Factor (SAF). It has multi-threaded compactions, making it especially suitable for storing multiple terabytes of data in a single database. Start with example usage here: https://github.com/facebook/rocksdb/tree/main/examples See the [github wiki](https://github.com/facebook/rocksdb/wiki) for more explanation. The public interface is in `include/`. Callers should not include or rely on the details of any other header files in this package. Those internal APIs may be changed without warning. Questions and discussions are welcome on the [RocksDB Developers Public](https://www.facebook.com/groups/rocksdb.dev/) Facebook group and [email list](https://groups.google.com/g/rocksdb) on Google Groups. ## License RocksDB is dual-licensed under both the GPLv2 (found in the COPYING file in the root directory) and Apache 2.0 License (found in the LICENSE.Apache file in the root directory). You may select, at your option, one of the above-listed licenses. --- ## File: docs/_docs/faq.md --- docid: support-faq title: FAQ layout: docs permalink: /docs/support/faq.html --- Here is an ever-growing list of frequently asked questions around RocksDB ## What is RocksDB? RocksDB is an embeddable persistent key-value store for fast storage. RocksDB can also be the foundation for a client-server database but our current focus is on embedded workloads. RocksDB builds on [LevelDB](https://code.google.com/p/leveldb/) to be scalable to run on servers with many CPU cores, to efficiently use fast storage, to support IO-bound, in-memory and write-once workloads, and to be flexible to allow for innovation. For the latest details, watch [Mark Callaghan’s and Igor Canadi’s talk at CMU on 10/2015](https://scs.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=f4e0eb37-ae18-468f-9248-cb73edad3e56). [Dhruba Borthakur’s introductory talk](https://github.com/facebook/rocksdb/blob/gh-pages-old/intro.pdf?raw=true) from the Data @ Scale 2013 conference provides some perspective about how RocksDB has evolved. ## How does performance compare? We benchmarked LevelDB and found that it was unsuitable for our server workloads. The [benchmark results](http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html) look awesome at first sight, but we quickly realized that those results were for a database whose size was smaller than the size of RAM on the test machine – where the entire database could fit in the OS page cache. When we performed the same benchmarks on a database that was at least 5 times larger than main memory, the performance results were dismal. By contrast, we’ve published the [RocksDB benchmark results](https://github.com/facebook/rocksdb/wiki/Performance-Benchmarks) for server side workloads on Flash. We also measured the performance of LevelDB on these server-workload benchmarks and found that RocksDB solidly outperforms LevelDB for these IO bound workloads. We found that LevelDB’s single-threaded compaction process was insufficient to drive server workloads. We saw frequent write-stalls with LevelDB that caused 99-percentile latency to be tremendously large. We found that mmap-ing a file into the OS cache introduced performance bottlenecks for reads. We could not make LevelDB consume all the IOs offered by the underlying Flash storage. ## What is RocksDB suitable for? RocksDB can be used by applications that need low latency database accesses. Possibilities include: * A user-facing application that stores the viewing history and state of users of a website. * A spam detection application that needs fast access to big data sets. * A graph-search query that needs to scan a data set in realtime. * A cache data from Hadoop, thereby allowing applications to query Hadoop data in realtime. * A message-queue that supports a high number of inserts and deletes. ## How big is RocksDB adoption? RocksDB is an embedded storage engine that is used in a number of backend systems at Facebook. In the Facebook newsfeed’s backend, it replaced another internal storage engine called Centrifuge and is one of the many components used. ZippyDB, a distributed key value store service used by Facebook products relies RocksDB. Details on ZippyDB are in [Muthu Annamalai’s talk at Data@Scale in Seattle](https://youtu.be/DfiN7pG0D0k). Dragon, a distributed graph query engine part of the social graph infrastructure, is using RocksDB to store data. Parse has been running [MongoDB on RocksDB in production](http://blog.parse.com/announcements/mongodb-rocksdb-parse/) since early 2015. RocksDB is proving to be a useful component for a lot of other groups in the industry. For a list of projects currently using RocksDB, take a look at our USERS.md list on github. ## How good is RocksDB as a database storage engine? Our engineering team at Facebook firmly believes that RocksDB has great potential as storage engine for databases. It has been proven in production with MongoDB: [MongoRocks](https://github.com/mongodb-partners/mongo-rocks) is the RocksDB based storage engine for MongoDB. [MyRocks](https://code.facebook.com/posts/190251048047090/myrocks-a-space-and-write-optimized-mysql-database/) is the RocksDB based storage engine for MySQL. Using RocksDB we have managed to achieve 2x better compression and 10x less write amplification for our benchmarks compared to our existing MySQL setup. Given our current results, work is currently underway to develop MyRocks into a production ready solution for web-scale MySQL workloads. Follow along on [GitHub](https://github.com/facebook/mysql-5.6)! ## Why is RocksDB open sourced? We are open sourcing this project on [GitHub](http://github.com/facebook/rocksdb) because we think it will be useful beyond Facebook. We are hoping that software programmers and database developers will use, enhance, and customize RocksDB for their use-cases. We would also like to engage with the academic community on topics related to efficiency for modern database algorithms. --- ## File: docs/_docs/getting-started.md --- docid: getting-started title: Getting started layout: docs permalink: /docs/getting-started.html --- ## Overview The RocksDB library provides a persistent key value store. Keys and values are arbitrary byte arrays. The keys are ordered within the key value store according to a user-specified comparator function. The library is maintained by the Facebook Database Engineering Team, and is based on [LevelDB](https://github.com/google/leveldb), by Sanjay Ghemawat and Jeff Dean at Google. This overview gives some simple examples of how RocksDB is used. For the story of why RocksDB was created in the first place, see [Dhruba Borthakur’s introductory talk](https://github.com/facebook/rocksdb/blob/gh-pages-old/intro.pdf?raw=true) from the Data @ Scale 2013 conference. ## Opening A Database A rocksdb database has a name which corresponds to a file system directory. All of the contents of database are stored in this directory. The following example shows how to open a database, creating it if necessary: ```c++ #include #include "rocksdb/db.h" rocksdb::DB* db; rocksdb::Options options; options.create_if_missing = true; rocksdb::Status status = rocksdb::DB::Open(options, "/tmp/testdb", &db); assert(status.ok()); ... ``` If you want to raise an error if the database already exists, add the following line before the rocksdb::DB::Open call: ```c++ options.error_if_exists = true; ``` ## Status You may have noticed the `rocksdb::Status` type above. Values of this type are returned by most functions in RocksDB that may encounter an error. You can check if such a result is ok, and also print an associated error message: ```c++ rocksdb::Status s = ...; if (!s.ok()) cerr << s.ToString() << endl; ``` ## Closing A Database When you are done with a database, just delete the database object. For example: ```c++ /* open the db as described above */ /* do something with db */ delete db; ``` ## Reads And Writes The database provides Put, Delete, and Get methods to modify/query the database. For example, the following code moves the value stored under `key1` to `key2`. ```c++ std::string value; rocksdb::Status s = db->Get(rocksdb::ReadOptions(), key1, &value); if (s.ok()) s = db->Put(rocksdb::WriteOptions(), key2, value); if (s.ok()) s = db->Delete(rocksdb::WriteOptions(), key1); ``` ## Further documentation These are just simple examples of how RocksDB is used. The full documentation is currently on the [GitHub wiki](https://github.com/facebook/rocksdb/wiki). Here are some specific details about the RocksDB implementation: - [RocksDB Overview](https://github.com/facebook/rocksdb/wiki/RocksDB-Overview) - [Immutable BlockBased Table file format](https://github.com/facebook/rocksdb/wiki/Rocksdb-BlockBasedTable-Format) - [Log file format](https://github.com/facebook/rocksdb/wiki/Write-Ahead-Log-File-Format) --- ## File: docs/_top-level/support.md --- layout: top-level title: Support id: support category: support --- ## Need help? Do not hesitate to ask questions if you are having trouble with RocksDB. ### GitHub issues Use [GitHub issues](https://github.com/facebook/rocksdb/issues) to report bugs, issues and feature requests for the RocksDB codebase. ### Facebook Group Use the [RocksDB Facebook group](https://www.facebook.com/groups/rocksdb.dev/) for general questions and discussion about RocksDB. ### FAQ Check out a list of [commonly asked questions](https://github.com/facebook/rocksdb/wiki/RocksDB-FAQ) about RocksDB. --- ## File: docs/doc-type-examples/2016-04-07-blog-post-example.md --- title: Blog Post Example layout: post author: exampleauthor category: blog --- Any local blog posts would go in the `_posts` directory. This is an example blog post introduction, try to keep it short and about a paragraph long, to encourage people to click through to read the entire post. Everything below the `` tag will only show on the actual blog post page, not on the `/blog/` index. Author is defined in `_data/authors.yml` ## No posts? If you have no blog for your site, you can remove the entire `_posts` folder. Otherwise add markdown files in here. See CONTRIBUTING.md for details. --- ## File: docs/doc-type-examples/docs-hello-world.md --- docid: hello-world title: Hello, World! layout: docs permalink: /docs/hello-world.html --- Any local docs would go in the `_docs` directory. ## No documentation? If you have no documentation for your site, you can remove the entire `_docs` folder. Otherwise add markdown files in here. See CONTRIBUTING.md for details. --- ## File: docs/doc-type-examples/top-level-example.md --- layout: top-level title: Support Example id: top-level-example category: top-level --- This is a static page disconnected from the blog or docs collections that can be added at a top-level (i.e., the same level as `index.md`). --- ## File: docs/CONTRIBUTING.md This provides guidance on how to contribute various content to `rocksdb.org`. ## Getting started You should only have to do these one time. - Rename this file to `CONTRIBUTING.md`. - Rename `EXAMPLE-README-FOR-RUNNING-DOCS.md` to `README.md` (replacing the existing `README.md` that came with the template). - Rename `EXAMPLE-LICENSE` to `LICENSE`. - Review the [template information](./TEMPLATE-INFORMATION.md). - Review `./_config.yml`. - Make sure you update `title`, `description`, `tagline` and `gacode` (Google Analytics) in `./_config.yml`. ## Basic Structure Most content is written in markdown. You name the file `something.md`, then have a header that looks like this: ``` --- docid: getting-started title: Getting started with ProjectName layout: docs permalink: /docs/getting-started.html --- ``` Customize these values for each document, blog post, etc. > The filename of the `.md` file doesn't actually matter; what is important is the `docid` being unique and the `permalink` correct and unique too). ## Landing page Modify `index.md` with your new or updated content. If you want a `GridBlock` as part of your content, you can do so directly with HTML: ```

Your Features

  • The [Example](http://example.org/)
  • [Another Example](http://example.com)

    • More information

    Stuff here ``` or with a combination of changing `./_data/features.yml` and adding some Liquid to `index.md`, such as: ``` {% include content/gridblocks.html data_source=site.data.features imagealign="bottom"%} ``` ## Blog To modify a blog post, edit the appopriate markdown file in `./_posts/`. Adding a new blog post is a four-step process. > Some posts have a `permalink` and `comments` in the blog post YAML header. You will not need these for new blog posts. These are an artifact of migrating the blog from Wordpress to gh-pages. 1. Create your blog post in `./_posts/` in markdown (file extension `.md` or `.markdown`). See current posts in that folder or `./doc-type-examples/2016-04-07-blog-post-example.md` for an example of the YAML format. **If the `./_posts` directory does not exist, create it**. - You can add a `` tag in the middle of your post such that you show only the excerpt above that tag in the main `/blog` index on your page. 1. If you have not authored a blog post before, modify the `./_data/authors.yml` file with the `author` id you used in your blog post, along with your full name and Facebook ID to get your profile picture. 1. [Run the site locally](./README.md) to test your changes. It will be at `http://127.0.0.1/blog/your-new-blog-post-title.html` 1. Push your changes to GitHub. ## Docs To modify docs, edit the appropriate markdown file in `./_docs/`. To add docs to the site.... 1. Add your markdown file to the `./_docs/` folder. See `./doc-type-examples/docs-hello-world.md` for an example of the YAML header format. **If the `./_docs/` directory does not exist, create it**. - You can use folders in the `./_docs/` directory to organize your content if you want. 1. Update `_data/nav_docs.yml` to add your new document to the navigation bar. Use the `docid` you put in your doc markdown in as the `id` in the `_data/nav_docs.yml` file. 1. [Run the site locally](./README.md) to test your changes. It will be at `http://127.0.0.1/docs/your-new-doc-permalink.html` 1. Push your changes to GitHub. ## Header Bar To modify the header bar, change `./_data/nav.yml`. ## Top Level Page To modify a top-level page, edit the appropriate markdown file in `./top-level/` If you want a top-level page (e.g., http://your-site.com/top-level.html) -- not in `/blog/` or `/docs/`.... 1. Create a markdown file in the root `./top-level/`. See `./doc-type-examples/top-level-example.md` for more information. 1. If you want a visible link to that file, update `_data/nav.yml` to add a link to your new top-level document in the header bar. > This is not necessary if you just want to have a page that is linked to from another page, but not exposed as direct link to the user. 1. [Run the site locally](./README.md) to test your changes. It will be at `http://127.0.0.1/your-top-level-page-permalink.html` 1. Push your changes to GitHub. ## Other Changes - CSS: `./css/main.css` or `./_sass/*.scss`. - Images: `./static/images/[docs | posts]/....` - Main Blog post HTML: `./_includes/post.html` - Main Docs HTML: `./_includes/doc.html` --- ## File: docs/index.md --- layout: home title: RocksDB | A persistent key-value store id: home --- ## Features {% include content/gridblocks.html data_source=site.data.features align="center" %} --- ## File: docs/README.md ## User Documentation for rocksdb.org This directory will contain the user and feature documentation for RocksDB. The documentation will be hosted on GitHub pages. ### Contributing See [CONTRIBUTING.md](./CONTRIBUTING.md) for details on how to add or modify content. ### Run the Site Locally The requirements for running a GitHub pages site locally is described in [GitHub help](https://help.github.com/articles/setting-up-your-github-pages-site-locally-with-jekyll/#requirements). The steps below summarize these steps. > If you have run the site before, you can start with step 1 and then move on to step 5. 1. Ensure that you are in the `/docs` directory in your local RocksDB clone (i.e., the same directory where this `README.md` exists). The below RubyGems commands, etc. must be run from there. 1. Make sure you have Ruby and [RubyGems](https://rubygems.org/) installed. > Ruby >= 2.2 is required for the gems. On the latest versions of Mac OS X, Ruby 2.0 is the > default. Use `brew install ruby` (or your preferred upgrade mechanism) to install a newer > version of Ruby for your Mac OS X system. 1. Make sure you have [Bundler](http://bundler.io/) installed. ``` # may require sudo gem install bundler ``` 1. Install the project's dependencies ``` # run this in the 'docs' directory bundle install ``` > If you get an error when installing `nokogiri`, you may be running into the problem described > in [this nokogiri issue](https://github.com/sparklemotion/nokogiri/issues/1483). You can > either `brew uninstall xz` (and then `brew install xz` after the bundle is installed) or > `xcode-select --install` (although this may not work if you have already installed command > line tools). 1. Run Jekyll's server. - On first runs or for structural changes to the documentation (e.g., new sidebar menu item), do a full build. ``` bundle exec jekyll serve ``` - For content changes only, you can use `--incremental` for faster builds. ``` bundle exec jekyll serve --incremental ``` > We use `bundle exec` instead of running straight `jekyll` because `bundle exec` will always use the version of Jekyll from our `Gemfile`. Just running `jekyll` will use the system version and may not necessarily be compatible. - To run using an actual IP address, you can use `--host=0.0.0.0` ``` bundle exec jekyll serve --host=0.0.0.0 ``` This will allow you to use the IP address associated with your machine in the URL. That way you could share it with other people. e.g., on a Mac, you can your IP address with something like `ifconfig | grep "inet " | grep -v 127.0.0.1`. 1. Either of commands in the previous step will serve up the site on your local device at http://127.0.0.1:4000/ or http://localhost:4000. ### Updating the Bundle The site depends on Github Pages and the installed bundle is based on the `github-pages` gem. Occasionally that gem might get updated with new or changed functionality. If that is the case, you can run: ``` bundle update ``` to get the latest packages for the installation.