## 1. Project Overview & Quickstart (apache/spark)
## File: README.md
# Apache Spark
Spark is a unified analytics engine for large-scale data processing. It provides
high-level APIs in Scala, Java, Python, and R (Deprecated), and an optimized engine that
supports general computation graphs for data analysis. It also supports a
rich set of higher-level tools including Spark SQL for SQL and DataFrames,
pandas API on Spark for pandas workloads, MLlib for machine learning, GraphX for graph processing,
and Structured Streaming for stream processing.
- Official version:
- Development version:
[](https://opensource.org/licenses/Apache-2.0)
[](https://search.maven.org/search?q=g:org.apache.spark)
[](https://adoptium.net/temurin/releases/?version=17)
[](https://github.com/apache/spark/actions/workflows/build_main.yml)
[](https://codecov.io/gh/apache/spark)
[](https://pypi.org/project/pyspark/)
## Online Documentation
You can find the latest Spark documentation, including a programming
guide, on the [project web page](https://spark.apache.org/documentation.html).
This README file only contains basic setup instructions.
## Build Pipeline Status
| Branch | Status |
|------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| master | [](https://github.com/apache/spark/actions/workflows/release.yml) |
| | [](https://github.com/apache/spark/actions/workflows/publish_snapshot.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_infra_images_cache.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_java21.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_java25.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_non_ansi.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_codegen_jdk.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_uds.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_rockdb_as_ui_backend.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java25.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21_macos26.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21_arm.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_coverage.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.11.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.12_classic_only.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.12_arm.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.12_macos26.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.12_pandas_3.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.13.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.14.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.14_nogil.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_minimum.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_connect40.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_connect.yml) |
| | [](https://github.com/apache/spark/actions/workflows/build_sparkr_window.yml) |
| branch-4.x | [](https://github.com/apache/spark/actions/workflows/build_main.yml?query=branch%3Abranch-4.x) |
| | [](https://github.com/apache/spark/actions/workflows/build_java17.yml?query=branch%3Abranch-4.x) |
| | [](https://github.com/apache/spark/actions/workflows/build_java21.yml?query=branch%3Abranch-4.x) |
| | [](https://github.com/apache/spark/actions/workflows/build_java25.yml?query=branch%3Abranch-4.x) |
| | [](https://github.com/apache/spark/actions/workflows/build_non_ansi.yml?query=branch%3Abranch-4.x) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven.yml?query=branch%3Abranch-4.x) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21.yml?query=branch%3Abranch-4.x) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.11.yml?query=branch%3Abranch-4.x) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.14.yml?query=branch%3Abranch-4.x) |
| branch-4.3 | [](https://github.com/apache/spark/actions/workflows/build_main.yml?query=branch%3Abranch-4.3) |
| | [](https://github.com/apache/spark/actions/workflows/build_java17.yml?query=branch%3Abranch-4.3) |
| | [](https://github.com/apache/spark/actions/workflows/build_java21.yml?query=branch%3Abranch-4.3) |
| | [](https://github.com/apache/spark/actions/workflows/build_java25.yml?query=branch%3Abranch-4.3) |
| | [](https://github.com/apache/spark/actions/workflows/build_non_ansi.yml?query=branch%3Abranch-4.3) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven.yml?query=branch%3Abranch-4.3) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21.yml?query=branch%3Abranch-4.3) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.11.yml?query=branch%3Abranch-4.3) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.14.yml?query=branch%3Abranch-4.3) |
| branch-4.2 | [](https://github.com/apache/spark/actions/workflows/build_main.yml?query=branch%3Abranch-4.2) |
| | [](https://github.com/apache/spark/actions/workflows/build_java17.yml?query=branch%3Abranch-4.2) |
| | [](https://github.com/apache/spark/actions/workflows/build_java21.yml?query=branch%3Abranch-4.2) |
| | [](https://github.com/apache/spark/actions/workflows/build_java25.yml?query=branch%3Abranch-4.2) |
| | [](https://github.com/apache/spark/actions/workflows/build_non_ansi.yml?query=branch%3Abranch-4.2) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven.yml?query=branch%3Abranch-4.2) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21.yml?query=branch%3Abranch-4.2) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.11.yml?query=branch%3Abranch-4.2) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.14.yml?query=branch%3Abranch-4.2) |
| branch-4.1 | [](https://github.com/apache/spark/actions/workflows/build_main.yml?query=branch%3Abranch-4.1) |
| | [](https://github.com/apache/spark/actions/workflows/build_java17.yml?query=branch%3Abranch-4.1) |
| | [](https://github.com/apache/spark/actions/workflows/build_java21.yml?query=branch%3Abranch-4.1) |
| | [](https://github.com/apache/spark/actions/workflows/build_non_ansi.yml?query=branch%3Abranch-4.1) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven.yml?query=branch%3Abranch-4.1) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21.yml?query=branch%3Abranch-4.1) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.11.yml?query=branch%3Abranch-4.1) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.14.yml?query=branch%3Abranch-4.1) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_pypy3.10.yml?query=branch%3Abranch-4.1) |
| branch-4.0 | [](https://github.com/apache/spark/actions/workflows/build_main.yml?query=branch%3Abranch-4.0) |
| | [](https://github.com/apache/spark/actions/workflows/build_java17.yml?query=branch%3Abranch-4.0) |
| | [](https://github.com/apache/spark/actions/workflows/build_java21.yml?query=branch%3Abranch-4.0) |
| | [](https://github.com/apache/spark/actions/workflows/build_non_ansi.yml?query=branch%3Abranch-4.0) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven.yml?query=branch%3Abranch-4.0) |
| | [](https://github.com/apache/spark/actions/workflows/build_maven_java21.yml?query=branch%3Abranch-4.0) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.11.yml?query=branch%3Abranch-4.0) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_pypy3.10.yml?query=branch%3Abranch-4.0) |
| branch-3.5 | [](https://github.com/apache/spark/actions/workflows/build_main.yml?query=branch%3Abranch-3.5) |
| | [](https://github.com/apache/spark/actions/workflows/build_scala213.yml?query=branch%3Abranch-3.5) |
| | [](https://github.com/apache/spark/actions/workflows/build_python_3.9.yml?query=branch%3Abranch-3.5) |
## Building Spark
Spark is built using [Apache Maven](https://maven.apache.org/).
To build Spark and its example programs, run:
```bash
./build/mvn -DskipTests clean package
```
(You do not need to do this if you downloaded a pre-built package.)
More detailed documentation is available from the project site, at
["Building Spark"](https://spark.apache.org/docs/latest/building-spark.html).
For general development tips, including info on developing Spark using an IDE, see ["Useful Developer Tools"](https://spark.apache.org/developer-tools.html).
## Interactive Scala Shell
The easiest way to start using Spark is through the Scala shell:
```bash
./bin/spark-shell
```
Try the following command, which should return 1,000,000,000:
```scala
scala> spark.range(1000 * 1000 * 1000).count()
```
## Interactive Python Shell
Alternatively, if you prefer Python, you can use the Python shell:
```bash
./bin/pyspark
```
And run the following command, which should also return 1,000,000,000:
```python
>>> spark.range(1000 * 1000 * 1000).count()
```
## Example Programs
Spark also comes with several sample programs in the `examples` directory.
To run one of them, use `./bin/run-example [params]`. For example:
```bash
./bin/run-example SparkPi
```
will run the Pi example locally.
You can set the MASTER environment variable when running examples to submit
examples to a cluster. This can be spark:// URL,
"yarn" to run on YARN, and "local" to run
locally with one thread, or "local[N]" to run locally with N threads. You
can also use an abbreviated class name if the class is in the `examples`
package. For instance:
```bash
MASTER=spark://host:7077 ./bin/run-example SparkPi
```
Many of the example programs print usage help if no params are given.
## Running Tests
Testing first requires [building Spark](#building-spark). Once Spark is built, tests
can be run using:
```bash
./dev/run-tests
```
Please see the guidance on how to
[run tests for a module, or individual tests](https://spark.apache.org/developer-tools.html#individual-tests).
There is also a Kubernetes integration test, see resource-managers/kubernetes/integration-tests/README.md
## A Note About Hadoop Versions
Spark uses the Hadoop core library to talk to HDFS and other Hadoop-supported
storage systems. Because the protocols have changed in different versions of
Hadoop, you must build Spark against the same version that your cluster runs.
Please refer to the build documentation at
["Specifying the Hadoop Version and Enabling YARN"](https://spark.apache.org/docs/latest/building-spark.html#specifying-the-hadoop-version-and-enabling-yarn)
for detailed guidance on building for a particular distribution of Hadoop, including
building for particular Hive and Hive Thriftserver distributions.
## Configuration
Please refer to the [Configuration Guide](https://spark.apache.org/docs/latest/configuration.html)
in the online documentation for an overview on how to configure Spark.
## Contributing
Please review the [Contribution to Spark guide](https://spark.apache.org/contributing.html)
for information on how to get started contributing to the project.
---
## File: docs/index.md
---
layout: global
displayTitle: Apache Spark Overview
title: Overview
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
# Downloading
Get Spark from the [downloads page](https://spark.apache.org/downloads.html) of the project website. This documentation is for Spark version {{site.SPARK_VERSION}}. Spark uses Hadoop's client libraries for HDFS and YARN. Downloads are pre-packaged for a handful of popular Hadoop versions.
Users can also download a "Hadoop free" binary and run Spark with any Hadoop version
[by augmenting Spark's classpath](hadoop-provided.html).
Scala and Java users can include Spark in their projects using its Maven coordinates and Python users can install Spark from PyPI.
If you'd like to build Spark from
source, visit [Building Spark](building-spark.html).
Spark runs on both Windows and UNIX-like systems (e.g. Linux, Mac OS), and it should run on any platform that runs a supported version of Java. This should include JVMs on x86_64 and ARM64. It's easy to run locally on one machine --- all you need is to have `java` installed on your system `PATH`, or the `JAVA_HOME` environment variable pointing to a Java installation.
Spark runs on Java 17/21/25, Scala 2.13, Python 3.11+, and R 4.0+ (Deprecated).
Java 25 prior to version 25.0.3 support is deprecated as of Spark 4.2.0.
When using the Scala API, it is necessary for applications to use the same version of Scala that Spark was compiled for. Since Spark 4.0.0, it's Scala 2.13.
# Running the Examples and Shell
Spark comes with several sample programs. Python, Scala, Java, and R examples are in the
`examples/src/main` directory.
To run Spark interactively in a Python interpreter, use
`bin/pyspark`:
./bin/pyspark --master "local[2]"
Sample applications are provided in Python. For example:
./bin/spark-submit examples/src/main/python/pi.py 10
To run one of the Scala or Java sample programs, use
`bin/run-example [params]` in the top-level Spark directory. (Behind the scenes, this
invokes the more general
[`spark-submit` script](submitting-applications.html) for
launching applications). For example,
./bin/run-example SparkPi 10
You can also run Spark interactively through a modified version of the Scala shell. This is a
great way to learn the framework.
./bin/spark-shell --master "local[2]"
The `--master` option specifies the
[master URL for a distributed cluster](submitting-applications.html#master-urls), or `local` to run
locally with one thread, or `local[N]` to run locally with N threads. You should start by using
`local` for testing. For a full list of options, run the Spark shell with the `--help` option.
Since version 1.4, Spark has provided an [R API](sparkr.html) (only the DataFrame APIs are included).
To run Spark interactively in an R interpreter, use `bin/sparkR`:
./bin/sparkR --master "local[2]"
Example applications are also provided in R. For example:
./bin/spark-submit examples/src/main/r/dataframe.R
## Running Spark Client Applications Anywhere with Spark Connect
Spark Connect is a new client-server architecture introduced in Spark 3.4 that decouples Spark
client applications and allows remote connectivity to Spark clusters. The separation between
client and server allows Spark and its open ecosystem to be leveraged from anywhere, embedded
in any application. In Spark 3.4, Spark Connect provides DataFrame API coverage for PySpark and
DataFrame/Dataset API support in Scala.
To learn more about Spark Connect and how to use it, see [Spark Connect Overview](spark-connect-overview.html).
# Launching on a Cluster
The Spark [cluster mode overview](cluster-overview.html) explains the key concepts in running on a cluster.
Spark can run both by itself, or over several existing cluster managers. It currently provides several
options for deployment:
* [Standalone Deploy Mode](spark-standalone.html): simplest way to deploy Spark on a private cluster
* [Hadoop YARN](running-on-yarn.html)
* [Kubernetes](running-on-kubernetes.html)
# Where to Go from Here
**Programming Guides:**
* [Quick Start](quick-start.html): a quick introduction to the Spark API; start here!
* [RDD Programming Guide](rdd-programming-guide.html): overview of Spark basics - RDDs (core but old API), accumulators, and broadcast variables
* [Spark SQL, Datasets, and DataFrames](sql-programming-guide.html): processing structured data with relational queries (newer API than RDDs)
* [Structured Streaming](./streaming/index.html): processing structured data streams with relation queries (using Datasets and DataFrames, newer API than DStreams)
* [Spark Streaming](streaming-programming-guide.html): processing data streams using DStreams (old API)
* [MLlib](ml-guide.html): applying machine learning algorithms
* [GraphX](graphx-programming-guide.html): processing graphs
* [SparkR (Deprecated)](sparkr.html): processing data with Spark in R
* [PySpark](api/python/getting_started/index.html): processing data with Spark in Python
* [Spark SQL CLI](sql-distributed-sql-engine-spark-sql-cli.html): processing data with SQL on the command line
* [Declarative Pipelines](declarative-pipelines-programming-guide.html): building data pipelines that create and maintain multiple tables
**API Docs:**
* [Spark Python API (Sphinx)](api/python/index.html)
* [Spark Scala API (Scaladoc)](api/scala/org/apache/spark/index.html)
* [Spark Java API (Javadoc)](api/java/index.html)
* [Spark R API (Roxygen2)](api/R/index.html)
* [Spark SQL, Built-in Functions (MkDocs)](api/sql/index.html)
**Deployment Guides:**
* [Cluster Overview](cluster-overview.html): overview of concepts and components when running on a cluster
* [Submitting Applications](submitting-applications.html): packaging and deploying applications
* Deployment modes:
* [Standalone Deploy Mode](spark-standalone.html): launch a standalone cluster quickly without a third-party cluster manager
* [YARN](running-on-yarn.html): deploy Spark on top of Hadoop NextGen (YARN)
* [Kubernetes](running-on-kubernetes.html): deploy Spark apps on top of Kubernetes directly
* [Amazon EC2](https://github.com/amplab/spark-ec2): scripts that let you launch a cluster on EC2 in about 5 minutes
* [Spark Kubernetes Operator](https://github.com/apache/spark-kubernetes-operator):
* [SparkApp](https://github.com/apache/spark-kubernetes-operator/blob/main/examples/pyspark-pi.yaml): deploy Spark apps on top of Kubernetes via [operator patterns](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
* [SparkCluster](https://github.com/apache/spark-kubernetes-operator/blob/main/examples/cluster-with-template.yaml): deploy Spark clusters on top of Kubernetes via [operator patterns](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
**Other Documents:**
* [Configuration](configuration.html): customize Spark via its configuration system
* [Monitoring](monitoring.html): track the behavior of your applications
* [Web UI](web-ui.html): view useful information about your applications
* [Tuning Guide](tuning.html): best practices to optimize performance and memory use
* [Job Scheduling](job-scheduling.html): scheduling resources across and within Spark applications
* [Security](security.html): Spark security support
* [Hardware Provisioning](hardware-provisioning.html): recommendations for cluster hardware
* Integration with other storage systems:
* [Cloud Infrastructures](cloud-integration.html)
* [OpenStack Swift](storage-openstack-swift.html)
* [Migration Guide](migration-guide.html): migration guides for Spark components
* [Building Spark](building-spark.html): build Spark using the Maven system
* [Contributing to Spark](https://spark.apache.org/contributing.html)
* [Third Party Projects](https://spark.apache.org/third-party-projects.html): related third party Spark projects
**External Resources:**
* [Spark Homepage](https://spark.apache.org)
* [Spark Community](https://spark.apache.org/community.html) resources, including local meetups
* [StackOverflow tag `apache-spark`](http://stackoverflow.com/questions/tagged/apache-spark)
* [Mailing Lists](https://spark.apache.org/mailing-lists.html): ask questions about Spark here
* AMP Camps: a series of training camps at UC Berkeley that featured talks and
exercises about Spark, Spark Streaming, Mesos, and more. [Videos](https://www.youtube.com/user/BerkeleyAMPLab/search?query=amp%20camp),
are available online for free.
* [Code Examples](https://spark.apache.org/examples.html): more are also available in the `examples` subfolder of Spark ([Python]({{site.SPARK_GITHUB_URL}}/tree/master/examples/src/main/python),
[Scala]({{site.SPARK_GITHUB_URL}}/tree/master/examples/src/main/scala/org/apache/spark/examples),
[Java]({{site.SPARK_GITHUB_URL}}/tree/master/examples/src/main/java/org/apache/spark/examples),
[R]({{site.SPARK_GITHUB_URL}}/tree/master/examples/src/main/r))
---
## File: docs/README.md
---
license: |
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
---
Welcome to the Spark documentation!
This readme will walk you through navigating and building the Spark documentation, which is included
here with the Spark source code. You can also find documentation specific to release versions of
Spark at https://spark.apache.org/documentation.html.
Read on to learn more about viewing documentation in plain text (i.e., markdown) or building the
documentation yourself. Why build it yourself? So that you have the docs that correspond to
whichever version of Spark you currently have checked out of revision control.
## Prerequisites
The Spark documentation build uses a number of tools to build HTML docs and API docs in Scala, Java, Python, R, and SQL.
You need to have [Ruby 3][ruby] and [Python 3][python] installed. Make sure the `bundle` command is available. If not, install it as follows:
[ruby]: https://www.ruby-lang.org/en/documentation/installation/
[python]: https://www.python.org/downloads/
```sh
$ gem install bundler -v 2.4.22
```
After this all the required Ruby dependencies can be installed from the `docs/` directory via Bundler:
```sh
$ cd "$SPARK_HOME"/docs
$ bundle install
```
And the required Python dependencies can be installed using pip:
```sh
$ cd "$SPARK_HOME"
$ pip install --upgrade --group dev
```
To generate the Python or R API docs, you'll also need to [install Pandoc](https://pandoc.org/installing.html).
### R API Documentation (Optional)
If you'd like to generate R API documentation, install these libraries:
```sh
$ sudo Rscript -e 'install.packages(c("remotes", "knitr", "testthat", "rmarkdown"), repos="https://cloud.r-project.org/")'
$ sudo Rscript -e 'remotes::install_version("roxygen2", version = "7.1.2", repos="https://cloud.r-project.org/")'
$ sudo Rscript -e "remotes::install_version('pkgdown', version='2.0.1', repos='https://cloud.r-project.org')"
$ sudo Rscript -e "remotes::install_version('preferably', version='0.4', repos='https://cloud.r-project.org')"
```
Note: Other versions of roxygen2 might work in SparkR documentation generation but `RoxygenNote` field in `$SPARK_HOME/R/pkg/DESCRIPTION` is 7.1.2, which is updated if the version is mismatched.
## Generating the Documentation HTML
We include the Spark documentation as part of the source (as opposed to using a hosted wiki, such as
the github wiki, as the definitive documentation) to enable the documentation to evolve along with
the source code and be captured by revision control (currently git). This way the code automatically
includes the version of the documentation that is relevant regardless of which version or release
you have checked out or downloaded.
In this directory you will find text files formatted using Markdown, with an ".md" suffix. You can
read those text files directly if you want. Start with `index.md`.
Execute `SKIP_API=1 bundle exec jekyll build` from the `docs/` directory to compile the site. Compiling the site with
Jekyll will create a directory called `_site` containing `index.html` as well as the rest of the
compiled files.
```sh
$ cd docs
# Skip generating API docs (which takes a while)
$ SKIP_API=1 bundle exec jekyll build
```
You can also generate the default Jekyll build with API Docs as follows:
```sh
$ bundle exec jekyll build
# Serve content locally on port 4000
$ bundle exec jekyll serve --watch
# Build the site with extra features used on the live page
$ PRODUCTION=1 bundle exec jekyll build
```
## API Docs (Scaladoc, Javadoc, Sphinx, roxygen2, MkDocs)
You can build just the Spark scaladoc and javadoc by running `./build/sbt unidoc` from the `$SPARK_HOME` directory.
Similarly, you can build just the PySpark docs by running `make html` from the
`$SPARK_HOME/python/docs` directory. Documentation is only generated for classes that are listed as
public in `__init__.py`. The SparkR docs can be built by running `$SPARK_HOME/R/create-docs.sh`, and
the SQL docs can be built by running `$SPARK_HOME/sql/create-docs.sh`
after [building Spark](https://github.com/apache/spark#building-spark) first.
When you run `bundle exec jekyll build` in the `docs` directory, it will also copy over the scaladoc and javadoc for the various
Spark subprojects into the `docs` directory (and then also into the `_site` directory). We use a
jekyll plugin to run `./build/sbt unidoc` before building the site so if you haven't run it (recently) it
may take some time as it generates all of the scaladoc and javadoc using [Unidoc](https://github.com/sbt/sbt-unidoc).
The jekyll plugin also generates the PySpark docs using [Sphinx](http://sphinx-doc.org/), SparkR docs
using [roxygen2](https://cran.r-project.org/web/packages/roxygen2/index.html) and SQL docs
using [MkDocs](https://www.mkdocs.org/).
To control what API docs get built, you can set any combination of the following shell variables before you run `bundle exec jekyll build`:
* `SKIP_API=1`: Skip building all the API docs.
* `SKIP_SCALADOC=1`: Skip the Scala and Java API docs.
* `SKIP_PYTHONDOC=1`: Skip the Python API docs.
* `SKIP_RDOC=1`: Skip the R API docs.
* `SKIP_SQLDOC=1`: Skip the SQL API docs.
## Build docs with docker image (Optional)
As a Spark developer, you can generate all documents locally as follows:
Note: Before running it, you need to have `docker` installed.
```sh
$ dev/spark-test-image-util/docs/build-docs
```
It will generate all documents on the `container` and `host`.
Especially when there are conflicts between the libraries required by Python development environment
and the libraries required by generating Python docs environment, this is a good choice.
## 2. Official Technical Reference & Guides (apache/docs)
## File: README.md
## Apache HoraeDB (incubating) Website
> [!IMPORTANT]
> Apache HoraeDB (incubating) is an effort undergoing incubation at the Apache
> Software Foundation (ASF), sponsored by the Apache Incubator PMC.
>
> Please read the [DISCLAIMER](DISCLAIMER) and a full explanation of ["incubating"](https://incubator.apache.org/policy/incubation.html).
This repository stores all the source files of [HoraeDB website](https://horaedb.apache.org), it's built with [hugo](https://gohugo.io/) and [docsy](https://www.docsy.dev/).
## How to run
```bash
# Start a local server for preview
hugo serve
```
By default hugo will listen on http://localhost:1313
## Contributing
Welcome any contributions from the community. You can
- Open an [issue](https://github.com/apache/horaedb-docs/issues) with any suggestions, or
- Submit a [pull request](https://github.com/apache/horaedb-docs/pulls) to improve the website (trivial fixes are welcome).
---
## File: content/en/blog/2024/release-2.0.0.md
---
title: Release 2.0.0
date: 2024-04-23
---
## Upgrade from 1.x.x to 2.0.0
The transition from CeresDB to Apache HoraeDB introduces several breaking changes. To facilitate upgrading from older versions to v2.0.0, specific alterations are necessary.
#### Upgrade Steps
1. Setup required envs
```
export HORAEDB_DEFAULT_CATALOG=ceresdb
```
2. Update config
Etcd's root should be configured both in horaedb and horaemeta
For horaedb
```
[cluster_deployment.etcd_client]
server_addrs = ['127.0.0.1:2379']
root_path = "/rootPath"
```
For horaemeta
```
storage-root-path = "/rootPath"
```
3. Upgrade horaemeta
Horaedb will throw following errors, which is expected
```
2024-01-23 14:37:57.726 ERRO [src/cluster/src/cluster_impl.rs:136] Send heartbeat to meta failed, err:Failed to send heartbeat, cluster:defaultCluster, err:status: Unimplemented, message: "unknown service meta_service.MetaRpcService", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
```
4. Upgrade horaedb
After all server upgraded, the cluster should be ready for read/write, and old data could be queried like before.
## What's Changed
### Breaking Changes
- refactor!: refactor shard version logic by @ZuLiangWang in https://github.com/apache/horaedb/pull/1286
### Features
- feat: support re-acquire shard lock in a fast way by @ShiKaiWi in https://github.com/apache/horaedb/pull/1251
- feat: support alter partition table by @chunshao90 in https://github.com/apache/horaedb/pull/1244
- feat: support access etcd with tls by @ShiKaiWi in https://github.com/apache/horaedb/pull/1254
- feat: support schema validate in remote write by @ShiKaiWi in https://github.com/apache/horaedb/pull/1256
- feat: avoid flush when drop table by @jiacai2050 in https://github.com/apache/horaedb/pull/1257
- feat: opentsdb api support gzip body by @tanruixiang in https://github.com/apache/horaedb/pull/1261
- feat: infer timestamp constraint for single-timestamp column by @Dennis40816 in https://github.com/apache/horaedb/pull/1266
- feat: primary keys support sample by @jiacai2050 in https://github.com/apache/horaedb/pull/1243
- feat: cache space total memory by @jiacai2050 in https://github.com/apache/horaedb/pull/1278
- feat: skip record column values for level0 sst by @jiacai2050 in https://github.com/apache/horaedb/pull/1282
- feat: support write wal logs in columnar format by @ShiKaiWi in https://github.com/apache/horaedb/pull/1179
- feat: support stack size of read threads configurable by @ShiKaiWi in https://github.com/apache/horaedb/pull/1305
- feat: impl DoNothing wal by @jiacai2050 in https://github.com/apache/horaedb/pull/1311
- feat: slow log include remote query by @jiacai2050 in https://github.com/apache/horaedb/pull/1316
- feat: use string for request id by @jiacai2050 in https://github.com/apache/horaedb/pull/1349
- feat: support metrics for number of bytes fetched from object storage by @ShiKaiWi in https://github.com/apache/horaedb/pull/1363
- feat: avoid building dictionary for massive unique column values by @ShiKaiWi in https://github.com/apache/horaedb/pull/1365
- feat: utilize the column cardinality for deciding whether to do dict by @ShiKaiWi in https://github.com/apache/horaedb/pull/1372
- feat: avoid pulling unnecessary columns when querying append mode table by @Rachelint in https://github.com/apache/horaedb/pull/1307
- feat: dist sql analyze by @baojinri in https://github.com/apache/horaedb/pull/1260
- feat: impl priority runtime for read by @jiacai2050 in https://github.com/apache/horaedb/pull/1303
- feat: upgrade horaedbproto by @chunshao90 in https://github.com/apache/horaedb/pull/1408
- feat: block rules support query by @jiacai2050 in https://github.com/apache/horaedb/pull/1420
- feat: try load page indexes by @jiacai2050 in https://github.com/apache/horaedb/pull/1425
- feat: support setting meta_addr&etcd_addrs by env by @chunshao90 in https://github.com/apache/horaedb/pull/1427
- feat: add table status check by @ZuLiangWang in https://github.com/apache/horaedb/pull/1418
- feat: support docker-compose and update README by @chunshao90 in https://github.com/apache/horaedb/pull/1429
- feat: impl layered memtable to reduce duplicated encode during scan by @Rachelint in https://github.com/apache/horaedb/pull/1271
- feat: update disk cache in another thread to avoid blocking normal query process by @jiacai2050 in https://github.com/apache/horaedb/pull/1431
- feat: update pgwire to 0.19 by @sunng87 in https://github.com/apache/horaedb/pull/1436
- feat: filter out MySQL federated components' emitted statements by @chunshao90 in https://github.com/apache/horaedb/pull/1439
- feat: add system_stats lib to collect system stats by @ShiKaiWi in https://github.com/apache/horaedb/pull/1442
- feat(horaectl): initial commit by @chunshao90 in https://github.com/apache/horaedb/pull/1450
- feat: support collect statistics about the engine by @ShiKaiWi in https://github.com/apache/horaedb/pull/1451
- feat: persist sst meta size by @jiacai2050 in https://github.com/apache/horaedb/pull/1440
- feat: add sst level config for benchmark by @zealchen in https://github.com/apache/horaedb/pull/1482
- feat: add exponential backoff when retry by @zealchen in https://github.com/apache/horaedb/pull/1486
### Refactor
- refactor: move wal structs and traits to wal crate by @tisonkun in https://github.com/apache/horaedb/pull/1263
- refactor: improve error readability by @jiacai2050 in https://github.com/apache/horaedb/pull/1265
- refactor: move wal crate to under src folder by @tisonkun in https://github.com/apache/horaedb/pull/1270
- refactor: use `notifier::RequestNotifiers` instead of `dedup_requests::RequestNotifiers` by @baojinri in https://github.com/apache/horaedb/pull/1249
- refactor: conditionally compile wal impls by @tisonkun in https://github.com/apache/horaedb/pull/1272
- refactor: remove unused min/max timestamp in the RowGroup by @ShiKaiWi in https://github.com/apache/horaedb/pull/1297
- refactor: avoid duplicate codes by @ShiKaiWi in https://github.com/apache/horaedb/pull/1371
- refactor: avoid returning metrics in non-analyze sql by @baojinri in https://github.com/apache/horaedb/pull/1410
- refactor: move sub crates to the src directory by @chunshao90 in https://github.com/apache/horaedb/pull/1443
- refactor: adjust cpu's stats by @ShiKaiWi in https://github.com/apache/horaedb/pull/1457
- refactor: refactor compaction process for remote compaction by @Rachelint in https://github.com/apache/horaedb/pull/1476
### Fixed
- fix: dist query dedup by @Rachelint in https://github.com/apache/horaedb/pull/1269
- fix: log third party crates by @jiacai2050 in https://github.com/apache/horaedb/pull/1289
- fix: ensure primary key order by @jiacai2050 in https://github.com/apache/horaedb/pull/1292
- fix: use flag in preflush to indicate whether reorder is required by @jiacai2050 in https://github.com/apache/horaedb/pull/1298
- fix: alter partition table tag column by @chunshao90 in https://github.com/apache/horaedb/pull/1304
- fix: increase wait duration for flush by @jiacai2050 in https://github.com/apache/horaedb/pull/1315
- fix: add license to workspace members by @jiacai2050 in https://github.com/apache/horaedb/pull/1317
- fix: ensure channel size non zero by @jiacai2050 in https://github.com/apache/horaedb/pull/1345
- fix: fix create table result by @ZuLiangWang in https://github.com/apache/horaedb/pull/1354
- Revert "fix: fix create table result" by @ZuLiangWang in https://github.com/apache/horaedb/pull/1355
- fix: fix test create table result by @ZuLiangWang in https://github.com/apache/horaedb/pull/1357
- fix: no write stall by @ShiKaiWi in https://github.com/apache/horaedb/pull/1388
- fix: collect metrics for `get_ranges` by @ShiKaiWi in https://github.com/apache/horaedb/pull/1364
- fix: ignore collecting fetched bytes stats when sst file is read only once by @ShiKaiWi in https://github.com/apache/horaedb/pull/1369
- fix: publich nightly image by @chunshao90 in https://github.com/apache/horaedb/pull/1396
- fix: missing and verbose logs by @ShiKaiWi in https://github.com/apache/horaedb/pull/1398
- fix: fix broken link by @caicancai in https://github.com/apache/horaedb/pull/1399
- fix: the broken link about the issue status by @ShiKaiWi in https://github.com/apache/horaedb/pull/1402
- fix: skip wal encoding when data wal is disabled by @jiacai2050 in https://github.com/apache/horaedb/pull/1401
- fix: disable percentile for distributed tables by @jiacai2050 in https://github.com/apache/horaedb/pull/1406
- fix: compatible for old table options by @Rachelint in https://github.com/apache/horaedb/pull/1432
- fix: get_ranges is not spawned in io-runtime by @ShiKaiWi in https://github.com/apache/horaedb/pull/1426
- fix: table name is normalized when find timestamp column by @jiacai2050 in https://github.com/apache/horaedb/pull/1446
- fix: changes required for migrate dev to main by @jiacai2050 in https://github.com/apache/horaedb/pull/1455
- fix: missing filter index over the primary keys by @ShiKaiWi in https://github.com/apache/horaedb/pull/1456
- fix: random failure of test_collect_system_stats by @ShiKaiWi in https://github.com/apache/horaedb/pull/1459
- fix(ci): refactor ci trigger conditions by @jiacai2050 in https://github.com/apache/horaedb/pull/1474
### Docs
- chore(docs): rename CeresDB to HoraeDB by @caicancai in https://github.com/apache/horaedb/pull/1337
- chore/docs: remove broken link by @caicancai in https://github.com/apache/horaedb/pull/1341
- docs: update CONTRIBUTING.md by @suyanhanx in https://github.com/apache/horaedb/pull/1382
- doc: fix broken link by @caicancai in https://github.com/apache/horaedb/pull/1358
- chore/doc: rename ceresdb to horaedb by @caicancai in https://github.com/apache/horaedb/pull/1332
- doc: add MySQL-Client in README by @jackwener in https://github.com/apache/horaedb/pull/1331
- doc: fix link in illegal markdown format by @jackwener in https://github.com/apache/horaedb/pull/1334
- style: normalize comments/doc in rustfmt by @jackwener in https://github.com/apache/horaedb/pull/1335
- docs: add sudo for install commands by @caicancai in https://github.com/apache/horaedb/pull/1347
- docs: sync GH activities to commits only by @tisonkun in https://github.com/apache/horaedb/pull/1385
- chore(docs): fix invalid repo links by @SYaoJun in https://github.com/apache/horaedb/pull/1452
- chore(docs): fix invalid repo links by @Apricity001 in https://github.com/apache/horaedb/pull/1472
### Chore
- chore(deps): bump golang.org/x/net from 0.5.0 to 0.17.0 in /integration_tests/sdk/go by @dependabot in https://github.com/apache/horaedb/pull/1258
- chore: delete the configuration related to github cache by @tanruixiang in https://github.com/apache/horaedb/pull/1259
- chore: remove backtrace of blocked table by @chunshao90 in https://github.com/apache/horaedb/pull/1267
- ci: setup golang in CI by @tisonkun in https://github.com/apache/horaedb/pull/1275
- chore: remove default features in analytic_engine by @jiacai2050 in https://github.com/apache/horaedb/pull/1277
- chore(deps): bump google.golang.org/grpc from 1.53.0 to 1.56.3 in /integration_tests/sdk/go by @dependabot in https://github.com/apache/horaedb/pull/1280
- test: simplify ceresmeta-server installation by @tisonkun in https://github.com/apache/horaedb/pull/1287
- chore: enable blank issue by @ShiKaiWi in https://github.com/apache/horaedb/pull/1290
- chore: add metrics to inspect write path by @Rachelint in https://github.com/apache/horaedb/pull/1264
- chore: refactor build_meta.sh in integration-test by @chunshao90 in https://github.com/apache/horaedb/pull/1306
- chore: rename ceresdb to horaedb by @chunshao90 in https://github.com/apache/horaedb/pull/1310
- edit: add schema id, schema name, catalog name in TableData by @dust1 in https://github.com/apache/horaedb/pull/1294
- chore: ignore seq check for DoNothing wal by @jiacai2050 in https://github.com/apache/horaedb/pull/1314
- chore: remove community by @jiacai2050 in https://github.com/apache/horaedb/pull/1318
- chore: try to clear ceresdb stuff by @tisonkun in https://github.com/apache/horaedb/pull/1320
- chore: change copyright owner by @tisonkun in https://github.com/apache/horaedb/pull/1321
- ci: stop release docker image before we finish the rename and transfer by @tisonkun in https://github.com/apache/horaedb/pull/1323
- chore: bump deps by @jiacai2050 in https://github.com/apache/horaedb/pull/1325
- chore: rename ceresmeta to horaemeta by @chunshao90 in https://github.com/apache/horaedb/pull/1327
- chore: rename binary to horaedb-server and more by @tisonkun in https://github.com/apache/horaedb/pull/1330
- chore(license): rename `license-header.txt`'s CeresDB to HoraeDB by @caicancai in https://github.com/apache/horaedb/pull/1336
- chore: replace ceresdb with horaedb by @jackwener in https://github.com/apache/horaedb/pull/1338
- chore: more rename to horaedb by @tisonkun in https://github.com/apache/horaedb/pull/1340
- chore: update create table integration test result by @ZuLiangWang in https://github.com/apache/horaedb/pull/1344
- chore: disable frequently failed tests by @jiacai2050 in https://github.com/apache/horaedb/pull/1352
- test: add integration test for alter table options by @caicancai in https://github.com/apache/horaedb/pull/1346
- chore: ignore flush failure when flush by @ShiKaiWi in https://github.com/apache/horaedb/pull/1362
- chore: disable timeout for http api by @jiacai2050 in https://github.com/apache/horaedb/pull/1367
- chore: disable block for http api by @jiacai2050 in https://github.com/apache/horaedb/pull/1368
- config: add .asf.yaml by @chunshao90 in https://github.com/apache/horaedb/pull/1377
- ci: remove missing Required status by @tisonkun in https://github.com/apache/horaedb/pull/1383
- chore: git repo link type fix by @fengmk2 in https://github.com/apache/horaedb/pull/1378
- chore: apply ASF license header by @tanruixiang in https://github.com/apache/horaedb/pull/1384
- chore: add dev mail list and rename ceresdb to horaedb by @tanruixiang in https://github.com/apache/horaedb/pull/1375
- chore: more rename to horaedb by @chunshao90 in https://github.com/apache/horaedb/pull/1387
- chore: add push-nightly-image in workflow by @chunshao90 in https://github.com/apache/horaedb/pull/1389
- chore: update README by @chunshao90 in https://github.com/apache/horaedb/pull/1390
- chore: refactor for better readability by @jiacai2050 in https://github.com/apache/horaedb/pull/1400
- chore: add error log for remote server by @jiacai2050 in https://github.com/apache/horaedb/pull/1407
- chore: update website url by @chunshao90 in https://github.com/apache/horaedb/pull/1404
- chore: upload horaedb logo by @chunshao90 in https://github.com/apache/horaedb/pull/1409
- chore: add slack link by @tanruixiang in https://github.com/apache/horaedb/pull/1395
- chore: update logo by @chunshao90 in https://github.com/apache/horaedb/pull/1414
- chore: update horaedb logo by @chunshao90 in https://github.com/apache/horaedb/pull/1415
- chore: rename ceresformat to logformat by @ZuLiangWang in https://github.com/apache/horaedb/pull/1417
- chore: fix logo link in readme by @chunshao90 in https://github.com/apache/horaedb/pull/1416
- chore: update github pages by @chunshao90 in https://github.com/apache/horaedb/pull/1421
- chore: more rename to horaedb by @chunshao90 in https://github.com/apache/horaedb/pull/1419
- chore: fix error message by @jiacai2050 in https://github.com/apache/horaedb/pull/1412
- chore: remove github pages in asf.yaml by @chunshao90 in https://github.com/apache/horaedb/pull/1428
- chore: skip wal seq check when wal is disabled by @jiacai2050 in https://github.com/apache/horaedb/pull/1430
- chore: enable merge on github by @ShiKaiWi in https://github.com/apache/horaedb/pull/1435
- chore: merge change sets on the dev branch by @ShiKaiWi in https://github.com/apache/horaedb/pull/1423
- chore: fix issue status of README-CN.md by @ShiKaiWi in https://github.com/apache/horaedb/pull/1437
- chore(deps): bump h2 from 0.3.17 to 0.3.24 by @dependabot in https://github.com/apache/horaedb/pull/1448
- chore(deps): bump shlex from 1.1.0 to 1.3.0 by @dependabot in https://github.com/apache/horaedb/pull/1458
- chore: update create tables result by @ZuLiangWang in https://github.com/apache/horaedb/pull/1454
- chore: merge HoraeMeta code into HoreaDB repository by @ZuLiangWang in https://github.com/apache/horaedb/pull/1460
- chore(deps): bump google.golang.org/grpc from 1.47.0 to 1.56.3 in /horaemeta by @dependabot in https://github.com/apache/horaedb/pull/1464
- chore(deps): bump golang.org/x/net from 0.16.0 to 0.17.0 in /horaemeta by @dependabot in https://github.com/apache/horaedb/pull/1465
- chore(deps): bump golang.org/x/crypto from 0.14.0 to 0.17.0 in /horaemeta by @dependabot in https://github.com/apache/horaedb/pull/1462
- chore: rename ci's prefix name by @tanruixiang in https://github.com/apache/horaedb/pull/1467
- chore: fix github issue template by @ZuLiangWang in https://github.com/apache/horaedb/pull/1470
- chore(horaemeta&horaectl): refactor clusters/diagnose response body by @chunshao90 in https://github.com/apache/horaedb/pull/1475
- chore: free disk for ci by @jiacai2050 in https://github.com/apache/horaedb/pull/1484
- deps: bump datafusion by @tanruixiang in https://github.com/apache/horaedb/pull/1445
- horaectl: remove go implementation of horaectl by @chunshao90 in https://github.com/apache/horaedb/pull/1490
- chore: update version to 2.0.0, prepare for releasing v2.0.0 by @chunshao90 in https://github.com/apache/horaedb/pull/1487
## New Contributors
- @Dennis40816 made their first contribution in https://github.com/apache/horaedb/pull/1266
- @caicancai made their first contribution in https://github.com/apache/horaedb/pull/1332
- @jackwener made their first contribution in https://github.com/apache/horaedb/pull/1331
- @suyanhanx made their first contribution in https://github.com/apache/horaedb/pull/1382
- @fengmk2 made their first contribution in https://github.com/apache/horaedb/pull/1378
- @sunng87 made their first contribution in https://github.com/apache/horaedb/pull/1436
- @SYaoJun made their first contribution in https://github.com/apache/horaedb/pull/1452
- @Apricity001 made their first contribution in https://github.com/apache/horaedb/pull/1472
**Full Changelog**: https://github.com/apache/horaedb/compare/v1.2.7...v2.0.0
---
## File: content/en/blog/2024/release-2.1.0.md
---
title: Release 2.1.0
date: 2024-11-18
tags:
- release
---
The Apache HoraeDB(incubating) team are pleased to announce that v2.1.0 is released, which has closed over 60 issues, including two major features:
## 1. New WAL implementation based on local disk.
In previous version, there is a RocksDB-based WAL. Although it works well in most cases, it has following issues:
- Compiling from source can be a challenging task, especially since RocksDB is primarily written in C++.
- For WAL, RocksDB can be somewhat overkill. If you are not familiar with RocksDB, tuning it can be very challenging.
With this new WAL, the above two problems are solved very well, and in performance test, the new WAL slightly outperforms the previous implementation, giving a solid foundation for future optimizations.
Interested readers can refer to the design documentation [here]({{< ref "wal_on_disk" >}}) for more details on this feature.
### How to enable
```
[analytic.wal]
type = "Local"
data_dir = "/path/to/local/wal"
```
## 2. Access object store with [Apache OpenDAL](https://github.com/apache/opendal)
OpenDAL (Open Data Access Layer) is a project that provides a unified API for accessing various data storage backends.
It offers several advantages for developers and organizations. Here are some key benefits:
- Unified API. OpenDAL provides a consistent and unified API for accessing different storage backends, such as AWS S3, Azure Blob Storage, and local file systems.
- Optimized for Efficiency: OpenDAL is built with performance in mind. It includes optimizations to ensure efficient data access and manipulation, making it suitable for high-performance applications.
- Comprehensive Documentation: The project provides detailed documentation, making it easier for developers to get started and understand how to use the library effectively.
In newer versions of OpenDAL, [object_store integration](https://github.com/apache/opendal/tree/main/integrations/object_store) is provided, which is very beneficial for HoraeDB code migration, as the APIs used by the upper layers remain virtually unchanged, and only the object store part needs to be abstracted to a unified OpenDAL operator:
```rust
// Create a new operator
let operator = Operator::new(S3::default())?.finish();
// Create a new object store
let object_store = Arc::new(OpendalStore::new(operator));
```
Additionally, since the Apache OpenDAL implementation of `object_store` is based on the latest version of the object_store, which has breaking changes from the previous version that HoraeDB is using, we've chosen to make it compatible in order to keep the scope of this upgrade as manageable as possible.
In the process of adapting to the new API, the `put_multipart` interface has changed the most, so the main adaptation logic is also here, HoraeDB's approach is to encapsulate the underlying `put_multipart` interface to ensure that the upper layer code is not modified, the details can be found in the reference:
https://github.com/apache/horaedb/blob/v2.1.0/src/components/object_store/src/multi_part.rs
> Note: The adaptation logic is only practical when parquet version < 52.0.0.
## Download
Go to [download pages](/downloads).
## Conclusion
Other bug fixes and improvements can be seen here:
- https://github.com/apache/horaedb/releases/tag/v2.1.0
> As always, we warmly welcome you to join our [community](/community) and share your insights.
---
## File: content/en/blog/_index.md
---
title: "Blog"
weight: 1
simple_list: true
menu:
main:
weight: 30
pre:
---
---
## File: content/en/community/_index.md
---
title: "Community"
params:
contributingUrl: https://github.com/apache/horaedb/blob/main/CONTRIBUTING.md
menu:
main:
weight: 40
---
{{% blocks/section color="white" %}}
# How to Subscribe Mail List
Mailing lists are a form of communication used by the Apache community. Generally speaking, many things in the Apache community are hosted by mailing lists, For example: project q&A, technical discussion, transaction decision, release vote, etc，By subscribing, you can get the first update of the HoraeDB community and keep up with the community. The following list is the main list we are using:
- dev@horaedb.apache.org, Community activity information
- commits@horaedb.apache.org, Code repo update information
In order to subscribe to the dev@horaedb.apache.org mailing list, the steps are as follows：
1. Send an email without any content or subject: dev-subscribe@horaedb.apache.org
2. Wait until you receive an email with the subject line confirm subscribe to dev@horaedb.apache.org (if you have not received it for a long time, please confirm whether the email is blocked by your email, if you have not been blocked and will receive a reply for a long time, return to step 1)
3. Reply directly to the email without changing the subject line or adding the email content.
4. Wait until you receive an email with the subject line WELCOME to dev@horaedb.apache.org .
5. If you receive an email from dev, you have successfully subscribed to the email. To initiate a discussion, you can send an email directly to dev@horaedb.apache.org, which will be sent to everyone who subscribed to the mailing list.
## Unsubscribe from the mail list
The steps for unsubscribing to a mailing list are similar to those for subscribing to a mailing list:
1. Send an email without any content or subject to: dev-unsubscribe@horaedb.apache.org
2. Wait until you receive an email with the subject line confirm unsubscribe from dev@horaedb.apache.org
3. Reply directly to the email without changing the subject line or adding the email content
4. Wait until you receive an email with the subject line GOODBYE from dev@horaedb.apache.org
5. Unsubscribe success
{{% /blocks/section %}}
---
## File: content/en/docs/design/architecture.md
---
title: "Introduction to HoraeDB's Architecture"
---
## Target
- Provide the overview of HoraeDB to the developers who want to know more about HoraeDB but have no idea where to start.
- Make a brief introduction to the important modules of HoraeDB and the connections between these modules but details about their implementations are not be involved.
## Motivation
HoraeDB is a timeseries database (**TSDB**). However, HoraeDB's goal is to handle both timeseries and analytic workloads compared with the classic TSDB, which usually have a poor performance in handling analytic workloads.
In the classic timeseries database, the `Tag` columns (InfluxDB calls them `Tag` and Prometheus calls them `Label`) are normally indexed by generating an inverted index. However, it is found that the cardinality of `Tag` varies in different scenarios. And in some scenarios the cardinality of `Tag` is very high (we name this case after analytic workload), and it takes a very high cost to store and retrieve the inverted index. On the other hand, it is observed that scanning+pruning often used by the analytical databases can do a good job to handle such analytic workload.
The basic design idea of HoraeDB is to adopt a hybrid storage format and the corresponding query method for a better performance in processing both timeseries and analytic workloads.
## Architecture
```plaintext
┌──────────────────────────────────────────┐
│ RPC Layer (HTTP/gRPC/MySQL) │
└──────────────────────────────────────────┘
┌──────────────────────────────────────────┐
│ SQL Layer │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Parser │ │ Planner │ │
│ └─────────────────┘ └─────────────────┘ │
└──────────────────────────────────────────┘
┌───────────────────┐ ┌───────────────────┐
│ Interpreter │ │ Catalog │
└───────────────────┘ └───────────────────┘
┌──────────────────────────────────────────┐
│ Query Engine │
│ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Optimizer │ │ Executor │ │
│ └─────────────────┘ └─────────────────┘ │
└──────────────────────────────────────────┘
┌──────────────────────────────────────────┐
│ Pluggable Table Engine │
│ ┌────────────────────────────────────┐ │
│ │ Analytic │ │
│ │┌────────────────┐┌────────────────┐│ │
│ ││ Wal ││ Memtable ││ │
│ │└────────────────┘└────────────────┘│ │
│ │┌────────────────┐┌────────────────┐│ │
│ ││ Flush ││ Compaction ││ │
│ │└────────────────┘└────────────────┘│ │
│ │┌────────────────┐┌────────────────┐│ │
│ ││ Manifest ││ Object Store ││ │
│ │└────────────────┘└────────────────┘│ │
│ └────────────────────────────────────┘ │
│ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
│ Another Table Engine │ │
│ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ │
└──────────────────────────────────────────┘
```
The figure above shows the architecture of HoraeDB stand-alone service and the details of some important modules will be described in the following part.
### RPC Layer
module path: https://github.com/apache/incubator-horaedb/tree/main/server
The current RPC supports multiple protocols including HTTP, gRPC, MySQL.
Basically, HTTP and MySQL are used to debug HoraeDB, query manually and perform DDL operations (such as creating, deleting tables, etc.). And gRPC protocol can be regarded as a customized protocol for high-performance, which is suitable for massive reading and writing operations.
### SQL Layer
module path: https://github.com/apache/incubator-horaedb/tree/main/query_frontend
SQL layer takes responsibilities for parsing sql and generating the query plan.
Based on [sqlparser](https://github.com/sqlparser-rs/sqlparser-rs) a sql dialect, which introduces some key concepts including `Tag` and `Timestamp`, is provided for processing timeseries data. And by utilizing [DataFusion](https://github.com/apache/arrow-datafusion) the planner is able to generate both regular logical plans and tailored ones which is used to implement the special operators defined by timeseries queries, e.g `PromQL`.
### Interpreter
module path: https://github.com/apache/incubator-horaedb/tree/main/interpreters
The `Interpreter` module encapsulates the SQL `CRUD` operations. In the query procedure, a sql received by HoraeDB is parsed, converted into the query plan and then executed in some specific interpreter, such as `SelectInterpreter`, `InsertInterpreter` and etc.
### Catalog
module path: https://github.com/apache/incubator-horaedb/tree/main/catalog_impls
`Catalog` is actually the module managing metadata and the levels of metadata adopted by HoraeDB is similar to PostgreSQL: `Catalog > Schema > Table`, but they are only used as namespace.
At present, `Catalog` and `Schema` have two different kinds of implementation for standalone and distributed mode because some strategies to generate ids and ways to persist metadata differ in different mode.
### Query Engine
module path: https://github.com/apache/incubator-horaedb/tree/main/query_engine
`Query Engine` is responsible for optimizing and executing query plan given a basic SQL plan provided by SQL layer and now such work is mainly delegated to [DataFusion](https://github.com/apache/arrow-datafusion).
In addition to the basic functions of SQL, HoraeDB also defines some customized query protocols and optimization rules for some specific query plans by utilizing the extensibility provided by [DataFusion](https://github.com/apache/arrow-datafusion). For example, the implementation of `PromQL` is implemented in this way and read it if you are interested.
### Pluggable Table Engine
module path: https://github.com/apache/incubator-horaedb/tree/main/table_engine
`Table Engine` is actually a storage engine for managing tables in HoraeDB and the pluggability of `Table Engine` is a core design of HoraeDB which matters in achieving our long-term target, e.g supporting handle log or tracing workload by implementing new storage engines. HoraeDB will have multiple kinds of `Table Engine` for different workloads and the most appropriate one should be chosen as the storage engine according to the workload pattern.
Now the requirements for a `Table Engine` are:
- Manage all the shared resources under the engine:
- Memory
- Storage
- CPU
- Manage metadata of tables such as table schema and table options;
- Provide `Table` instances which provides `read` and `write` methods;
- Take responsibilities for creating, opening, dropping and closing `Table` instance;
- ....
Actually the things that a `Table Engine` needs to process are a little complicated. And now in HoraeDB only one `Table Engine` called `Analytic` is provided and does a good job in processing analytical workload, but it is not ready yet to handle the timeseries workload (we plan to enhance it for a better performance by adding some indexes which help handle timeseries workload).
The following part gives a description about details of `Analytic Table Engine`.
#### WAL
module path: https://github.com/apache/incubator-horaedb/tree/main/wal
The model of HoraeDB processing data is `WAL` + `MemTable` that the recent written data is written to `WAL` first and then to `MemTable` and after a certain amount of data in `MemTable` is accumulated, the data will be organized in a query-friendly form to persistent devices.
Now three implementations of `WAL` are provided for standalone and distributed mode:
- For standalone mode, `WAL` is based on `RocksDB` and data is persisted on the local disk.
- For distributed mode, `WAL` is required as a distributed component and to be responsible for durability of the newly written data, so now we provide an implementation based on [OceanBase](https://github.com/oceanbase/oceanbase).
- For distributed mode, in addition to [OceanBase](https://github.com/oceanbase/oceanbase), we also provide a more lightweight implementation based on [`Apache Kafka`](https://github.com/apache/kafka).
#### MemTable
module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/memtable
For `WAL` can't provide efficient data retrieval, the newly written data is also stored in `Memtable` for efficient data retrieval, after a certain amount of data is reached, HoraeDB organizes the data in `MemTable` into a query-friendly storage format (`SST`) and stores it to the persistent device.
The current implementation of `MemTable` is based on [agatedb's skiplist](https://github.com/tikv/agatedb/blob/8510bff2bfde5b766c3f83cf81c00141967d48a4/skiplist). It allows concurrent reads and writes and can control memory usage based on [Arena](https://github.com/apache/incubator-horaedb/tree/main/components/skiplist).
#### Flush
module path: https://github.com/apache/incubator-horaedb/blob/main/analytic_engine/src/instance/flush_compaction.rs
What `Flush` does is that when the memory usage of `MemTable` reaches the threshold, some `MemTables` are selected for flushing into query-friendly `SST`s saved on persistent device.
During the flushing procedure, the data will be divided by a certain time range (which is configured by table option `Segment Duration`), and any `SST` is ensured that the timestamps of the data in it are in the same `Segment`. Actually this is also a common operation in most timeseries databases which organizes data in the time dimension to speed up subsequent time-related operations, such as querying data over a time range and assisting purge data outside the `TTL`.
#### Compaction
module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/compaction
The data of `MemTable` is flushed as `SST`s, but the file size of recently flushed `SST` may be very small. And too small or too many `SST`s lead to the poor query performance. Therefore, `Compaction` is then introduced to rearrange the `SST`s so that the multiple smaller `SST` files can be compacted into a larger `SST` file.
#### Manifest
module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/meta
`Manifest` takes responsibilities for managing tables' metadata of `Analytic Engine` including:
- Table schema and table options;
- The sequence number where the newest flush finishes;
- The information of all the `SST`s belonging to the table.
Now the `Manifest` is based on `WAL` and `Object Storage`. The newly written updates on the `Manifest` are persisted as logs in `WAL`, and in order to avoid infinite expansion of `Manifest` (actually every `Flush` leads to an update), `Snapshot` is also introduced to clean up the history of metadata updates, and the generated `Snapshot` will be saved to `Object Storage`.
#### Object Storage
module path: https://github.com/apache/incubator-horaedb/tree/main/components/object_store
The `SST` generated by `Flush` needs to be persisted and the abstraction of the persistent storage device is `ObjectStore` including multiple implementations:
- Based on local file system;
- Based on [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service).
The distributed architecture of HoraeDB separates storage and computing, which requires `Object Store` needs to be a highly available and reliable service independent of HoraeDB. Therefore, storage systems like [Amazon S3](https://aws.amazon.com/s3/), [Alibaba Cloud OSS](https://www.alibabacloud.com/product/object-storage-service) is a good choice and in the future implementations on storage systems of some other cloud service providers is planned to provide.
#### SST
module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/sst
`SST` is actually an abstraction that can have multiple specific implementations. The current implementation is based on [Parquet](https://parquet.apache.org/), which is a column-oriented data file format designed for efficient data storage and retrieval.
The format of `SST` is very critical for retrieving data and is also the most important part to perform well in handling both timeseries and analytic workloads. At present, our [Parquet](https://parquet.apache.org/)-based implementation is good at processing analytic workload but is poor at processing timeseries workload. In our roadmap, we will explore more storage formats in order to achieve a good performance in both workloads.
#### Space
module path: https://github.com/apache/incubator-horaedb/blob/main/analytic_engine/src/space.rs
In `Analytic Engine`, there is a concept called `space` and here is an explanation for it to resolve some ambiguities when read source code. Actually `Analytic Engine` does not have the concept of `catalog` and `schema` and only provides two levels of relationship: `space` and `table`. And in the implementation, the `schema id` (which should be unique across all `catalog`s) on the upper layer is actually mapped to `space id`.
The `space` in `Analytic Engine` serves mainly for isolation of resources for different tenants, such as the usage of memory.
## Critical Path
After a brief introduction to some important modules of HoraeDB, we will give a description for some critical paths in code, hoping to provide interested developers with a guide for reading the code.
### Query
```plaintext
┌───────┐ ┌───────┐ ┌───────┐
│ │──1──▶│ │──2──▶│ │
│Server │ │ SQL │ │Catalog│
│ │◀─10──│ │◀─3───│ │
└───────┘ └───────┘ └───────┘
│ ▲
4│ 9│
│ │
▼ │
┌─────────────────────────────────────┐
│ │
│ Interpreter │
│ │
└─────────────────────────────────────┘
│ ▲
5│ 8│
│ │
▼ │
┌──────────────────┐
│ │
│ Query Engine │
│ │
└──────────────────┘
│ ▲
6│ 7│
│ │
▼ │
┌─────────────────────────────────────┐
│ │
│ Table Engine │
│ │
└─────────────────────────────────────┘
```
Take `SELECT` SQL as an example. The figure above shows the query procedure and the numbers in it indicates the order of calling between the modules.
Here are the details:
- Server module chooses a proper rpc module (it may be HTTP, gRPC or mysql) to process the requests according the protocol used by the requests;
- Parse SQL in the request by the parser;
- With the parsed sql and the information provided by catalog/schema module, [DataFusion](https://github.com/apache/arrow-datafusion) can generate the logical plan;
- With the logical plan, the corresponding `Interpreter` is created and logical plan will be executed by it;
- For the logical plan of normal `Select` SQL, it will be executed through `SelectInterpreter`;
- In the `SelectInterpreter` the specific query logic is executed by the `Query Engine`:
- Optimize the logical plan;
- Generate the physical plan;
- Optimize the physical plan;
- Execute the physical plan;
- The execution of physical plan involves `Analytic Engine`:
- Data is obtained by `read` method of `Table` instance provided by `Analytic Engine`;
- The source of the table data is `SST` and `Memtable`, and the data can be filtered by the pushed down predicates;
- After retrieving the table data, `Query Engine` will complete the specific computation and generate the final results;
- `SelectInterpreter` gets the results and feeds them to the protocol module;
- After the protocol layer converts the results, the server module responds to the client with them.
The following is the flow of function calls in version [v1.2.2](https://github.com/apache/incubator-horaedb/releases/tag/v1.2.2):
```
/* Detailed source-code truncated for AI context efficiency. */
```
1. The received request will be forwarded to `handle_sql` after various protocol conversions, and since the request may not be processed by this node, it may need to be forwarded to `maybe_forward_sql_query` to handle the forwarding logic.
2. After constructing the `ForwardRequest` in `maybe_forward_sql_query`, call `forward`
3. After constructing the `RouteRequest` in `forward`, call `route`
4. Use `route` to get the destination node `endpoint` and return to `forward`.
5. Call `forward_with_endpoint` to forward the request
6. return `forward`
7. return `maybe_forward_sql_query`
8. return `handle_sql`
9. If this is a `Local` request, call `fetch_sql_query_output` to process it
10. Call `parse_sql` to parse `sql` into `Statment`
11. return `fetch_sql_query_output`
12. Call `statement_to_plan` with `Statment`
13. Construct `Planner` with `ctx` and `Statment`, and call the `statement_to_plan` method of `Planner`
14. The `planner` will call the corresponding `planner` method for the requested category, at this point our `sql` is a query and will call `sql_statement_to_plan`
15. Call `sql_statement_to_datafusion_plan` , which will generate the `datafusion` object, and then call `SqlToRel::sql_statement_to_plan`
16. The generated logical plan is returned from `SqlToRel::sql_statement_to_plan`
17. return
18. return
19. return
20. Call `execute_plan_involving_partition_table` (in the default configuration) for subsequent optimization and execution of this logical plan
21. Call `build_interpreter` to generate `Interpreter`
22. return
23. Call `Interpreter's` `interpreter_execute_plan` method for logical plan execution.
24. The corresponding `execute` function is called, at this time the `sql` is a query, so the execute of the `SelectInterpreter` will be called
25. call `execute_logical_plan` , which will call `build_df_session_ctx` to generate the optimizer
26. `build_df_session_ctx` will use the `config` information to generate the corresponding context, first using datafusion and some custom optimization rules (in logical_optimize_rules()) to generate the logical plan optimizer, using `apply_adapters_for_physical_optimize_rules` to generate the physical plan optimizer
27. return optimizer
28. Call `optimize_plan`, using the optimizer just generated to first optimize the logical plan and then the physical plan
29. Return to optimized physical plan
30. execute physical plan
31. returned after execution
32. After collecting the results of all slices, return
33. return
34. return
35. return
36. Return to the upper layer for network protocol conversion and finally return to the request sender
### Write
```plaintext
┌───────┐ ┌───────┐ ┌───────┐
│ │──1──▶│ │──2──▶│ │
│Server │ │ SQL │ │Catalog│
│ │◀─8───│ │◀─3───│ │
└───────┘ └───────┘ └───────┘
│ ▲
4│ 7│
│ │
▼ │
┌─────────────────────────────────────┐
│ │
│ Interpreter │
│ │
└─────────────────────────────────────┘
│ ▲
│ │
│ │
│ │
│ │ ┌──────────────────┐
│ │ │ │
5│ 6│ │ Query Engine │
│ │ │ │
│ │ └──────────────────┘
│ │
│ │
│ │
▼ │
┌─────────────────────────────────────┐
│ │
│ Table Engine │
│ │
└─────────────────────────────────────┘
```
Take `INSERT` SQL as an example. The figure above shows the query procedure and the numbers in it indicates the order of calling between the modules.
Here are the details:
- Server module chooses a proper rpc module (it may be HTTP, gRPC or mysql) to process the requests according the protocol used by the requests;
- Parse SQL in the request by the parser;
- With the parsed sql and the catalog/schema module, [DataFusion](https://github.com/apache/arrow-datafusion) can generate the logical plan;
- With the logical plan, the corresponding `Interpreter` is created and logical plan will be executed by it;
- For the logical plan of normal `INSERT` SQL, it will be executed through `InsertInterpreter`;
- In the `InsertInterpreter`, `write` method of `Table` provided `Analytic Engine` is called:
- Write the data into `WAL` first;
- Write the data into `MemTable` then;
- Before writing to `MemTable`, the memory usage will be checked. If the memory usage is too high, the flush process will be triggered:
- Persist some old MemTables as `SST`s;
- Store updates about the new `SST`s and the flushed sequence number of `WAL` to `Manifest`;
- Delete the corresponding `WAL` entries;
- Server module responds to the client with the execution result.
---
## File: content/en/docs/design/clustering.md
---
title: "Introduction to Architecture of HoraeDB Cluster"
---Note: Some of the features mentioned in the article have not yet been implemented.
## Overview
```
/* Detailed source-code truncated for AI context efficiency. */
```
The diagram above describes the architecture of a HoraeDB cluster, where some key concepts need to be explained:
- `HoraeMeta Cluster`: Takes responsibilities for managing the metadata and resource scheduling of the cluster;
- `Shard(L)/Shard(F)`: Leader shard and follower shard consisting of multiple tables;
- `HoraeDB`: One HoraeDB instance consisting of multiple shards;
- `WAL Service`: Write-ahead log service for storing new-written real-time data;
- `Object Storage`: Object storage service for storing SST converted from memtable;
From the architecture diagram above, it can be concluded that the compute and storage are separated in the HoraeDB cluster, which makes it easy to implement useful distributed features, such as elastic autoscaling of compute/storage resources, high availability, load balancing, and so on.
Let's dive into some of the key components mentioned above before explaining how these features are implemented.
### Shard
`Shard` is the basic scheduling unit in the cluster, which consists of a group of tables. And the tables in a shard share the same region for better storage locality in the `WAL Service`, and because of this, it is efficient to recover the data of all tables in the shard by scanning the entire WAL region. For most of implementations of `WAL Service`, without the shard concept, it costs a lot to recover the table data one by one due to massive random IO, and this case will deteriorate sharply when the number of tables grows to a certain level.
A specific role, `Leader` or `Follower`, should be assigned to a shard. A pair of leader-follower shards share the same set of tables, and the leader shard can serve the write and query requests from the client while the follower shard can only serve the read-only requests, and must synchronize the newly written data from the WAL service in order to provide the latest snapshot for data retrieval. Actually, the follower is not needed if the high availability is not required, while with at least one follower, it takes only a short time to resume service by simply switching the `Follower` to `Leader` when the HoraeDB instance on which the leader shard exists crashes.
The diagram below concludes the relationship between HoraeDB instance, `Shard`, `Table`. As shown in the diagram, the leader and follower shards are interleaved on the HoraeDB instance.
```plaintext
┌─HoraeDB Instance0──────┐ ┌─HoraeDB Instance1──────┐
│ ┌─Shard0(L)────────┐ │ │ ┌─Shard0(F)────────┐ │
│ │ ┌────┬────┬────┐ │ │ │ │ ┌────┬────┬────┐ │ │
│ │ │ T0 │ T1 │ T2 │ │ │ │ │ │ T0 │ T1 │ T2 │ │ │
│ │ └────┴────┴────┘ │ │ │ │ └────┴────┴────┘ │ │
│ └──────────────────┘ │ │ └──────────────────┘ │
│ │ │ │
│ ┌─Shard1(F)────────┐ │ │ ┌─Shard1(L)────────┐ │
│ │ ┌────┬────┬────┐ │ │ │ │ ┌────┬────┬────┐ │ │
│ │ │ T0 │ T1 │ T2 │ │ │ │ │ │ T0 │ T1 │ T2 │ │ │
│ │ └────┴────┴────┘ │ │ │ │ └────┴────┴────┘ │ │
│ └──────────────────┘ │ │ └──────────────────┘ │
└────────────────────────┘ └────────────────────────┘
```
Since `Shard` is the basic scheduling unit, it is natural to introduce some basic shard operations:
- Create/Drop table to/from a shard;
- Open/Close a shard;
- Split one shard into two shards;
- Merge two shards into one shard;
- Switch the role of a shard;
With these basic shard operations, some complex scheduling logic can be implemented, e.g. perform an expansion by splitting one shard into two shards and migrating one of them to the new HoraeDB instance.
### HoraeMeta
`HoraeMeta` is implemented by embedding an ETCD inside to ensure consistency and takes responsibilities for cluster metadata management and scheduling.
The cluster metadata includes:
- Table information, such as table name, table ID, and which cluster the table belongs to;
- The mapping between table and shard and between shard and HoraeDB instance;
- ...
As for the cluster scheduling work, it mainly includes:
- Receiving the heartbeats from the HoraeDB instances and determining the online status of these registered instances;
- Assigning specific role shards to the registered HoraeDB instances;
- Participating in table creation by assigning a unique table ID and the most appropriate shard to the table;
- Performing load balancing through shard operations according to the load information sent with the heartbeats;
- Performing expansion through shard operations when new instances are registered;
- Initiating failover through shard operations when old instances go offline;
### Route
In order to avoid the overhead of forwarding requests, the communication between clients and the HoraeDB instances is peer-to-peer, that is to say, the client should retrieve routing information from the server before sending any specific write/query requests.
Actually, the routing information is decided by the `HoraeMeta`, but clients are only allowed the access to it through the HoraeDB instances rather than `HoraeMeta`, to avoid potential performance issues on the `HoraeMeta`.
### WAL Service & Object Storage
In the HoraeDB cluster, `WAL Service` and `Object Storage` exist as separate distributed systems featured with HA, data replication and scalability. Current distributed implementations for `WAL Service` includes `Kafka` and `OBKV` (access `OceanBase` by its table api), and the implementations for `Object Storage` include popular object storage services, such as AWS S3, Azure object storage and Aliyun OSS.
The two components are similar in that they are introduced to serve as the underlying storage layer for separating compute and storage, while the difference between two components is obvious that `WAL Service` is used to store the newly written data from the real-time write requests whose individual size is small but quantity is large, and `Object Storage` is used to store the read-friendly data files (SST) organized in the background, whose individual size is large and aggregate size is much larger.
The two components make it much easier to implement the horaedb cluster, which features horizontal scalability, high availability and load balancing.
## Scalability
Scalability is an important feature for a distributed system. Let's take a look at to how the horizontal scalability of the HoraeDB cluster is achieved.
First, the two storage components (`WAL Service` and `Object Storage`) should be horizontally scalable when deciding on the actual implementations for them, so the two storage services can be expanded separately if the storage capacity is not sufficient.
It will be a little bit complex when discussing the scalability of the compute service. Basically, these cases will bring the capacity problem:
- Massive queries on massive tables;
- Massive queries on a single large table;
- Massive queries on a normal table;
For the first case, it is easy to achieve horizontal scalability just by assigning shards that are created or split from old shards to expanded HoraeDB instances.
For the second case, the table partitioning is proposed and after partitioning, massive queries are distributed across multiple HoraeDB instances.
And the last case is the most important and the most difficult. Actually, the follower shard can handle part of the queries, but the number of follower shards is limited by the throughput threshold of the synchronization from the WAL regions. As shown in the diagram below, a pure compute shard can be introduced if the followers are not enough to handle the massive queries. Such a shard is not required to synchronize data with the leader shard, and retrieves the newly written data from the leader/follower shard only when the query comes. As for the SSTs required by the query, they will be downloaded from `Object Storage` and cached afterwards. With the two parts of the data, the compute resources are fully utilized to execute the CPU-intensive query plan. As we can see, such a shard can be added with only a little overhead (retrieving some data from the leader/follower shard when it needs), so to some extent, the horizontal scalability is achieved.
```plaintext
┌HoraeDB─────┬┬─┐
┌──newly written─│ │ │TableN││ │
▼ └──Shard(L/F)┴┴─┘
┌───────┐ Query ┌HoraeDB─────┬┬─┐
│client │────────▶│ │ │TableN││ │
└───────┘ └──Shard─────┴┴─┘ ┌───────────────┐
▲ │ Object │
└───old SST──────│ Storage │
└───────────────┘
```
## High Availability
Assuming that `WAL service` and `Object Storage` are highly available, the high availability of the HoraeDB cluster can be achieved by such a procedure:
- When detecting that the heartbeat is broken, `HoraeMeta` determines that the HoraeDB instance is offline;
- The follower shards whose paired leader shards exist on the offline instance are switched to leader shards for fast failover;
- A slow failover can be achieved by opening the crashed shards on another instance if such follower shards don't exist.
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Load Balancing
HoraeMeta collects the instance load information contained in the received heartbeats to create a load overview of the whole cluster, according to which the load balancing can be implemented as an automatic mechanism:
- Pick a shard on a low-load instance for the newly created table;
- Migrate a shard from a high-load instance load to another low-load instance;
- Split the large shard on the high-load instance and migrate the split shards to other low-load instances;
---
## File: content/en/docs/design/compaction_offload.md
---
title: "Compaction Offload"
---
**Note: This feature is still in development.**
This chapter discusses compaction offload, which is designed to separate the compaction workload from the local horaedb nodes and delegate it to external compaction nodes.
## Overview
```plaintext
┌─────────────────────────────────────────────────────────────────────────┐
│ │
│ HoraeMeta Cluster │
│ │
└─────────────────────────────────────────────────────────────────────────┘
▲ ▲ | |
│ │1.Fetch compaction │(Monitor compaction│
│ │ node info │node) │
| │ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐ ┌────────────┐
│ │ │ │2.Offload Task│ │ │ │
│ HoraeDB │ │ HoraeDB │ ─────────▶ │ Compaction │ │ Compaction │
│ │ │ │ ◀───────── │ Node │ │ Node │
└────────────┘ └────────────┘ 4.Ret Result └────────────┘ └────────────┘
| | | |
│ 5.Update the │ │ 3.Compact │
│ SSTable │ │ │
▼ ▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ ┌─────────────────────┐ │
│ Object Storage │ Temporary Workspace │ │
│ └─────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
```
The diagram above describes the architecture of cluster for compaction offload, where some key concepts need to be explained:
- `Compaction Node`: Takes responsibility to handle offloaded compaction tasks. The compaction node receives the compaction task and performs the actual merging of SSTables, then sends back the task result to HoraeDB.
- `HoraeMeta Cluster`: HoraeMeta acts as a compaction nodes manager in the compaction offload scenario. It monitors the compaction nodes cluster and schedule the compaction nodes.
The procedure of remote compaction based above architecture diagram is:
1. HoraeDB servers fetch the information of suitable compaction nodes from the HoraeMeta.
2. HoraeDB submit the compaction task to the remote compaction node, according to the information fetched from HoraeMeta.
3. Compaction node executes the task and write results to the temporary workspace.
4. Compaction node sends compaction results back to HoraeDB.
5. HoraeDB receives the result, installs the data in temporary workspace and purges compaction input files.
The architecture above makes it easy to implement some wonderful features like load balancing and high availability. Let's dive into the key components in the architecture and talking about how these features are implemented.
### Compaction Node
`Compaction Node` runs the main logic of compaction. It is implemented based on HoraeDB and distinguished by:
- `NodeType`: A config parameter used to distinguished the `HoraeDB` and `CompactionNode`. This info would be sent to HoraeMeta through heartbeat.
The compaction service is implemented as grpc service.
### HoraeMeta
`HoraeMeta` manages the compaction nodes cluster with `CompactionNodeManager`, which takes responsibilities for compaction nodes metadata management and scheduling.
The compaction nodes metadata includes:
- Compaction node information, such as node name, node state;
- A compaction node name list, used as the key to access compaction node info, for better scheduling with round-robin strategy;
- ...
As for the compaction nodes scheduling work, it mainly includes:
- Receiving the heartbeats from the compaction node and determining the online status of these registered nodes.
- Performing load balancing according to the compaction nodes cluster info.
- Providing the info of suitable compaction node for HoraeDB when remote compaction execution is needed.
## Load Balancing
Load Balancing is critical for compaction nodes cluster to make their overall processing more efficient. The effect of load balancing mainly based on the schedule algorithm for compaction nodes impl in `CompactionNodeManager`.
_(ps: The current implementation of schedule algorithm is round-robin strategy for easiness.)_
The main process for the schedule algorithm based on real load is:
- HoraeMeta collects the compaction nodes load information through the heartbeats to create a load overview of the compaction nodes cluster.
- Pick a compaction node with low load according to the load overview.
## High Availability
The fault tolerance of above architecture can be achieved by such a procedure:
- When detecting that the heartbeat is broken, `HoraeMeta` determines that the compaction node is offline.
- When `HoraeMeta` can not provide suitable compaction node for HoraeDB or compaction node doesn't return the task result successfully, HoraeDB would switches to run compaction task locally.
---
## File: content/en/docs/design/storage.md
---
title: "Storage"
---
The storage engine mainly provides the following two functions：
1. Persistence of data
2. Under the premise of ensuring the correctness of the data, organize the data in the most reasonable way to meet the query needs of different scenarios.
This document will introduce the internal implementation of the storage engine in HoraeDB. Readers can refer to the content here to explore how to use HoraeDB efficiently.
# Overall Structure
HoraeDB is a distributed storage system based on the share-nothing architecture.
Data between different servers is isolated from each other and does not affect each other. The storage engine in each stand-alone machine is a variant of [log-structured merge-tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree), which is optimized for time-series scenarios. The following figure shows its core components:
## Write Ahead Log (WAL)
A write request will be written to
1. memtable in memory
2. WAL in durable storage
Since memtable is not persisted to the underlying storage system in real time, so WAL is required to ensure the reliability of the data in memtable.
On the other hand, due to the design of the [distributed architecture](cluster.md), WAL itself is required to be highly available. Now there are following implementations in HoraeDB:
- [Local disk](wal_on_rocksdb.md) (based on [RocksDB](http://rocksdb.org/), no distributed high availability)
- [OceanBase](https://www.oceanbase.com)
- [Kafka](wal_on_kafka.md)
## Memtable
Memtable is a memory data structure used to hold recently written table data. Different tables have its corresponding memtable.
Memtable is read-write by default (aka active), and when the write reaches some threshold, it will become read-only and be replaced by a new memtable.
The read-only memtable will be flushed to the underlying storage system in SST format by background thread. After flush is completed, the read-only memtable can be destroyed, and the corresponding data in WAL can also be deleted.
## Sorted String Table（SST）
SST is a persistent format for data, which is stored in the order of primary keys of table. Currently, HoraeDB uses parquet format for this.
For HoraeDB, SST has an important option: segment_duration, only SST within the same segment can be merged, which is benefical for time-series data. And it is also convenient to eliminate expired data.
In addition to storing the original data, the statistical information of the data will also be stored in the SST to speed up the query, such as the maximum value, the minimum value, etc.
## Compactor
Compactor can merge multiple small SST files into one, which is used to solve the problem of too many small files. In addition, Compactor will also delete expired data and duplicate data during the compaction. In future, compaction maybe add more task, such as downsample.
The current compaction strategy in HoraeDB reference Cassandra:
- [SizeTieredCompactionStrategy](https://cassandra.apache.org/doc/latest/cassandra/operating/compaction/stcs.html)
- [TimeWindowCompactionStrategy](https://cassandra.apache.org/doc/latest/cassandra/operating/compaction/twcs.html)
## Manifest
Manifest records metadata of table, SST file, such as: the minimum and maximum timestamps of the data in an SST.
Due to the design of the distributed architecture, the manifest itself is required to be highly available. Now in HoraeDB, there are mainly the following implementations：
- WAL
- ObjectStore
## ObjectStore
ObjectStore is place where data (i.e. SST) is persisted.
Generally speaking major cloud vendors should provide corresponding services, such as Alibaba Cloud's OSS and AWS's S3.