tvm Documentation

1. Project Overview & Quickstart (apache/tvm)

File: README.md

Open Machine Learning Compiler Framework

Documentation |
Contributors |
Community |
Release Notes

Apache TVM is an open machine learning compilation framework,
following the following principles:

Python-first development that enables quick customization of machine learning compiler pipelines.
Universal deployment to bring models into minimum deployable modules.

License

TVM is licensed under the Apache-2.0 license.

Getting Started

Check out the TVM Documentation site for installation instructions, tutorials, examples, and more.
The Getting Started with TVM tutorial is a great
place to start.

Contribute to TVM

TVM adopts the Apache committer model. We aim to create an open-source project maintained and owned by the community.
Check out the Contributor Guide.

History and Acknowledgement

TVM started as a research project for deep learning compilation.
The first version of the project benefited a lot from the following projects:

Halide: Part of TVM's TIR and arithmetic simplification module
originates from Halide. We also learned and adapted some parts of the lowering pipeline from Halide.
Loopy: use of integer set analysis and its loop transformation primitives.
Theano: the design inspiration of symbolic scan operator for recurrence.

Since then, the project has gone through several rounds of redesigns.
The current design is also drastically different from the initial design, following the
development trend of the ML compiler community.

The most recent version focuses on a cross-level design with TensorIR as the tensor-level representation
and Relax as the graph-level representation and Python-first transformations.
The project's current design goal is to make the ML compiler accessible by enabling most
transformations to be customizable in Python and bringing a cross-level representation that can jointly
optimize computational graphs, tensor programs, and libraries. The project is also a foundation
infra for building Python-first vertical compilers for domains, such as LLMs.

File: apps/ios_rpc/README.md

iOS TVM RPC

This folder contains iOS RPC app that allows us to launch an rpc server on a iOS
device. You will need XCode and an iOS device to use this.

Building
- Building TVM runtime and custom DSO loader plugin
- Building iOS TVM RPC application
Workflow
Communication without Wi-Fi and speed up in case of slow Wi-Fi

Building

Building TVM runtime and custom DSO loader plugin

While iOS platform itself doesn't allow us to run an unsigned binary, there is a
partial ability to run JIT code on real iOS devices. While application is
running under debug session, system allows allocating memory with write and
execute permissions (a debugger requirement). So we can use this feature to
implement the tvm.rpc.server.load_module PackedFunc, used to load code over
RPC. For this purpose we use custom version of dlopen function which doesn't
check signature and permissions for module loading. This custom dlopen
mechanic is integrated into TVM RPC as plugin and registered to execution only
inside iOS RPC application.

The custom implementation of dlopen and other functions from dlfcn.h header are placed in separate repository,
and will be downloaded automatically during CMake build for iOS.

Also, it is necessary to build libtvm_runtime.dylib for our iOS device. The
iOS TVM RPC application will be linked with this library.

Run the build using the following commands:

shell

export DEVELOPER_DIR=/Applications/Xcode.app  # iOS SDK is part of Xcode bundle. Have to set it as default Dev Env
cmake ..
  -DCMAKE_BUILD_TYPE=Debug
  -DCMAKE_SYSTEM_NAME=iOS
  -DCMAKE_SYSTEM_VERSION=14.0
  -DCMAKE_OSX_SYSROOT=iphoneos
  -DCMAKE_OSX_ARCHITECTURES=arm64
  -DCMAKE_OSX_DEPLOYMENT_TARGET=14.0
  -DCMAKE_BUILD_WITH_INSTALL_NAME_DIR=ON
  -DUSE_IOS_RPC=ON  # to enable build iOS RPC application from TVM project tree
  -DUSE_METAL=ON    # to enable Metal runtime

cmake --build . --target custom_dso_loader tvm_runtime

Building iOS TVM RPC application

Before start, please run init_proj.py to update XCode developer metadata:

shell

python3 init_proj.py --team_id XXXXXXXXXX --tvm_build_dir "/path/to/tvm/ios/build/folder"

You can get value of your team_id in the following ways:

You have registered Apple Developer Profile. In this case you developer
Team ID available at https://developer.apple.com/account/#/membership
You are using your local developer profile. In this case, leave XXXXXXXXXX
in the command instead of substituting a Team ID. Then open tvmrpc.xcodeproj
by using XCode, click on the project name (tvmrpc) on the left panel. Then
select target tvmrpc. At the bottom of this panel go to Signing & Capabilities tab and in the field Team select your local developer profile
(Your Name (Personal Team)).

On the first run of the application you may see message Could not launch "tvmrpc" in the XCode and message Untrusted Developer on your device. In
this case it will be necessary to check the certificate. Open
Settings -> General -> Device Management -> Apple Development: <your_email> -> Trust "Apple Development: <your_email>" and click Trust. After than you
should rerun your application in the XCode.

After this step, open tvmrpc.xcodeproj by using XCode, build the App and
install the App on the phone.

Workflow

Due to security restriction of iOS10. We cannot upload dynamic libraries to the
App and load it from sandbox. Instead, we need to build a list of libraries,
pack them into the app bundle, launch the RPC server and connect to test the
bundled libraries. For more on the approach we use to work around this
limitation, please take a look into section
Building TVM runtime and custom DSO loader integration.

The test script tests/ios_rpc_test.py and
tests/ios_rpc_mobilenet.py are good templates for
demonstrating the workflow.

We have three different modes for iOS RPC server:

Standalone RPC: In this mode RPC server open port on the device and listening. Then
client connects to the server directly without any mediators.
iOS RPC application with Proxy: RPC server and RPC client communicates through
rpc_proxy. The RPC server on iOS device notify rpc_proxy which was run on
host machine about itself and wait for incoming connections. Communications
between client and server works through rpc_proxy.
iOS RPC application with Tracker: RPC server registered in the rpc_tracker
and client connects to the RPC server through rpc_tracker.

Standalone RPC

Start RPC server on your iOS device:

Push on the Connect button.

After that you supposed to see something like this in the app on the device:

text

IP: <device_ip>
Port: <rpc_server_port>

Printed IP is the IP address of your device and PORT is the number of port
which was open for RPC connection. Next you should use them for connect your RPC
client to the server.

Let's check that direct RPC connection works and we can upload a library with
model and execute it on the device. For this purpose we will use
ios_rpc_test.py. Run it:

shell

python3 tests/ios_rpc_test.py --host <device_ip> --port <rpc_server_port> --mode "standalone"

This will compile TVM IR to shared libraries (CPU and Metal) and run vector
addition on your iOS device. You are supposed to see something like this:

text

Metal: 0.000338692 secs/op
CPU: 0.000219308 secs/op

iOS RPC App with proxy

Start the RPC proxy by running in a terminal:

shell

python3 -m tvm.exec.rpc_proxy --host 0.0.0.0 --port 9090

On success, you should see something like this:

text

INFO:root:RPCProxy: client port bind to 0.0.0.0:9090
INFO:root:RPCProxy: Websock port bind to 8888

Connect your iOS device to the RPC proxy via the iOS TVM RPC application. Set
the Address and Port fields to the address and port of the RPC tracker
respectively. Select mode Proxy and push Connect button. In success the
text on the button will be changed to Disconnect and Disconnected in the top
of the screen will be changed to Connected.
On RPC proxy side you can see the next message in a log:

text

INFO:root:Handler ready TCPSocketProxy::server:iphone

Then we can check that RPC connection works and we can upload a library with
model and execute it on the target device. For this purpose we will use
ios_rpc_test.py. Run it:

shell

python3 tests/ios_rpc_test.py --host <host_ip_address> --port 9090 --mode "proxy"

The output should be the same as it was in previous section.

iOS RPC App with tracker

First start an RPC tracker using

shell

python3 -m tvm.exec.rpc_tracker --host 0.0.0.0 --port 9190

On success, you should see something like this:

text

INFO:RPCTracker:bind to 0.0.0.0:9190

Connect your iOS device to the RPC tracker via the iOS TVM RPC applcation. Set
the Address and Port fields to the address and port of the RPC tracker
respectively. Select mode Tracker and push Connect button. In success the
text on the button will be changed to Disconnect and Disconnected in the top
of the screen will be changed to Connected. On the host side you can check the
connect by the following command:

shell

python3 -m tvm.exec.query_rpc_tracker --port 9190

You are supposed to see something like this:

text

Tracker address 127.0.0.1:9190

Server List
----------------------------
server-address  key
----------------------------
192.168.1.57:9190       server:iphone
----------------------------

Queue Status
------------------------------
key      total  free  pending
------------------------------
iphone   1      1     0
------------------------------

Then we can check that RPC connection works and we can upload a library with
model and execute it on the target device. For this purpose we will use
ios_rpc_test.py. Run it:

shell

python3 tests/ios_rpc_test.py --host <host_ip_address> --port 9190 --mode "tracker"

The output will be the same as in section
Standalone RPC.

Communication without Wi-Fi and speed up in case of slow Wi-Fi

Connection to the RPC server through usbmux can be used then you have slow,
unstable or don't have any Wi-Fi connection. usbmux is used for binding local
TCP port to port on the device and transfer packages between these ports by USB
cable.

First of all you should install usbmux to your system. You can do it with
brew:

shell

brew install usbmuxd

After that you can use iproxy program for binding ports. You can use it for
all described workflows. Let's take a look how it works for
Standalone RPC.

First, start RPC server on your iOS device. You may see something like this in
the app on the device:

text

IP: unknown
Port: <rpc_server_port>

Note. Here IP: unknown because there was no Internet connection on the iOS
device.
Printed Port is the port of the RPC server on your iOS device. We will use it
in binding ports. Run iproxy, specify local port which should be used for
communication with device and the printed port on the device:

shell

iproxy <local_port>:<rpc_server_port>

After this command you should see something like this:

text

Creating listening port <local_port> for device port <rpc_server_port>
waiting for connection

Now we can check that RPC connection through usbmux works and we can upload a
library with model and execute it on the device. For this purpose we will use
ios_rpc_test.py. Run it:

shell

python3 tests/ios_rpc_test.py --host 0.0.0.0 --port <local_port> --mode standalone

The output should be the same as in all previous runs.

File: apps/cpp_rpc/README.md

TVM RPC Server

This folder contains a simple recipe to make RPC server in c++.

Usage (Non-Windows)

Configure the tvm CMake build with config.cmake ensuring that USE_CPP_RPC is set to ON in the config.
If cross compiling for Android, add the following options to the CMake config or specify them when invoking CMake:

text

# Whether to build the C++ RPC server binary
  set(USE_CPP_RPC ON)
  # Path to the Android NDK CMake toolchain
  set(CMAKE_TOOLCHAIN_FILE $ENV{ANDROID_NDK}/build/cmake/android.toolchain.cmake)
  # The Android ABI and platform to target
  set(ANDROID_ABI "arm64-v8a")
  set(ANDROID_PLATFORM android-28)

Similarly, if cross compiling for embedded Linux add the following options to CMake config:

text

# Needed to ensure pthread is linked
  set(OS Linux)
  # Path to the desired C++ cross compiler
  set(CMAKE_CXX_COMPILER /path/to/cross/compiler/executable)

If you need to build cpp_rpc with OpenCL support, specify variable USE_OPENCL in the config:
text
```
set(USE_OPENCL ON)
```
In this case OpenCL-wrapper or OpenCL installed to your system will be used.
When OpenCL-wrapper is used, it will dynamically load OpenCL library on the device.
If the device doesn't have OpenCL library on it, then you'll see in the runtime that OpenCL library cannot be opened.

If linking against a custom device OpenCL library is needed, in the config specify the path to the OpenCL SDK containing the include/CL headers and lib/ or lib64/libOpenCL.so:

text

set(USE_OPENCL /path/to/opencl-sdk)

From within the configured tvm build directory, compile tvm_runtime and the tvm_rpc server:

text

cmake --build $TVM_ROOT/build --target tvm_runtime tvm_rpc -j$(nproc)

Use ./tvm_rpc server to start the RPC server

Usage (Windows)

Configure the tvm CMake build with config.cmake ensuring that USE_CPP_RPC is set to ON in the config.
Install LLVM pre-build binaries, making sure to select the option to add it to the PATH.
Verify Python 3.6 or newer is installed and in the PATH.
Use <tvm_output_dir>\tvm_rpc.exe to start the RPC server

How it works

The tvm runtime dll is linked along with this executable and when the RPC server starts it will load the tvm runtime library.

text

Command line usage
 server       - Start the server
--host        - The listen address of the server, Default=0.0.0.0 (any)
--port        - The port of the RPC server, Default=9090
--port-end    - The end search port of the RPC server, Default=9099
--tracker     - The RPC tracker address in host:port format e.g. 10.1.1.2:9190 Default=""
--key         - The key used to identify the device type in tracker. Default=""
--custom-addr - Custom IP Address to Report to RPC Tracker. Default=""
--silent      - Whether to run in silent mode. Default=False
  Example
  ./tvm_rpc server --host=0.0.0.0 --port=9090 --port-end=9099 --tracker=127.0.0.1:9190 --key=rasp

Note

Currently support is only there for Linux / Android / Windows environment and proxy mode isn't supported currently.

File: apps/android_rpc/README.md

Android TVM RPC

This folder contains Android RPC app that allows us to launch an RPC server on a Android device and connect to it through python script and do testing on the python side as normal TVM RPC.

You will need JDK, Android NDK and an Android device to use this.

Build and Installation

Build APK

We use Gradle to build. Please follow the installation instruction for your operating system.

Before you build the Android application, please refer to TVM4J Installation Guide and install tvm4j-core to your local maven repository. You can find tvm4j dependency declare in app/build.gradle. Modify it if it is necessary.

text

dependencies {
    implementation fileTree(dir: 'libs', include: ['*.jar'])
    androidTestImplementation('androidx.test.espresso:espresso-core:3.1.0', {
        exclude group: 'com.android.support', module: 'support-annotations'
    })
    implementation 'androidx.appcompat:appcompat:1.6.1'
    implementation 'androidx.constraintlayout:constraintlayout:2.1.4'
    implementation 'com.google.android.material:material:1.8.0'
    implementation files('../../../jvm/core/target/tvm4j-core-0.0.1-SNAPSHOT.jar')
    testImplementation 'junit:junit:4.13.2'
}

Now use Gradle to compile JNI, resolve Java dependencies and build the Android application together with tvm4j. Run following script to generate the apk file.

bash

export ANDROID_HOME=[Path to your Android SDK, e.g., ~/Android/sdk]
cd apps/android_rpc
gradle clean build

In app/build/outputs/apk you'll find app-release-unsigned.apk, use dev_tools/gen_keystore.sh to generate a signature and use dev_tools/sign_apk.sh to get the signed apk file app/build/outputs/apk/release/tvmrpc-release.apk.

Upload tvmrpc-release.apk to your Android device and install it:

bash

$ANDROID_HOME/platform-tools/adb install app/build/outputs/apk/release/tvmrpc-release.apk

If you see error:

text

adb: failed to install app/build/outputs/apk/release/tvmrpc-release.apk:
  Failure [INSTALL_FAILED_UPDATE_INCOMPATIBLE:
  Package org.apache.tvm.tvmrpc signatures do not match the previously installed version; ignoring!]

Run uninstall first:

bash

$ANDROID_HOME/platform-tools/adb uninstall org.apache.tvm.tvmrpc

Build with OpenCL

Application is building with OpenCL support by default.
OpenCL-wrapper is used and will dynamically load OpenCL library on the device.
If the device doesn't have OpenCL library on it, then you'll see in the runtime that OpenCL library cannot be opened.
If you want to build this application without OpenCL then set USE_OPENCL = 0
in config.mk

Cross Compile and Run on Android Devices

Architecture and Android Standalone Toolchain

In order to cross compile a shared library (.so) for your android device, you have to know the target triple for the device. (Refer to Cross-compilation using Clang for more information). Run adb shell cat /proc/cpuinfo to list the device's CPU information.

Now use NDK to generate standalone toolchain for your device. For my test device, I use following command.

bash

cd /opt/android-ndk/build/tools/
./make-standalone-toolchain.sh --platform=android-24 --use-llvm --arch=arm64 --install-dir=/opt/android-toolchain-arm64

If everything goes well, you will find compile tools in /opt/android-toolchain-arm64/bin. For example, bin/aarch64-linux-android-g++ can be used to compile C++ source codes and create shared libraries for arm64 Android devices.

Cross Compile and Upload to the Android Device

First start an RPC tracker using

python -m tvm.exec.rpc_tracker --port [PORT]

and connect your Android device to this RPC tracker via the TVM RPC application. Open the app,
set the Address and Port fields to the address and port of the RPC tracker respectively.
The key should be set to "android" if you wish to avoid modifying the default test script.

After pushing "START RPC" button on the app, you can check the connect by run

python -m tvm.exec.query_rpc_tracker --port [PORT]

on your host machine.
You are supposed to find a free "android" in the queue status.

text

...

Queue Status
-------------------------------
key       total  free  pending
-------------------------------
android   1      1     0
-------------------------------

Then checkout android_rpc/tests/android_rpc_test.py and run,

bash

# Specify the RPC tracker
export TVM_TRACKER_HOST=0.0.0.0
export TVM_TRACKER_PORT=[PORT]
# Specify the standalone Android C++ compiler
export TVM_NDK_CC=/opt/android-toolchain-arm64/bin/aarch64-linux-android-g++
python android_rpc_test.py

This will compile TVM IR to shared libraries (CPU, OpenCL and Vulkan) and run vector addition on your Android device. To verify compiled TVM IR shared libraries on OpenCL target set 'test_opencl = True' and on Vulkan target set 'test_vulkan = True' in tests/android_rpc_test.py, by default on CPU target will execute.
On my test device, it gives following results.

bash

Run CPU test ...
0.000962932 secs/op

Run GPU(OpenCL Flavor) test ...
0.000155807 secs/op

[23:29:34] /home/tvm/src/runtime/vulkan/vulkan_device_api.cc:674: Cannot initialize vulkan: [23:29:34] /home/tvm/src/runtime/vulkan/vulkan_device_api.cc:512: Check failed: __e == VK_SUCCESS Vulan Error, code=-9: VK_ERROR_INCOMPATIBLE_DRIVER

Stack trace returned 10 entries:
[bt] (0) /home/user/.local/lib/python3.6/site-packages/tvm-0.4.0-py3.6-linux-x86_64.egg/tvm/libtvm.so(dmlc::StackTrace[abi:cxx11]()+0x53) [0x7f477f5399f3]
.........

You can still compile vulkan module but cannot run locally
Run GPU(Vulkan Flavor) test ...
0.000225198 secs/op

You can define your own TVM operators and test via this RPC app on your Android device to find the most optimized TVM schedule.

Troubleshooting

If you build the application in Android Studio and see error similar to this one:

text

A problem occurred evaluating project ':app'.
> Failed to apply plugin 'com.android.internal.version-check'.
   > Minimum supported Gradle version is 7.5. Current version is 7.4. If using the gradle wrapper, try editing the distributionUrl in /Users/echuraev/Workspace/OctoML/tvm_android_test/apps/android_deploy/gradle/wrapper/gradle-wrapper.properties to gradle-7.5-all.zip

Run project syncing File -> Sync Project with Gradle Files. It should sync the
project and create gradle-wrapper files.

File: ci/jenkins/README.md

TVM CI

TVM runs CI jobs on every commit to an open pull request and to branches in the apache/tvm repo (such as main). These jobs are essential to keeping the TVM project in a healthy state and preventing breakages.

Jenkins

Jenkins runs all of the linux-based TVM CI-enabled regression tests. This includes tests against accelerated hardware such as GPUs. It excludes those regression tests that run against hardware not available in the cloud (those tests aren't currently exercised in TVM CI). The tests run by Jenkins represent most of the merge-blocking tests (and passing Jenkins should mostly correlate with passing the remaining Windows/Mac builds).

GitHub Actions

GitHub Actions is used to run Windows jobs, MacOS jobs, and various on-GitHub automations. These are defined in .github/workflows. These automations include bots to:

allow non-committers to merge approved / CI passing PRs

https://github.com/apache/tvm/actions has the logs for each of these workflows. Note that when debugging these workflows changes from PRs from forked repositories won't be reflected in the PR. These should be tested in the forked repository first and linked in the PR body.

Jenkins CI

TVM uses Jenkins for running Linux continuous integration (CI) tests on
branches and
pull requests through a
build configuration specified in a Jenkinsfile.
Other jobs run in GitHub Actions for Windows and MacOS jobs.

`Jenkinsfile`

The template files in this directory are used to generate the Jenkinsfile used by Jenkins to run CI jobs for each commit to PRs and branches.

To regenerate the Jenkinsfile, run

bash

python3 -mvenv _venv
_venv/bin/pip3 install -r ci/jenkins/requirements.txt
_venv/bin/python3 ci/jenkins/generate.py

Infrastructure

While all TVM tests are contained within the apache/tvm repository, the infrastructure used to run the tests is donated by the TVM Community. To encourage collaboration, the configuration for TVM's CI infrastructure is stored in a public GitHub repository. TVM community members are encouraged to contribute improvements. The configuration, along with documentation of TVM's CI infrastructure, is in the tlc-pack/ci repo.

File: ci/README.md

Apache TVM Continuous Integration (CI)

Overview

TVM's Continuous Integration is responsible for verifying the code in apache/tvm and testing PRs
before they merge to inform TVM contributors and committers. These jobs are essential to keeping the
TVM project in a healthy state and preventing breakages. CI in TVM is broken into these pieces:

Lint scripts in tests/lint.
The tests themselves, all of which live underneath tests.
Definitions of test suites, with each suite defined as a separate task_ script in
tests/scripts.
Scripts and automation ci/scripts.
The linux test sequence (in Jenkinsfile), which lints and builds TVM and runs test
suites using Docker on Linux.
The Windows and Mac test sequences (in .github/actions).
GitHub Actions that support the code review process (in .github/actions).
Tools to reproduce the CI locally (in tests/scripts).
Infrastructure-as-Code that configures the cloud services that provide Jenkins for the TVM CI (in the
tlc-pack/ci repo).

CI Documentation Index

The CI documentation belongs with the implementation it describes. To make that concrete, the
documentation is split like so:

An overview of the CI is in this file.
User-facing documentation lives in apache/tvm's docs/contribute sub-directory and is served on the
TVM docs site.
Documentation of the tools that run TVM's various regression tests locally and the test suites
are in this sub-directory.
Documentation of the cloud services and their configuration lives in the
tlc-pack/ci repo.

Jenkins

Jenkins runs all of the Linux-based TVM CI-enabled regression tests. This includes tests against accelerated hardware such as GPUs. It excludes those regression tests that run against hardware not available in the cloud (those tests aren't currently exercised in TVM CI). The tests run by Jenkins represent most of the merge-blocking tests (and passing Jenkins should mostly correlate with passing the remaining Windows/Mac builds).

GitHub Actions

GitHub Actions is used to run Windows jobs, MacOS jobs, and various on-GitHub automations. These are defined in .github/workflows. These automations include bots to:

allow non-committers to merge approved / CI passing PRs

Docker Images

Each CI job runs most of its work inside a Docker container, built from files
in the docker/ folder. These
files are built nightly in Jenkins via the tvm-docker job.
The images for these containers are hosted in the tlcpack Docker Hub
and referenced in the jenkins/templates. These can be inspected and run
locally via standard Docker commands.

`ci-docker-staging`

The ci-docker-staging
branch is used to test updates to Docker images and Jenkinsfile changes. When
running a build for a normal PR from a forked repository, Jenkins uses the code
from the PR except for the Jenkinsfile itself, which comes from the base branch.
When branches are built, the Jenkinsfile in the branch is used, so a committer
with write access must push PRs to a branch in apache/tvm to properly test
Jenkinsfile changes. If your PR makes changes to the Jenkinsfile, make sure
to @ a committer
and ask them to push your PR as a branch to test the changes.

Jenkins CI

`Jenkinsfile`

The template files in this directory are used to generate the Jenkinsfile used by Jenkins to run CI jobs for each commit to PRs and branches.

To regenerate the Jenkinsfile, run make in the ci/jenkins dir.

2. Official Technical Reference & Guides (apache/docs)

File: README.md

Apache HoraeDB (incubating) Website

[!IMPORTANT]
Apache HoraeDB (incubating) is an effort undergoing incubation at the Apache
Software Foundation (ASF), sponsored by the Apache Incubator PMC.

Please read the DISCLAIMER and a full explanation of "incubating".

This repository stores all the source files of HoraeDB website, it's built with hugo and docsy.

How to run

bash

# Start a local server for preview
hugo serve

By default hugo will listen on http://localhost:1313

Contributing

Welcome any contributions from the community. You can

Open an issue with any suggestions, or
Submit a pull request to improve the website (trivial fixes are welcome).

File: content/en/blog/2024/release-2.0.0.md

title: Release 2.0.0 date: 2024-04-23

Upgrade from 1.x.x to 2.0.0

The transition from CeresDB to Apache HoraeDB introduces several breaking changes. To facilitate upgrading from older versions to v2.0.0, specific alterations are necessary.

Upgrade Steps

Setup required envs

text

export HORAEDB_DEFAULT_CATALOG=ceresdb

Update config

Etcd's root should be configured both in horaedb and horaemeta

For horaedb

text

[cluster_deployment.etcd_client]
server_addrs = ['127.0.0.1:2379']
root_path = "/rootPath"

For horaemeta

text

storage-root-path = "/rootPath"

Upgrade horaemeta

Horaedb will throw following errors, which is expected

text

2024-01-23 14:37:57.726 ERRO [src/cluster/src/cluster_impl.rs:136] Send heartbeat to meta failed, err:Failed to send heartbeat, cluster:defaultCluster, err:status: Unimplemented, message: "unknown service meta_service.MetaRpcService", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }

Upgrade horaedb

After all server upgraded, the cluster should be ready for read/write, and old data could be queried like before.

What's Changed

Breaking Changes

refactor!: refactor shard version logic by @ZuLiangWang in https://github.com/apache/horaedb/pull/1286

Features

feat: support re-acquire shard lock in a fast way by @ShiKaiWi in https://github.com/apache/horaedb/pull/1251
feat: support alter partition table by @chunshao90 in https://github.com/apache/horaedb/pull/1244
feat: support access etcd with tls by @ShiKaiWi in https://github.com/apache/horaedb/pull/1254
feat: support schema validate in remote write by @ShiKaiWi in https://github.com/apache/horaedb/pull/1256
feat: avoid flush when drop table by @jiacai2050 in https://github.com/apache/horaedb/pull/1257
feat: opentsdb api support gzip body by @tanruixiang in https://github.com/apache/horaedb/pull/1261
feat: infer timestamp constraint for single-timestamp column by @Dennis40816 in https://github.com/apache/horaedb/pull/1266
feat: primary keys support sample by @jiacai2050 in https://github.com/apache/horaedb/pull/1243
feat: cache space total memory by @jiacai2050 in https://github.com/apache/horaedb/pull/1278
feat: skip record column values for level0 sst by @jiacai2050 in https://github.com/apache/horaedb/pull/1282
feat: support write wal logs in columnar format by @ShiKaiWi in https://github.com/apache/horaedb/pull/1179
feat: support stack size of read threads configurable by @ShiKaiWi in https://github.com/apache/horaedb/pull/1305
feat: impl DoNothing wal by @jiacai2050 in https://github.com/apache/horaedb/pull/1311
feat: slow log include remote query by @jiacai2050 in https://github.com/apache/horaedb/pull/1316
feat: use string for request id by @jiacai2050 in https://github.com/apache/horaedb/pull/1349
feat: support metrics for number of bytes fetched from object storage by @ShiKaiWi in https://github.com/apache/horaedb/pull/1363
feat: avoid building dictionary for massive unique column values by @ShiKaiWi in https://github.com/apache/horaedb/pull/1365
feat: utilize the column cardinality for deciding whether to do dict by @ShiKaiWi in https://github.com/apache/horaedb/pull/1372
feat: avoid pulling unnecessary columns when querying append mode table by @Rachelint in https://github.com/apache/horaedb/pull/1307
feat: dist sql analyze by @baojinri in https://github.com/apache/horaedb/pull/1260
feat: impl priority runtime for read by @jiacai2050 in https://github.com/apache/horaedb/pull/1303
feat: upgrade horaedbproto by @chunshao90 in https://github.com/apache/horaedb/pull/1408
feat: block rules support query by @jiacai2050 in https://github.com/apache/horaedb/pull/1420
feat: try load page indexes by @jiacai2050 in https://github.com/apache/horaedb/pull/1425
feat: support setting meta_addr&etcd_addrs by env by @chunshao90 in https://github.com/apache/horaedb/pull/1427
feat: add table status check by @ZuLiangWang in https://github.com/apache/horaedb/pull/1418
feat: support docker-compose and update README by @chunshao90 in https://github.com/apache/horaedb/pull/1429
feat: impl layered memtable to reduce duplicated encode during scan by @Rachelint in https://github.com/apache/horaedb/pull/1271
feat: update disk cache in another thread to avoid blocking normal query process by @jiacai2050 in https://github.com/apache/horaedb/pull/1431
feat: update pgwire to 0.19 by @sunng87 in https://github.com/apache/horaedb/pull/1436
feat: filter out MySQL federated components' emitted statements by @chunshao90 in https://github.com/apache/horaedb/pull/1439
feat: add system_stats lib to collect system stats by @ShiKaiWi in https://github.com/apache/horaedb/pull/1442
feat(horaectl): initial commit by @chunshao90 in https://github.com/apache/horaedb/pull/1450
feat: support collect statistics about the engine by @ShiKaiWi in https://github.com/apache/horaedb/pull/1451
feat: persist sst meta size by @jiacai2050 in https://github.com/apache/horaedb/pull/1440
feat: add sst level config for benchmark by @zealchen in https://github.com/apache/horaedb/pull/1482
feat: add exponential backoff when retry by @zealchen in https://github.com/apache/horaedb/pull/1486

Refactor

refactor: move wal structs and traits to wal crate by @tisonkun in https://github.com/apache/horaedb/pull/1263
refactor: improve error readability by @jiacai2050 in https://github.com/apache/horaedb/pull/1265
refactor: move wal crate to under src folder by @tisonkun in https://github.com/apache/horaedb/pull/1270
refactor: use notifier::RequestNotifiers instead of dedup_requests::RequestNotifiers by @baojinri in https://github.com/apache/horaedb/pull/1249
refactor: conditionally compile wal impls by @tisonkun in https://github.com/apache/horaedb/pull/1272
refactor: remove unused min/max timestamp in the RowGroup by @ShiKaiWi in https://github.com/apache/horaedb/pull/1297
refactor: avoid duplicate codes by @ShiKaiWi in https://github.com/apache/horaedb/pull/1371
refactor: avoid returning metrics in non-analyze sql by @baojinri in https://github.com/apache/horaedb/pull/1410
refactor: move sub crates to the src directory by @chunshao90 in https://github.com/apache/horaedb/pull/1443
refactor: adjust cpu's stats by @ShiKaiWi in https://github.com/apache/horaedb/pull/1457
refactor: refactor compaction process for remote compaction by @Rachelint in https://github.com/apache/horaedb/pull/1476

Fixed

fix: dist query dedup by @Rachelint in https://github.com/apache/horaedb/pull/1269
fix: log third party crates by @jiacai2050 in https://github.com/apache/horaedb/pull/1289
fix: ensure primary key order by @jiacai2050 in https://github.com/apache/horaedb/pull/1292
fix: use flag in preflush to indicate whether reorder is required by @jiacai2050 in https://github.com/apache/horaedb/pull/1298
fix: alter partition table tag column by @chunshao90 in https://github.com/apache/horaedb/pull/1304
fix: increase wait duration for flush by @jiacai2050 in https://github.com/apache/horaedb/pull/1315
fix: add license to workspace members by @jiacai2050 in https://github.com/apache/horaedb/pull/1317
fix: ensure channel size non zero by @jiacai2050 in https://github.com/apache/horaedb/pull/1345
fix: fix create table result by @ZuLiangWang in https://github.com/apache/horaedb/pull/1354
Revert "fix: fix create table result" by @ZuLiangWang in https://github.com/apache/horaedb/pull/1355
fix: fix test create table result by @ZuLiangWang in https://github.com/apache/horaedb/pull/1357
fix: no write stall by @ShiKaiWi in https://github.com/apache/horaedb/pull/1388
fix: collect metrics for get_ranges by @ShiKaiWi in https://github.com/apache/horaedb/pull/1364
fix: ignore collecting fetched bytes stats when sst file is read only once by @ShiKaiWi in https://github.com/apache/horaedb/pull/1369
fix: publich nightly image by @chunshao90 in https://github.com/apache/horaedb/pull/1396
fix: missing and verbose logs by @ShiKaiWi in https://github.com/apache/horaedb/pull/1398
fix: fix broken link by @caicancai in https://github.com/apache/horaedb/pull/1399
fix: the broken link about the issue status by @ShiKaiWi in https://github.com/apache/horaedb/pull/1402
fix: skip wal encoding when data wal is disabled by @jiacai2050 in https://github.com/apache/horaedb/pull/1401
fix: disable percentile for distributed tables by @jiacai2050 in https://github.com/apache/horaedb/pull/1406
fix: compatible for old table options by @Rachelint in https://github.com/apache/horaedb/pull/1432
fix: get_ranges is not spawned in io-runtime by @ShiKaiWi in https://github.com/apache/horaedb/pull/1426
fix: table name is normalized when find timestamp column by @jiacai2050 in https://github.com/apache/horaedb/pull/1446
fix: changes required for migrate dev to main by @jiacai2050 in https://github.com/apache/horaedb/pull/1455
fix: missing filter index over the primary keys by @ShiKaiWi in https://github.com/apache/horaedb/pull/1456
fix: random failure of test_collect_system_stats by @ShiKaiWi in https://github.com/apache/horaedb/pull/1459
fix(ci): refactor ci trigger conditions by @jiacai2050 in https://github.com/apache/horaedb/pull/1474

Docs

chore(docs): rename CeresDB to HoraeDB by @caicancai in https://github.com/apache/horaedb/pull/1337
chore/docs: remove broken link by @caicancai in https://github.com/apache/horaedb/pull/1341
docs: update CONTRIBUTING.md by @suyanhanx in https://github.com/apache/horaedb/pull/1382
doc: fix broken link by @caicancai in https://github.com/apache/horaedb/pull/1358
chore/doc: rename ceresdb to horaedb by @caicancai in https://github.com/apache/horaedb/pull/1332
doc: add MySQL-Client in README by @jackwener in https://github.com/apache/horaedb/pull/1331
doc: fix link in illegal markdown format by @jackwener in https://github.com/apache/horaedb/pull/1334
style: normalize comments/doc in rustfmt by @jackwener in https://github.com/apache/horaedb/pull/1335
docs: add sudo for install commands by @caicancai in https://github.com/apache/horaedb/pull/1347
docs: sync GH activities to commits only by @tisonkun in https://github.com/apache/horaedb/pull/1385
chore(docs): fix invalid repo links by @SYaoJun in https://github.com/apache/horaedb/pull/1452
chore(docs): fix invalid repo links by @Apricity001 in https://github.com/apache/horaedb/pull/1472

Chore

chore(deps): bump golang.org/x/net from 0.5.0 to 0.17.0 in /integration_tests/sdk/go by @dependabot in https://github.com/apache/horaedb/pull/1258
chore: delete the configuration related to github cache by @tanruixiang in https://github.com/apache/horaedb/pull/1259
chore: remove backtrace of blocked table by @chunshao90 in https://github.com/apache/horaedb/pull/1267
ci: setup golang in CI by @tisonkun in https://github.com/apache/horaedb/pull/1275
chore: remove default features in analytic_engine by @jiacai2050 in https://github.com/apache/horaedb/pull/1277
chore(deps): bump google.golang.org/grpc from 1.53.0 to 1.56.3 in /integration_tests/sdk/go by @dependabot in https://github.com/apache/horaedb/pull/1280
test: simplify ceresmeta-server installation by @tisonkun in https://github.com/apache/horaedb/pull/1287
chore: enable blank issue by @ShiKaiWi in https://github.com/apache/horaedb/pull/1290
chore: add metrics to inspect write path by @Rachelint in https://github.com/apache/horaedb/pull/1264
chore: refactor build_meta.sh in integration-test by @chunshao90 in https://github.com/apache/horaedb/pull/1306
chore: rename ceresdb to horaedb by @chunshao90 in https://github.com/apache/horaedb/pull/1310
edit: add schema id, schema name, catalog name in TableData by @dust1 in https://github.com/apache/horaedb/pull/1294
chore: ignore seq check for DoNothing wal by @jiacai2050 in https://github.com/apache/horaedb/pull/1314
chore: remove community by @jiacai2050 in https://github.com/apache/horaedb/pull/1318
chore: try to clear ceresdb stuff by @tisonkun in https://github.com/apache/horaedb/pull/1320
chore: change copyright owner by @tisonkun in https://github.com/apache/horaedb/pull/1321
ci: stop release docker image before we finish the rename and transfer by @tisonkun in https://github.com/apache/horaedb/pull/1323
chore: bump deps by @jiacai2050 in https://github.com/apache/horaedb/pull/1325
chore: rename ceresmeta to horaemeta by @chunshao90 in https://github.com/apache/horaedb/pull/1327
chore: rename binary to horaedb-server and more by @tisonkun in https://github.com/apache/horaedb/pull/1330
chore(license): rename license-header.txt's CeresDB to HoraeDB by @caicancai in https://github.com/apache/horaedb/pull/1336
chore: replace ceresdb with horaedb by @jackwener in https://github.com/apache/horaedb/pull/1338
chore: more rename to horaedb by @tisonkun in https://github.com/apache/horaedb/pull/1340
chore: update create table integration test result by @ZuLiangWang in https://github.com/apache/horaedb/pull/1344
chore: disable frequently failed tests by @jiacai2050 in https://github.com/apache/horaedb/pull/1352
test: add integration test for alter table options by @caicancai in https://github.com/apache/horaedb/pull/1346
chore: ignore flush failure when flush by @ShiKaiWi in https://github.com/apache/horaedb/pull/1362
chore: disable timeout for http api by @jiacai2050 in https://github.com/apache/horaedb/pull/1367
chore: disable block for http api by @jiacai2050 in https://github.com/apache/horaedb/pull/1368
config: add .asf.yaml by @chunshao90 in https://github.com/apache/horaedb/pull/1377
ci: remove missing Required status by @tisonkun in https://github.com/apache/horaedb/pull/1383
chore: git repo link type fix by @fengmk2 in https://github.com/apache/horaedb/pull/1378
chore: apply ASF license header by @tanruixiang in https://github.com/apache/horaedb/pull/1384
chore: add dev mail list and rename ceresdb to horaedb by @tanruixiang in https://github.com/apache/horaedb/pull/1375
chore: more rename to horaedb by @chunshao90 in https://github.com/apache/horaedb/pull/1387
chore: add push-nightly-image in workflow by @chunshao90 in https://github.com/apache/horaedb/pull/1389
chore: update README by @chunshao90 in https://github.com/apache/horaedb/pull/1390
chore: refactor for better readability by @jiacai2050 in https://github.com/apache/horaedb/pull/1400
chore: add error log for remote server by @jiacai2050 in https://github.com/apache/horaedb/pull/1407
chore: update website url by @chunshao90 in https://github.com/apache/horaedb/pull/1404
chore: upload horaedb logo by @chunshao90 in https://github.com/apache/horaedb/pull/1409
chore: add slack link by @tanruixiang in https://github.com/apache/horaedb/pull/1395
chore: update logo by @chunshao90 in https://github.com/apache/horaedb/pull/1414
chore: update horaedb logo by @chunshao90 in https://github.com/apache/horaedb/pull/1415
chore: rename ceresformat to logformat by @ZuLiangWang in https://github.com/apache/horaedb/pull/1417
chore: fix logo link in readme by @chunshao90 in https://github.com/apache/horaedb/pull/1416
chore: update github pages by @chunshao90 in https://github.com/apache/horaedb/pull/1421
chore: more rename to horaedb by @chunshao90 in https://github.com/apache/horaedb/pull/1419
chore: fix error message by @jiacai2050 in https://github.com/apache/horaedb/pull/1412
chore: remove github pages in asf.yaml by @chunshao90 in https://github.com/apache/horaedb/pull/1428
chore: skip wal seq check when wal is disabled by @jiacai2050 in https://github.com/apache/horaedb/pull/1430
chore: enable merge on github by @ShiKaiWi in https://github.com/apache/horaedb/pull/1435
chore: merge change sets on the dev branch by @ShiKaiWi in https://github.com/apache/horaedb/pull/1423
chore: fix issue status of README-CN.md by @ShiKaiWi in https://github.com/apache/horaedb/pull/1437
chore(deps): bump h2 from 0.3.17 to 0.3.24 by @dependabot in https://github.com/apache/horaedb/pull/1448
chore(deps): bump shlex from 1.1.0 to 1.3.0 by @dependabot in https://github.com/apache/horaedb/pull/1458
chore: update create tables result by @ZuLiangWang in https://github.com/apache/horaedb/pull/1454
chore: merge HoraeMeta code into HoreaDB repository by @ZuLiangWang in https://github.com/apache/horaedb/pull/1460
chore(deps): bump google.golang.org/grpc from 1.47.0 to 1.56.3 in /horaemeta by @dependabot in https://github.com/apache/horaedb/pull/1464
chore(deps): bump golang.org/x/net from 0.16.0 to 0.17.0 in /horaemeta by @dependabot in https://github.com/apache/horaedb/pull/1465
chore(deps): bump golang.org/x/crypto from 0.14.0 to 0.17.0 in /horaemeta by @dependabot in https://github.com/apache/horaedb/pull/1462
chore: rename ci's prefix name by @tanruixiang in https://github.com/apache/horaedb/pull/1467
chore: fix github issue template by @ZuLiangWang in https://github.com/apache/horaedb/pull/1470
chore(horaemeta&horaectl): refactor clusters/diagnose response body by @chunshao90 in https://github.com/apache/horaedb/pull/1475
chore: free disk for ci by @jiacai2050 in https://github.com/apache/horaedb/pull/1484
deps: bump datafusion by @tanruixiang in https://github.com/apache/horaedb/pull/1445
horaectl: remove go implementation of horaectl by @chunshao90 in https://github.com/apache/horaedb/pull/1490
chore: update version to 2.0.0, prepare for releasing v2.0.0 by @chunshao90 in https://github.com/apache/horaedb/pull/1487

New Contributors

@Dennis40816 made their first contribution in https://github.com/apache/horaedb/pull/1266
@caicancai made their first contribution in https://github.com/apache/horaedb/pull/1332
@jackwener made their first contribution in https://github.com/apache/horaedb/pull/1331
@suyanhanx made their first contribution in https://github.com/apache/horaedb/pull/1382
@fengmk2 made their first contribution in https://github.com/apache/horaedb/pull/1378
@sunng87 made their first contribution in https://github.com/apache/horaedb/pull/1436
@SYaoJun made their first contribution in https://github.com/apache/horaedb/pull/1452
@Apricity001 made their first contribution in https://github.com/apache/horaedb/pull/1472

Full Changelog: https://github.com/apache/horaedb/compare/v1.2.7...v2.0.0

File: content/en/blog/2024/release-2.1.0.md

title: Release 2.1.0
date: 2024-11-18
tags:

release

The Apache HoraeDB(incubating) team are pleased to announce that v2.1.0 is released, which has closed over 60 issues, including two major features:

1. New WAL implementation based on local disk.

In previous version, there is a RocksDB-based WAL. Although it works well in most cases, it has following issues:

Compiling from source can be a challenging task, especially since RocksDB is primarily written in C++.
For WAL, RocksDB can be somewhat overkill. If you are not familiar with RocksDB, tuning it can be very challenging.

With this new WAL, the above two problems are solved very well, and in performance test, the new WAL slightly outperforms the previous implementation, giving a solid foundation for future optimizations.

Interested readers can refer to the design documentation [here]({{< ref "wal_on_disk" >}}) for more details on this feature.

How to enable

text

[analytic.wal]
type = "Local"
data_dir = "/path/to/local/wal"

2. Access object store with [Apache OpenDAL](https://github.com/apache/opendal)

OpenDAL (Open Data Access Layer) is a project that provides a unified API for accessing various data storage backends.
It offers several advantages for developers and organizations. Here are some key benefits:

Unified API. OpenDAL provides a consistent and unified API for accessing different storage backends, such as AWS S3, Azure Blob Storage, and local file systems.
Optimized for Efficiency: OpenDAL is built with performance in mind. It includes optimizations to ensure efficient data access and manipulation, making it suitable for high-performance applications.
Comprehensive Documentation: The project provides detailed documentation, making it easier for developers to get started and understand how to use the library effectively.

In newer versions of OpenDAL, object_store integration is provided, which is very beneficial for HoraeDB code migration, as the APIs used by the upper layers remain virtually unchanged, and only the object store part needs to be abstracted to a unified OpenDAL operator:

rust

// Create a new operator
let operator = Operator::new(S3::default())?.finish();

// Create a new object store
let object_store = Arc::new(OpendalStore::new(operator));

Additionally, since the Apache OpenDAL implementation of object_store is based on the latest version of the object_store, which has breaking changes from the previous version that HoraeDB is using, we've chosen to make it compatible in order to keep the scope of this upgrade as manageable as possible.

In the process of adapting to the new API, the put_multipart interface has changed the most, so the main adaptation logic is also here, HoraeDB's approach is to encapsulate the underlying put_multipart interface to ensure that the upper layer code is not modified, the details can be found in the reference:

https://github.com/apache/horaedb/blob/v2.1.0/src/components/object_store/src/multi_part.rs

Note: The adaptation logic is only practical when parquet version < 52.0.0.

Download

Go to download pages.

Conclusion

Other bug fixes and improvements can be seen here:

https://github.com/apache/horaedb/releases/tag/v2.1.0

As always, we warmly welcome you to join our community and share your insights.

File: content/en/blog/_index.md

title: "Blog"
weight: 1
simple_list: true
menu:
main:
weight: 30
pre:

File: content/en/community/_index.md

title: "Community"
params:
contributingUrl: https://github.com/apache/horaedb/blob/main/CONTRIBUTING.md
menu:
main:
weight: 40

Mailing lists are a form of communication used by the Apache community. Generally speaking, many things in the Apache community are hosted by mailing lists, For example: project q&A, technical discussion, transaction decision, release vote, etc，By subscribing, you can get the first update of the HoraeDB community and keep up with the community. The following list is the main list we are using:

[email protected], Community activity information
[email protected], Code repo update information

In order to subscribe to the [email protected] mailing list, the steps are as follows：

Send an email without any content or subject: [email protected]
Wait until you receive an email with the subject line confirm subscribe to [email protected] (if you have not received it for a long time, please confirm whether the email is blocked by your email, if you have not been blocked and will receive a reply for a long time, return to step 1)
Reply directly to the email without changing the subject line or adding the email content.
Wait until you receive an email with the subject line WELCOME to [email protected] .
If you receive an email from dev, you have successfully subscribed to the email. To initiate a discussion, you can send an email directly to [email protected], which will be sent to everyone who subscribed to the mailing list.

Unsubscribe from the mail list

The steps for unsubscribing to a mailing list are similar to those for subscribing to a mailing list:

Send an email without any content or subject to: [email protected]
Wait until you receive an email with the subject line confirm unsubscribe from [email protected]
Reply directly to the email without changing the subject line or adding the email content
Wait until you receive an email with the subject line GOODBYE from [email protected]
Unsubscribe success

File: content/en/docs/design/architecture.md

title: "Introduction to HoraeDB's Architecture"

Target

Provide the overview of HoraeDB to the developers who want to know more about HoraeDB but have no idea where to start.
Make a brief introduction to the important modules of HoraeDB and the connections between these modules but details about their implementations are not be involved.

Motivation

HoraeDB is a timeseries database (TSDB). However, HoraeDB's goal is to handle both timeseries and analytic workloads compared with the classic TSDB, which usually have a poor performance in handling analytic workloads.

In the classic timeseries database, the Tag columns (InfluxDB calls them Tag and Prometheus calls them Label) are normally indexed by generating an inverted index. However, it is found that the cardinality of Tag varies in different scenarios. And in some scenarios the cardinality of Tag is very high (we name this case after analytic workload), and it takes a very high cost to store and retrieve the inverted index. On the other hand, it is observed that scanning+pruning often used by the analytical databases can do a good job to handle such analytic workload.

The basic design idea of HoraeDB is to adopt a hybrid storage format and the corresponding query method for a better performance in processing both timeseries and analytic workloads.

Architecture

plaintext

┌──────────────────────────────────────────┐
│       RPC Layer (HTTP/gRPC/MySQL)        │
└──────────────────────────────────────────┘
┌──────────────────────────────────────────┐
│                 SQL Layer                │
│ ┌─────────────────┐  ┌─────────────────┐ │
│ │     Parser      │  │     Planner     │ │
│ └─────────────────┘  └─────────────────┘ │
└──────────────────────────────────────────┘
┌───────────────────┐  ┌───────────────────┐
│    Interpreter    │  │      Catalog      │
└───────────────────┘  └───────────────────┘
┌──────────────────────────────────────────┐
│               Query Engine               │
│ ┌─────────────────┐  ┌─────────────────┐ │
│ │    Optimizer    │  │    Executor     │ │
│ └─────────────────┘  └─────────────────┘ │
└──────────────────────────────────────────┘
┌──────────────────────────────────────────┐
│         Pluggable Table Engine           │
│  ┌────────────────────────────────────┐  │
│  │              Analytic              │  │
│  │┌────────────────┐┌────────────────┐│  │
│  ││      Wal       ││    Memtable    ││  │
│  │└────────────────┘└────────────────┘│  │
│  │┌────────────────┐┌────────────────┐│  │
│  ││     Flush      ││   Compaction   ││  │
│  │└────────────────┘└────────────────┘│  │
│  │┌────────────────┐┌────────────────┐│  │
│  ││    Manifest    ││  Object Store  ││  │
│  │└────────────────┘└────────────────┘│  │
│  └────────────────────────────────────┘  │
│  ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           Another Table Engine        │  │
│  └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
└──────────────────────────────────────────┘

The figure above shows the architecture of HoraeDB stand-alone service and the details of some important modules will be described in the following part.

RPC Layer

module path: https://github.com/apache/incubator-horaedb/tree/main/server

The current RPC supports multiple protocols including HTTP, gRPC, MySQL.

Basically, HTTP and MySQL are used to debug HoraeDB, query manually and perform DDL operations (such as creating, deleting tables, etc.). And gRPC protocol can be regarded as a customized protocol for high-performance, which is suitable for massive reading and writing operations.

SQL Layer

module path: https://github.com/apache/incubator-horaedb/tree/main/query_frontend

SQL layer takes responsibilities for parsing sql and generating the query plan.

Based on sqlparser a sql dialect, which introduces some key concepts including Tag and Timestamp, is provided for processing timeseries data. And by utilizing DataFusion the planner is able to generate both regular logical plans and tailored ones which is used to implement the special operators defined by timeseries queries, e.g PromQL.

Interpreter

module path: https://github.com/apache/incubator-horaedb/tree/main/interpreters

The Interpreter module encapsulates the SQL CRUD operations. In the query procedure, a sql received by HoraeDB is parsed, converted into the query plan and then executed in some specific interpreter, such as SelectInterpreter, InsertInterpreter and etc.

Catalog

module path: https://github.com/apache/incubator-horaedb/tree/main/catalog_impls

Catalog is actually the module managing metadata and the levels of metadata adopted by HoraeDB is similar to PostgreSQL: Catalog > Schema > Table, but they are only used as namespace.

At present, Catalog and Schema have two different kinds of implementation for standalone and distributed mode because some strategies to generate ids and ways to persist metadata differ in different mode.

Query Engine

module path: https://github.com/apache/incubator-horaedb/tree/main/query_engine

Query Engine is responsible for optimizing and executing query plan given a basic SQL plan provided by SQL layer and now such work is mainly delegated to DataFusion.

In addition to the basic functions of SQL, HoraeDB also defines some customized query protocols and optimization rules for some specific query plans by utilizing the extensibility provided by DataFusion. For example, the implementation of PromQL is implemented in this way and read it if you are interested.

Pluggable Table Engine

module path: https://github.com/apache/incubator-horaedb/tree/main/table_engine

Table Engine is actually a storage engine for managing tables in HoraeDB and the pluggability of Table Engine is a core design of HoraeDB which matters in achieving our long-term target, e.g supporting handle log or tracing workload by implementing new storage engines. HoraeDB will have multiple kinds of Table Engine for different workloads and the most appropriate one should be chosen as the storage engine according to the workload pattern.

Now the requirements for a Table Engine are:

Manage all the shared resources under the engine:
- Memory
- Storage
- CPU
Manage metadata of tables such as table schema and table options;
Provide Table instances which provides read and write methods;
Take responsibilities for creating, opening, dropping and closing Table instance;
....

Actually the things that a Table Engine needs to process are a little complicated. And now in HoraeDB only one Table Engine called Analytic is provided and does a good job in processing analytical workload, but it is not ready yet to handle the timeseries workload (we plan to enhance it for a better performance by adding some indexes which help handle timeseries workload).

The following part gives a description about details of Analytic Table Engine.

WAL

module path: https://github.com/apache/incubator-horaedb/tree/main/wal

The model of HoraeDB processing data is WAL + MemTable that the recent written data is written to WAL first and then to MemTable and after a certain amount of data in MemTable is accumulated, the data will be organized in a query-friendly form to persistent devices.

Now three implementations of WAL are provided for standalone and distributed mode:

For standalone mode, WAL is based on RocksDB and data is persisted on the local disk.
For distributed mode, WAL is required as a distributed component and to be responsible for durability of the newly written data, so now we provide an implementation based on OceanBase.
For distributed mode, in addition to OceanBase, we also provide a more lightweight implementation based on Apache Kafka.

MemTable

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/memtable

For WAL can't provide efficient data retrieval, the newly written data is also stored in Memtable for efficient data retrieval, after a certain amount of data is reached, HoraeDB organizes the data in MemTable into a query-friendly storage format (SST) and stores it to the persistent device.

The current implementation of MemTable is based on agatedb's skiplist. It allows concurrent reads and writes and can control memory usage based on Arena.

Flush

module path: https://github.com/apache/incubator-horaedb/blob/main/analytic_engine/src/instance/flush_compaction.rs

What Flush does is that when the memory usage of MemTable reaches the threshold, some MemTables are selected for flushing into query-friendly SSTs saved on persistent device.

During the flushing procedure, the data will be divided by a certain time range (which is configured by table option Segment Duration), and any SST is ensured that the timestamps of the data in it are in the same Segment. Actually this is also a common operation in most timeseries databases which organizes data in the time dimension to speed up subsequent time-related operations, such as querying data over a time range and assisting purge data outside the TTL.

Compaction

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/compaction

The data of MemTable is flushed as SSTs, but the file size of recently flushed SST may be very small. And too small or too many SSTs lead to the poor query performance. Therefore, Compaction is then introduced to rearrange the SSTs so that the multiple smaller SST files can be compacted into a larger SST file.

Manifest

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/meta

Manifest takes responsibilities for managing tables' metadata of Analytic Engine including:

Table schema and table options;
The sequence number where the newest flush finishes;
The information of all the SSTs belonging to the table.

Now the Manifest is based on WAL and Object Storage. The newly written updates on the Manifest are persisted as logs in WAL, and in order to avoid infinite expansion of Manifest (actually every Flush leads to an update), Snapshot is also introduced to clean up the history of metadata updates, and the generated Snapshot will be saved to Object Storage.

Object Storage

module path: https://github.com/apache/incubator-horaedb/tree/main/components/object_store

The SST generated by Flush needs to be persisted and the abstraction of the persistent storage device is ObjectStore including multiple implementations:

Based on local file system;
Based on Alibaba Cloud OSS.

The distributed architecture of HoraeDB separates storage and computing, which requires Object Store needs to be a highly available and reliable service independent of HoraeDB. Therefore, storage systems like Amazon S3, Alibaba Cloud OSS is a good choice and in the future implementations on storage systems of some other cloud service providers is planned to provide.

SST

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/sst

SST is actually an abstraction that can have multiple specific implementations. The current implementation is based on Parquet, which is a column-oriented data file format designed for efficient data storage and retrieval.

The format of SST is very critical for retrieving data and is also the most important part to perform well in handling both timeseries and analytic workloads. At present, our Parquet-based implementation is good at processing analytic workload but is poor at processing timeseries workload. In our roadmap, we will explore more storage formats in order to achieve a good performance in both workloads.

Space

module path: https://github.com/apache/incubator-horaedb/blob/main/analytic_engine/src/space.rs

In Analytic Engine, there is a concept called space and here is an explanation for it to resolve some ambiguities when read source code. Actually Analytic Engine does not have the concept of catalog and schema and only provides two levels of relationship: space and table. And in the implementation, the schema id (which should be unique across all catalogs) on the upper layer is actually mapped to space id.

The space in Analytic Engine serves mainly for isolation of resources for different tenants, such as the usage of memory.

Critical Path

After a brief introduction to some important modules of HoraeDB, we will give a description for some critical paths in code, hoping to provide interested developers with a guide for reading the code.

Query

plaintext

┌───────┐      ┌───────┐      ┌───────┐
│       │──1──▶│       │──2──▶│       │
│Server │      │  SQL  │      │Catalog│
│       │◀─10──│       │◀─3───│       │
└───────┘      └───────┘      └───────┘
                │    ▲
               4│   9│
                │    │
                ▼    │
┌─────────────────────────────────────┐
│                                     │
│             Interpreter             │
│                                     │
└─────────────────────────────────────┘
                           │    ▲
                          5│   8│
                           │    │
                           ▼    │
                   ┌──────────────────┐
                   │                  │
                   │   Query Engine   │
                   │                  │
                   └──────────────────┘
                           │    ▲
                          6│   7│
                           │    │
                           ▼    │
 ┌─────────────────────────────────────┐
 │                                     │
 │            Table Engine             │
 │                                     │
 └─────────────────────────────────────┘

Take SELECT SQL as an example. The figure above shows the query procedure and the numbers in it indicates the order of calling between the modules.

Here are the details:

Server module chooses a proper rpc module (it may be HTTP, gRPC or mysql) to process the requests according the protocol used by the requests;
Parse SQL in the request by the parser;
With the parsed sql and the information provided by catalog/schema module, DataFusion can generate the logical plan;
With the logical plan, the corresponding Interpreter is created and logical plan will be executed by it;
For the logical plan of normal Select SQL, it will be executed through SelectInterpreter;
In the SelectInterpreter the specific query logic is executed by the Query Engine:
- Optimize the logical plan;
- Generate the physical plan;
- Optimize the physical plan;
- Execute the physical plan;
The execution of physical plan involves Analytic Engine:
- Data is obtained by read method of Table instance provided by Analytic Engine;
- The source of the table data is SST and Memtable, and the data can be filtered by the pushed down predicates;
- After retrieving the table data, Query Engine will complete the specific computation and generate the final results;
SelectInterpreter gets the results and feeds them to the protocol module;
After the protocol layer converts the results, the server module responds to the client with them.

The following is the flow of function calls in version v1.2.2:

text

/* Detailed source-code truncated for AI context efficiency. */

The received request will be forwarded to handle_sql after various protocol conversions, and since the request may not be processed by this node, it may need to be forwarded to maybe_forward_sql_query to handle the forwarding logic.
After constructing the ForwardRequest in maybe_forward_sql_query, call forward
After constructing the RouteRequest in forward, call route
Use route to get the destination node endpoint and return to forward.
Call forward_with_endpoint to forward the request
return forward
return maybe_forward_sql_query
return handle_sql
If this is a Local request, call fetch_sql_query_output to process it
Call parse_sql to parse sql into Statment
return fetch_sql_query_output
Call statement_to_plan with Statment
Construct Planner with ctx and Statment, and call the statement_to_plan method of Planner
The planner will call the corresponding planner method for the requested category, at this point our sql is a query and will call sql_statement_to_plan
Call sql_statement_to_datafusion_plan , which will generate the datafusion object, and then call SqlToRel::sql_statement_to_plan
The generated logical plan is returned from SqlToRel::sql_statement_to_plan
return
return
return
Call execute_plan_involving_partition_table (in the default configuration) for subsequent optimization and execution of this logical plan
Call build_interpreter to generate Interpreter
return
Call Interpreter's interpreter_execute_plan method for logical plan execution.
The corresponding execute function is called, at this time the sql is a query, so the execute of the SelectInterpreter will be called
call execute_logical_plan , which will call build_df_session_ctx to generate the optimizer
build_df_session_ctx will use the config information to generate the corresponding context, first using datafusion and some custom optimization rules (in logical_optimize_rules()) to generate the logical plan optimizer, using apply_adapters_for_physical_optimize_rules to generate the physical plan optimizer
return optimizer
Call optimize_plan, using the optimizer just generated to first optimize the logical plan and then the physical plan
Return to optimized physical plan
execute physical plan
returned after execution
After collecting the results of all slices, return
return
return
return
Return to the upper layer for network protocol conversion and finally return to the request sender

Write

plaintext

┌───────┐      ┌───────┐      ┌───────┐
│       │──1──▶│       │──2──▶│       │
│Server │      │  SQL  │      │Catalog│
│       │◀─8───│       │◀─3───│       │
└───────┘      └───────┘      └───────┘
                │    ▲
               4│   7│
                │    │
                ▼    │
┌─────────────────────────────────────┐
│                                     │
│             Interpreter             │
│                                     │
└─────────────────────────────────────┘
      │    ▲
      │    │
      │    │
      │    │
      │    │       ┌──────────────────┐
      │    │       │                  │
     5│   6│       │   Query Engine   │
      │    │       │                  │
      │    │       └──────────────────┘
      │    │
      │    │
      │    │
      ▼    │
 ┌─────────────────────────────────────┐
 │                                     │
 │            Table Engine             │
 │                                     │
 └─────────────────────────────────────┘

Take INSERT SQL as an example. The figure above shows the query procedure and the numbers in it indicates the order of calling between the modules.

Here are the details:

Server module chooses a proper rpc module (it may be HTTP, gRPC or mysql) to process the requests according the protocol used by the requests;
Parse SQL in the request by the parser;
With the parsed sql and the catalog/schema module, DataFusion can generate the logical plan;
With the logical plan, the corresponding Interpreter is created and logical plan will be executed by it;
For the logical plan of normal INSERT SQL, it will be executed through InsertInterpreter;
In the InsertInterpreter, write method of Table provided Analytic Engine is called:
- Write the data into WAL first;
- Write the data into MemTable then;
Before writing to MemTable, the memory usage will be checked. If the memory usage is too high, the flush process will be triggered:
- Persist some old MemTables as SSTs;
- Store updates about the new SSTs and the flushed sequence number of WAL to Manifest;
- Delete the corresponding WAL entries;
Server module responds to the client with the execution result.

File: content/en/docs/design/clustering.md

title: "Introduction to Architecture of HoraeDB Cluster"
---Note: Some of the features mentioned in the article have not yet been implemented.

Overview

text

/* Detailed source-code truncated for AI context efficiency. */

The diagram above describes the architecture of a HoraeDB cluster, where some key concepts need to be explained:

HoraeMeta Cluster: Takes responsibilities for managing the metadata and resource scheduling of the cluster;
Shard(L)/Shard(F): Leader shard and follower shard consisting of multiple tables;
HoraeDB: One HoraeDB instance consisting of multiple shards;
WAL Service: Write-ahead log service for storing new-written real-time data;
Object Storage: Object storage service for storing SST converted from memtable;

From the architecture diagram above, it can be concluded that the compute and storage are separated in the HoraeDB cluster, which makes it easy to implement useful distributed features, such as elastic autoscaling of compute/storage resources, high availability, load balancing, and so on.

Let's dive into some of the key components mentioned above before explaining how these features are implemented.

Shard

Shard is the basic scheduling unit in the cluster, which consists of a group of tables. And the tables in a shard share the same region for better storage locality in the WAL Service, and because of this, it is efficient to recover the data of all tables in the shard by scanning the entire WAL region. For most of implementations of WAL Service, without the shard concept, it costs a lot to recover the table data one by one due to massive random IO, and this case will deteriorate sharply when the number of tables grows to a certain level.

A specific role, Leader or Follower, should be assigned to a shard. A pair of leader-follower shards share the same set of tables, and the leader shard can serve the write and query requests from the client while the follower shard can only serve the read-only requests, and must synchronize the newly written data from the WAL service in order to provide the latest snapshot for data retrieval. Actually, the follower is not needed if the high availability is not required, while with at least one follower, it takes only a short time to resume service by simply switching the Follower to Leader when the HoraeDB instance on which the leader shard exists crashes.

The diagram below concludes the relationship between HoraeDB instance, Shard, Table. As shown in the diagram, the leader and follower shards are interleaved on the HoraeDB instance.

plaintext

┌─HoraeDB Instance0──────┐     ┌─HoraeDB Instance1──────┐
│  ┌─Shard0(L)────────┐  │     │  ┌─Shard0(F)────────┐  │
│  │ ┌────┬────┬────┐ │  │     │  │ ┌────┬────┬────┐ │  │
│  │ │ T0 │ T1 │ T2 │ │  │     │  │ │ T0 │ T1 │ T2 │ │  │
│  │ └────┴────┴────┘ │  │     │  │ └────┴────┴────┘ │  │
│  └──────────────────┘  │     │  └──────────────────┘  │
│                        │     │                        │
│  ┌─Shard1(F)────────┐  │     │  ┌─Shard1(L)────────┐  │
│  │ ┌────┬────┬────┐ │  │     │  │ ┌────┬────┬────┐ │  │
│  │ │ T0 │ T1 │ T2 │ │  │     │  │ │ T0 │ T1 │ T2 │ │  │
│  │ └────┴────┴────┘ │  │     │  │ └────┴────┴────┘ │  │
│  └──────────────────┘  │     │  └──────────────────┘  │
└────────────────────────┘     └────────────────────────┘

Since Shard is the basic scheduling unit, it is natural to introduce some basic shard operations:

Create/Drop table to/from a shard;
Open/Close a shard;
Split one shard into two shards;
Merge two shards into one shard;
Switch the role of a shard;

With these basic shard operations, some complex scheduling logic can be implemented, e.g. perform an expansion by splitting one shard into two shards and migrating one of them to the new HoraeDB instance.

HoraeMeta

HoraeMeta is implemented by embedding an ETCD inside to ensure consistency and takes responsibilities for cluster metadata management and scheduling.

The cluster metadata includes:

Table information, such as table name, table ID, and which cluster the table belongs to;
The mapping between table and shard and between shard and HoraeDB instance;
...

As for the cluster scheduling work, it mainly includes:

Receiving the heartbeats from the HoraeDB instances and determining the online status of these registered instances;
Assigning specific role shards to the registered HoraeDB instances;
Participating in table creation by assigning a unique table ID and the most appropriate shard to the table;
Performing load balancing through shard operations according to the load information sent with the heartbeats;
Performing expansion through shard operations when new instances are registered;
Initiating failover through shard operations when old instances go offline;

Route

In order to avoid the overhead of forwarding requests, the communication between clients and the HoraeDB instances is peer-to-peer, that is to say, the client should retrieve routing information from the server before sending any specific write/query requests.

Actually, the routing information is decided by the HoraeMeta, but clients are only allowed the access to it through the HoraeDB instances rather than HoraeMeta, to avoid potential performance issues on the HoraeMeta.

WAL Service & Object Storage

In the HoraeDB cluster, WAL Service and Object Storage exist as separate distributed systems featured with HA, data replication and scalability. Current distributed implementations for WAL Service includes Kafka and OBKV (access OceanBase by its table api), and the implementations for Object Storage include popular object storage services, such as AWS S3, Azure object storage and Aliyun OSS.

The two components are similar in that they are introduced to serve as the underlying storage layer for separating compute and storage, while the difference between two components is obvious that WAL Service is used to store the newly written data from the real-time write requests whose individual size is small but quantity is large, and Object Storage is used to store the read-friendly data files (SST) organized in the background, whose individual size is large and aggregate size is much larger.

The two components make it much easier to implement the horaedb cluster, which features horizontal scalability, high availability and load balancing.

Scalability

Scalability is an important feature for a distributed system. Let's take a look at to how the horizontal scalability of the HoraeDB cluster is achieved.

First, the two storage components (WAL Service and Object Storage) should be horizontally scalable when deciding on the actual implementations for them, so the two storage services can be expanded separately if the storage capacity is not sufficient.

It will be a little bit complex when discussing the scalability of the compute service. Basically, these cases will bring the capacity problem:

Massive queries on massive tables;
Massive queries on a single large table;
Massive queries on a normal table;

For the first case, it is easy to achieve horizontal scalability just by assigning shards that are created or split from old shards to expanded HoraeDB instances.

For the second case, the table partitioning is proposed and after partitioning, massive queries are distributed across multiple HoraeDB instances.

And the last case is the most important and the most difficult. Actually, the follower shard can handle part of the queries, but the number of follower shards is limited by the throughput threshold of the synchronization from the WAL regions. As shown in the diagram below, a pure compute shard can be introduced if the followers are not enough to handle the massive queries. Such a shard is not required to synchronize data with the leader shard, and retrieves the newly written data from the leader/follower shard only when the query comes. As for the SSTs required by the query, they will be downloaded from Object Storage and cached afterwards. With the two parts of the data, the compute resources are fully utilized to execute the CPU-intensive query plan. As we can see, such a shard can be added with only a little overhead (retrieving some data from the leader/follower shard when it needs), so to some extent, the horizontal scalability is achieved.

plaintext

┌HoraeDB─────┬┬─┐
                            ┌──newly written─│  │  │TableN││ │
                            ▼                └──Shard(L/F)┴┴─┘
┌───────┐  Query  ┌HoraeDB─────┬┬─┐
│client │────────▶│  │  │TableN││ │
└───────┘         └──Shard─────┴┴─┘          ┌───────────────┐
                            ▲                │    Object     │
                            └───old SST──────│    Storage    │
                                             └───────────────┘

High Availability

Assuming that WAL service and Object Storage are highly available, the high availability of the HoraeDB cluster can be achieved by such a procedure:

When detecting that the heartbeat is broken, HoraeMeta determines that the HoraeDB instance is offline;
The follower shards whose paired leader shards exist on the offline instance are switched to leader shards for fast failover;
A slow failover can be achieved by opening the crashed shards on another instance if such follower shards don't exist.

text

/* Detailed source-code truncated for AI context efficiency. */

Load Balancing

HoraeMeta collects the instance load information contained in the received heartbeats to create a load overview of the whole cluster, according to which the load balancing can be implemented as an automatic mechanism:

Pick a shard on a low-load instance for the newly created table;
Migrate a shard from a high-load instance load to another low-load instance;
Split the large shard on the high-load instance and migrate the split shards to other low-load instances;

File: content/en/docs/design/compaction_offload.md

title: "Compaction Offload"

Note: This feature is still in development.

This chapter discusses compaction offload, which is designed to separate the compaction workload from the local horaedb nodes and delegate it to external compaction nodes.

Overview

plaintext

┌─────────────────────────────────────────────────────────────────────────┐
│                                                                         │
│                           HoraeMeta Cluster                             │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
    ▲                ▲                           |                   |
    │                │1.Fetch compaction         │(Monitor compaction│
    │                │  node  info               │node)              │
    |                │                           ▼                   ▼
┌────────────┐  ┌────────────┐              ┌────────────┐   ┌────────────┐
│            │  │            │2.Offload Task│            │   │            │
│  HoraeDB   │  │  HoraeDB   │  ─────────▶  │ Compaction │   │ Compaction │
│            │  │            │  ◀─────────  │ Node       │   │ Node       │
└────────────┘  └────────────┘ 4.Ret Result └────────────┘   └────────────┘
    |                |                            |                 |
    │  5.Update the  │                            │    3.Compact    │
    │    SSTable     │                            │                 │
    ▼                ▼                            ▼                 ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                                                ┌─────────────────────┐  │
│                            Object Storage      │ Temporary Workspace │  │
│                                                └─────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────┘

The diagram above describes the architecture of cluster for compaction offload, where some key concepts need to be explained:

Compaction Node: Takes responsibility to handle offloaded compaction tasks. The compaction node receives the compaction task and performs the actual merging of SSTables, then sends back the task result to HoraeDB.
HoraeMeta Cluster: HoraeMeta acts as a compaction nodes manager in the compaction offload scenario. It monitors the compaction nodes cluster and schedule the compaction nodes.

The procedure of remote compaction based above architecture diagram is:

HoraeDB servers fetch the information of suitable compaction nodes from the HoraeMeta.
HoraeDB submit the compaction task to the remote compaction node, according to the information fetched from HoraeMeta.
Compaction node executes the task and write results to the temporary workspace.
Compaction node sends compaction results back to HoraeDB.
HoraeDB receives the result, installs the data in temporary workspace and purges compaction input files.

The architecture above makes it easy to implement some wonderful features like load balancing and high availability. Let's dive into the key components in the architecture and talking about how these features are implemented.

Compaction Node

Compaction Node runs the main logic of compaction. It is implemented based on HoraeDB and distinguished by:

NodeType: A config parameter used to distinguished the HoraeDB and CompactionNode. This info would be sent to HoraeMeta through heartbeat.

The compaction service is implemented as grpc service.

HoraeMeta

HoraeMeta manages the compaction nodes cluster with CompactionNodeManager, which takes responsibilities for compaction nodes metadata management and scheduling.

The compaction nodes metadata includes:

Compaction node information, such as node name, node state;
A compaction node name list, used as the key to access compaction node info, for better scheduling with round-robin strategy;
...

As for the compaction nodes scheduling work, it mainly includes:

Receiving the heartbeats from the compaction node and determining the online status of these registered nodes.
Performing load balancing according to the compaction nodes cluster info.
Providing the info of suitable compaction node for HoraeDB when remote compaction execution is needed.

Load Balancing

Load Balancing is critical for compaction nodes cluster to make their overall processing more efficient. The effect of load balancing mainly based on the schedule algorithm for compaction nodes impl in CompactionNodeManager.

(ps: The current implementation of schedule algorithm is round-robin strategy for easiness.)

The main process for the schedule algorithm based on real load is:

HoraeMeta collects the compaction nodes load information through the heartbeats to create a load overview of the compaction nodes cluster.
Pick a compaction node with low load according to the load overview.

High Availability

The fault tolerance of above architecture can be achieved by such a procedure:

When detecting that the heartbeat is broken, HoraeMeta determines that the compaction node is offline.
When HoraeMeta can not provide suitable compaction node for HoraeDB or compaction node doesn't return the task result successfully, HoraeDB would switches to run compaction task locally.

File: content/en/docs/design/storage.md

title: "Storage"

The storage engine mainly provides the following two functions：

Persistence of data
Under the premise of ensuring the correctness of the data, organize the data in the most reasonable way to meet the query needs of different scenarios.

This document will introduce the internal implementation of the storage engine in HoraeDB. Readers can refer to the content here to explore how to use HoraeDB efficiently.

Overall Structure

HoraeDB is a distributed storage system based on the share-nothing architecture.

Data between different servers is isolated from each other and does not affect each other. The storage engine in each stand-alone machine is a variant of log-structured merge-tree, which is optimized for time-series scenarios. The following figure shows its core components:

Write Ahead Log (WAL)

A write request will be written to

memtable in memory
WAL in durable storage

Since memtable is not persisted to the underlying storage system in real time, so WAL is required to ensure the reliability of the data in memtable.

On the other hand, due to the design of the distributed architecture, WAL itself is required to be highly available. Now there are following implementations in HoraeDB:

Local disk (based on RocksDB, no distributed high availability)
OceanBase
Kafka

Memtable

Memtable is a memory data structure used to hold recently written table data. Different tables have its corresponding memtable.

Memtable is read-write by default (aka active), and when the write reaches some threshold, it will become read-only and be replaced by a new memtable.

The read-only memtable will be flushed to the underlying storage system in SST format by background thread. After flush is completed, the read-only memtable can be destroyed, and the corresponding data in WAL can also be deleted.

Sorted String Table（SST）

SST is a persistent format for data, which is stored in the order of primary keys of table. Currently, HoraeDB uses parquet format for this.

For HoraeDB, SST has an important option: segment_duration, only SST within the same segment can be merged, which is benefical for time-series data. And it is also convenient to eliminate expired data.

In addition to storing the original data, the statistical information of the data will also be stored in the SST to speed up the query, such as the maximum value, the minimum value, etc.

Compactor

Compactor can merge multiple small SST files into one, which is used to solve the problem of too many small files. In addition, Compactor will also delete expired data and duplicate data during the compaction. In future, compaction maybe add more task, such as downsample.

The current compaction strategy in HoraeDB reference Cassandra:

Manifest

Manifest records metadata of table, SST file, such as: the minimum and maximum timestamps of the data in an SST.

Due to the design of the distributed architecture, the manifest itself is required to be highly available. Now in HoraeDB, there are mainly the following implementations：

WAL
ObjectStore

ObjectStore

ObjectStore is place where data (i.e. SST) is persisted.

Generally speaking major cloud vendors should provide corresponding services, such as Alibaba Cloud's OSS and AWS's S3.

Docker & Self-Hosted Deployment Verified OCI

1. Project Overview & Quickstart (apache/tvm) Copy Chapter

File: README.md Copy Chapter

Open Machine Learning Compiler Framework

License Copy Chapter

Getting Started Copy Chapter

Contribute to TVM Copy Chapter

History and Acknowledgement Copy Chapter

File: apps/ios_rpc/README.md Copy Chapter

iOS TVM RPC

Table of Contents Copy Chapter

Building Copy Chapter

Building TVM runtime and custom DSO loader plugin Copy Section

Building iOS TVM RPC application Copy Section

Workflow Copy Chapter

Standalone RPC Copy Section

iOS RPC App with proxy Copy Section

iOS RPC App with tracker Copy Section

Communication without Wi-Fi and speed up in case of slow Wi-Fi Copy Chapter

File: apps/cpp_rpc/README.md Copy Chapter

TVM RPC Server

Usage (Non-Windows) Copy Chapter

Usage (Windows) Copy Chapter

How it works Copy Chapter

Note Copy Chapter

File: apps/android_rpc/README.md Copy Chapter

Android TVM RPC

Build and Installation Copy Chapter

Build APK Copy Section

Build with OpenCL Copy Section

Cross Compile and Run on Android Devices Copy Chapter

Architecture and Android Standalone Toolchain Copy Section

Cross Compile and Upload to the Android Device Copy Section

Troubleshooting Copy Section

File: ci/jenkins/README.md Copy Chapter

TVM CI

Jenkins Copy Chapter

GitHub Actions Copy Chapter

Jenkins CI

`Jenkinsfile` Copy Chapter

Infrastructure

File: ci/README.md Copy Chapter

Apache TVM Continuous Integration (CI)

Overview Copy Chapter

CI Documentation Index Copy Chapter

Jenkins Copy Chapter

GitHub Actions Copy Chapter

Docker Images Copy Chapter

`ci-docker-staging` Copy Section

Jenkins CI

`Jenkinsfile` Copy Chapter

2. Official Technical Reference & Guides (apache/docs) Copy Chapter

File: README.md Copy Chapter

Apache HoraeDB (incubating) Website Copy Chapter

How to run Copy Chapter

Contributing Copy Chapter

File: content/en/blog/2024/release-2.0.0.md Copy Chapter

title: Release 2.0.0 date: 2024-04-23 Copy Chapter

Upgrade from 1.x.x to 2.0.0 Copy Chapter

Upgrade Steps

What's Changed Copy Chapter

Breaking Changes Copy Section

Features Copy Section

Refactor Copy Section

Fixed Copy Section

Docs Copy Section

Chore Copy Section

New Contributors Copy Chapter

File: content/en/blog/2024/release-2.1.0.md Copy Chapter

1. New WAL implementation based on local disk. Copy Chapter

How to enable Copy Section

2. Access object store with [Apache OpenDAL](https://github.com/apache/opendal) Copy Chapter

Download Copy Chapter

Conclusion Copy Chapter

File: content/en/blog/_index.md Copy Chapter

File: content/en/community/_index.md Copy Chapter

How to Subscribe Mail List

Unsubscribe from the mail list Copy Chapter

File: content/en/docs/design/architecture.md Copy Chapter

title: "Introduction to HoraeDB's Architecture" Copy Chapter

1. Project Overview & Quickstart (apache/tvm)

File: README.md

License

Getting Started

Contribute to TVM

History and Acknowledgement

File: apps/ios_rpc/README.md

Table of Contents

Building

Building TVM runtime and custom DSO loader plugin

Building iOS TVM RPC application

Workflow

Standalone RPC

iOS RPC App with proxy

iOS RPC App with tracker

Communication without Wi-Fi and speed up in case of slow Wi-Fi

File: apps/cpp_rpc/README.md

Usage (Non-Windows)

Usage (Windows)

How it works

Note

File: apps/android_rpc/README.md

Build and Installation

Build APK

Build with OpenCL

Cross Compile and Run on Android Devices

Architecture and Android Standalone Toolchain

Cross Compile and Upload to the Android Device

Troubleshooting

File: ci/jenkins/README.md

Jenkins

GitHub Actions

`Jenkinsfile`

File: ci/README.md

Overview

CI Documentation Index

Jenkins

GitHub Actions

Docker Images

`ci-docker-staging`

`Jenkinsfile`

2. Official Technical Reference & Guides (apache/docs)

File: README.md

Apache HoraeDB (incubating) Website

How to run

Contributing

File: content/en/blog/2024/release-2.0.0.md

title: Release 2.0.0 date: 2024-04-23

Upgrade from 1.x.x to 2.0.0

What's Changed

Breaking Changes

Features

Refactor

Fixed

Docs

Chore

New Contributors

File: content/en/blog/2024/release-2.1.0.md

1. New WAL implementation based on local disk.

How to enable

2. Access object store with [Apache OpenDAL](https://github.com/apache/opendal)

Download

Conclusion

File: content/en/blog/_index.md

File: content/en/community/_index.md

Unsubscribe from the mail list

File: content/en/docs/design/architecture.md

title: "Introduction to HoraeDB's Architecture"

Target

Motivation

Architecture

RPC Layer

SQL Layer

Interpreter

Catalog

Query Engine

Pluggable Table Engine

Critical Path

Query

Write