skywalking Documentation

APM, Application Performance Monitoring System

24,918 stars Java Java 1 file · ~30,851 tokens #apm#dapper#distributed-tracing#ebpf#logging#metrics#observability#open-telemetry
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy skywalking on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name skywalking -p 9200:9200 --restart always ${OAP_IMAGE:-ghcr.io/apache/skywalking/oap:latest}
Configured Services & Ports
⚙️ elasticsearch
image: ${ELASTICSEARCH_IMAGE:-docker.elastic.co/elasticsearch/elasticsearch-oss:7.4.2}
🔌 Ports: 9200:9200
⚙️ banyandb
image: ${BANYANDB_IMAGE}
🔌 Ports: 17913:17913
⚙️ oap-base
image: ${OAP_IMAGE:-ghcr.io/apache/skywalking/oap:latest}
🔌 Ports: 11800:11800, 12800:12800, 17128:17128
⚙️ oap-es
image: ${OAP_IMAGE:-ghcr.io/apache/skywalking/oap:latest}
🔌 Ports: 11800:11800, 12800:12800, 17128:17128
⚙️ oap-bdb
image: ${OAP_IMAGE:-ghcr.io/apache/skywalking/oap:latest}
🔌 Ports: 11800:11800, 12800:12800, 17128:17128
⚙️ ui
image: ${UI_IMAGE:-ghcr.io/apache/skywalking-horizon-ui:latest}
🔌 Ports: 8080:8081
docker-compose.yml
yaml (docker-compose.yml) 
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

services:
  elasticsearch:
    profiles:
      - "elasticsearch"
    image: ${ELASTICSEARCH_IMAGE:-docker.elastic.co/elasticsearch/elasticsearch-oss:7.4.2}
    container_name: elasticsearch
    ports:
      - "9200:9200"
    networks:
      - demo
    healthcheck:
      test: [ "CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1" ]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    environment:
      - discovery.type=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1

  banyandb:
    profiles:
      - "banyandb"
    image: ${BANYANDB_IMAGE}
    container_name: banyandb
    networks:
      - demo
    expose:
      - 17912
    ports:
      - 17913:17913
    command: standalone --stream-root-path /tmp/stream-data --measure-root-path /tmp/measure-data
    healthcheck:
      test: [ "CMD", "sh", "-c", "nc -nz 127.0.0.1 17912" ]
      interval: 5s
      timeout: 60s
      retries: 120

  oap-base: &oap-base
    profiles: [ "none" ]
    image: ${OAP_IMAGE:-ghcr.io/apache/skywalking/oap:latest}
    ports:
      - "11800:11800"
      - "12800:12800"
      - "17128:17128"
    networks:
      - demo
    healthcheck:
      # The OAP image (eclipse-temurin JRE) ships bash but no curl/wget/nc,
      # so probe the query port with bash's built-in /dev/tcp instead.
      test: [ "CMD", "bash", "-c", "cat < /dev/null > /dev/tcp/127.0.0.1/12800" ]
      interval: 15s
      timeout: 10s
      retries: 10
      # OAP can take 60-120s to boot (BanyanDB schema install); don't count
      # failures against the retry budget during that window.
      start_period: 90s
    environment: &oap-env
      SW_HEALTH_CHECKER: default
      SW_TELEMETRY: prometheus
      JAVA_OPTS: "-Xms2048m -Xmx2048m"

  oap-es:
    <<: *oap-base
    profiles: 
      - "elasticsearch"
    container_name: oap
    depends_on:
      elasticsearch:
        condition: service_healthy
    environment:
      <<: *oap-env
      SW_STORAGE: elasticsearch
      SW_STORAGE_ES_CLUSTER_NODES: elasticsearch:9200

  oap-bdb:
    <<: *oap-base
    profiles:
      - "banyandb"
    container_name: oap
    depends_on:
      banyandb:
        condition: service_healthy
    environment:
      <<: *oap-env
      SW_STORAGE: banyandb
      SW_STORAGE_BANYANDB_TARGETS: banyandb:17912

  # SkyWalking Horizon UI — official web UI as of 10.5.0.
  # Source: https://github.com/apache/skywalking-horizon-ui
  # Talks to OAP via the public GraphQL/REST port (12800) and to admin-server
  # for write APIs (UI templates, runtime-rule, dsl-debugging, status, inspect).
  ui:
    image: ${UI_IMAGE:-ghcr.io/apache/skywalking-horizon-ui:latest}
    container_name: ui
    ports:
      # Horizon listens on 8081 inside the container (HORIZON_SERVER_PORT).
      - "8080:8081"
    networks:
      - demo
    volumes:
      # Backend URLs (OAP query/admin/zipkin) and login users come ONLY from
      # this file; there are no env vars for them. The image's HORIZON_CONFIG
      # default points here.
      - ./horizon.yaml:/app/horizon.yaml:ro

networks:
  demo:

1. Project Overview & Quickstart (apache/skywalking)

File: README.md

Apache SkyWalking

SkyWalking: an APM (Application Performance Monitoring) system, especially designed for
microservices, cloud native and container-based architectures.


Abstract

SkyWalking is an open-source APM system that provides monitoring, tracing and diagnosing capabilities for distributed systems in Cloud Native architectures.

  • Distributed Tracing
    • End-to-end distributed tracing. Service topology analysis, service-centric observability and APIs dashboards.
  • Agents for your stack
    • Java, .Net Core, PHP, NodeJS, Golang, LUA, Rust, C++, Client JavaScript and Python agents with active development and maintenance.
  • eBPF early adoption
    • Rover agent works as a monitor and profiler powered by eBPF to monitor Kubernetes deployments and diagnose CPU and network performance.
  • Scaling
    • 100+ billion telemetry data could be collected and analyzed from one SkyWalking cluster.
  • Mature Telemetry Ecosystems Supported
    • Metrics, Traces, and Logs from mature ecosystems are supported, e.g. Zipkin, OpenTelemetry, Prometheus, Zabbix, Fluentd
  • Native APM Database
    • BanyanDB, an observability database, created in 2022, aims to ingest, analyze and store telemetry/observability data.
  • Consistent Metrics Aggregation
    • SkyWalking native meter format and widely known metrics format(OpenTelemetry, Telegraf, Zabbix, e.g.) are processed through the same script pipeline.
  • Log Management Pipeline
    • Support log formatting, extract metrics, various sampling policies through script pipeline in high performance.
  • Alerting and Telemetry Pipelines
    • Support service-centric, deployment-centric, API-centric alarm rule setting. Support forwarding alarms and all telemetry data to 3rd party.
  • AI Power Enabled
    • Machine Learning (ML) and Artificial Intelligence (AI) analyze observability data to identify patterns and enhance capabilities, such as recognizing HTTP URI patterns and automatically calculating metric baselines for intelligent alerting, improving anomaly detection.

Live Demo

Documentation

Downloads

Please head to the releases page to download a release of Apache SkyWalking.

Compiling project

Follow this document.

Code of conduct

This project adheres to the Contributor Covenant code of conduct. By participating, you are expected to uphold this code.
Please follow the REPORTING GUIDELINES to report unacceptable behavior.

Contact Us

Our Users

Hundreds of companies and organizations use SkyWalking for research, production, and commercial purposes.
Visit our website to find the user page.

License

Apache 2.0 License.

File: docker/README.md

Docker Images

This folder contains the Dockerfile used to build our
OAP Docker image. If you want
to use the image, please check
the user guide for OAP.

The web UI is no longer built from this repository — see the
SkyWalking Horizon UI project,
whose images are published to
ghcr.io/apache/skywalking-horizon-ui.

Quickstart

You can use Makefile located at the root folder to build the OAP docker image
with the current codebase.

shell
make docker
# OR skip the tests
make docker SKIP_TEST=true

It not only contains the process of building a docker image but also includes all the required steps, for instance, init
workspace, build artifact from scratch. It builds a single OAP image.

shell
docker image ls | grep skywalking
skywalking/oap                                  latest              2a6084450b44        6 minutes ago       862MB

Building variables

There are some environment variables to control image building.

`CONTEXT`

The Docker build context path, under this path, there should be the distribution tar ball.

shell
ls $CONTEXT
apache-skywalking-apm-bin.tar.gz

`DIST`

The distribution tar ball name, for example, apache-skywalking-apm-bin.tar.gz.

`HUB`

The hub of docker image. The default value is skywalking.

`TAG`

The tag of docker image. The default value is latest.

Running containers with docker-compose

We can start up backend cluster by docker-compose. There are two profiles with
different storage options that you can choose, elasticsearch and banyandb.

To start up the backend cluster with elasticsearch as the storage, run the
following command:

shell
docker compose --profile elasticsearch up

To start up the backend cluster with banyandb as the storage, run the
following command:

shell
docker compose --profile banyandb up

docker/.env file contains some configurations that you can customize,
such as the Docker image registry and tags.

After the services are up and running, you can send telemetry data to
localhost:11800 and access the Horizon UI at http://localhost:8080
(default login admin / admin, configured in docker/horizon.yaml).
The OAP admin host is exposed on localhost:17128 (UI templates, runtime-rule
hot update, DSL debugging, status, inspect).

File: docs/en/ui/README.md

UI Customization

The OAP backend in this repository ships the data plane — receivers,
analyzers, metric definitions, entity model, query surfaces. It does not
ship dashboards or marketplace visualizations. Those live in
apache/skywalking-horizon-ui,
the official web UI, which releases independently of the OAP backend.

As of 11.0.0 the boot-time UITemplateInitializer / UIMenuInitializer
have been removed and OAP no longer seeds any dashboard JSONs or menu
items on startup. The dashboard library, the sidebar menu, the
marketplace layout, i18n strings — every layer of visualization —
ships from Horizon UI.

What Horizon UI provides

Horizon UI is the official web UI and the visualization layer of
SkyWalking. It is a separate sub-project with its own release cadence
and configuration; OAP only persists the dashboards that operators save.
At a high level Horizon UI covers:

  • Setup — container image, horizon.yaml, OAP connection, file
    layout, debug logging.
  • Access control — local / LDAP / break-glass auth backends, RBAC
    roles and verbs, audit log, admin-page gating.
  • Customization — sidebar menu structure, layer dashboard templates,
    overview templates, adding a new layer.
  • Components — overview widgets, dashboard widgets, chart types.
  • Operate — cluster status & metadata, inspect.

Each area lives in the
Horizon UI documentation.

Bundled UI templates

The OAP backend does not seed dashboard JSONs at boot anymore.
Horizon UI owns and ships the entire bundled set in its own image: the
default dashboard configurations for every supported layer, several
cross-cutting overview and 3D infrastructure-map setups, the sidebar
menu, and i18n translations. Operators customize these in the UI's admin
pages; see the
Horizon UI documentation.

Editing dashboards from the UI

Horizon UI's dashboard editor drives the admin REST endpoints from the
browser. The write surface is gated behind the admin host's
security posture; see the
admin-server security notice.

File: docs/en/swip/readme.md

SWIP - SkyWalking Improvement Proposal

SWIP - SkyWalking Improvement Proposal, is an official document to propose a new feature and/or feature improvement,
which
are relative to end users and developers.

SkyWalking has been very stable since v9.x. We are getting over the rapid changing stage. The core concepts, protocols for
reporting telemetry and query, 3rd party integration, and the streaming process kernel are very stable. From now(2024) on,
SkyWalking community would focus more on improvement and controllable improvement. All major changes should be evaluated
more seriously, and try as good as possible to avoid incompatible breaking changes.

What is considered a major change?

The catalogs of a major change are listed as follows

  • New Feature. A feature doesn't exist for the latest version.
  • Any change of the network Interfaces, especially
    for Query Protocol,
    Data Collect Protocols,
    Dynamic Configuration APIs, Exporting APIs, AI pipeline APIs.
  • Any change of storage structure.

Q: Is Agent side feature or change considered a SWIP?

A: Right now, SWIP targets OAP and UI side changes. All agent side changes are pending on the reviews from the
committers of those agents.

SWIP Template

The purpose of this template should not be considered a hard requirement. The major purpose of SWIP is helping the PMC
and community member to understand the proposal better.

markdown
# Title: SWIP-1234 xxxx

## Motivation 
The description of new feature or improvement.

## Architecture Graph
Describe the relationship between your new proposal part and existing components.

## Proposed Changes
State your proposal in detail.

## Imported Dependencies libs and their licenses.

## Compatibility
Whether breaking configuration, storage structure, or protocols.

## General usage docs
This doesn't have to be a final version, but helps the reviewers to understand how to use this new feature.

SWIP Process

Here is the process for starting a SWIP.

  1. Start a SWIP discussion at GitHub Discussion Page
    with title [DISCUSS] xxxx.
  2. Fill in the sections as described above in SWIP Template.
  3. At least one SkyWalking committer commented on the discussion to show interest in adopting it.
  4. This committer could update this page to grant a SWIP ID, and update the title to [SWIP-ID NO.] [DISCUSS] xxxx.
  5. All further discussion could happen on the discussion page.
  6. Once the consensus is made by enough committer supporters, and/or through a mail list vote, this SWIP should be
    added here as SWIP-ID NO.md and listed in the below as Known SWIPs.

All accepted and proposed SWIPs can be found in here.

Known SWIPs

Next SWIP Number: 16

Proposed SWIPs

Accepted SWIPs

File: docs/en/swip/SWIP-1.md

Create and detect Service Hierarchy Relationship

Motivation

Service relationship is one of the most important parts of collaborating data in the APM. Service Map is supported for
years from tracing to trace analysis. But still due to the means of the probs, a service could be detected from multiple
methods, which is the same service in multiple
layers. v9 proposal mentioned the concept of the layer.
Through this proposal, we plan to establish a kernel-level concept to connect services detected in different layers.

Architecture Graph

There is no significant architecture-level change.

Propose Changes

The data sources of SkyWalking APM have covered traditional agent installed service, VMs, cloud infra, k8s, etc.

For example, a Java service is built in a docker image and is going to be deployed in a k8s cluster, with a sidecar
injected due to service mesh managed. The following services would be able to detect cross-layers

  1. Java service, detected as Java agent installed.
  2. A pod of k8s service is detected, due to k8s layer monitoring.
  3. Side car perspective service is detected.
  4. VM Linux monitoring for a general process, as the container of Java service is deployed on this specific k8s node.
  5. Virtual databases, caches, and queues conjectured by agents, and also monitored through k8s monitoring, even traffic
    monitored by service mesh.

All these services have logic connections or are identical from a physical perspective, but currently, they may be just
similar on name(s), no further metadata connection.

By those, we have a chance to move one step ahead to connect the dots of the whole infrastructure. This means, for the
first time, we are going to establish the connections among services detected from various layers.

In the v10, I am proposing a new concept Service Hierarchy. Service Hierarchy defines the relationships of
existing services in various layers. With more kinds of agent tech involved(such as eBPF) and deployment tools(such as
operator and agent injector), we could inject relative service/instance metadata and try to build the connections,
including,

  • Agent injector injects the pod ID into the system env, then Java agent could report the relationship through system
    properties.
  • Rover(eBPF agent) reveals its next iteration forward k8s monitoring rather than profiling. And add the capabilities to
    establish connections among k8s pods and service mesh srv.

Meanwhile, as usual with the new major version change, I would expect UI side changes as well. UI should have flexible
capabilities to show hierarchy services from the service view and topology view. Also, we could consider a deeper view
of the instance part as well.

Imported Dependencies libs and their licenses.

No new library is planned to be added to the codebase.

Compatibility

About the protocol, there should be no breaking changes, but enhancements only. New query protocols(
service-hierarchy and instance-hierarchy) are considered to be added, some new fields should be added on things like
topology query and instance dependencies to list relative services/instances from other layers directly rather than an
extra query.

About the data structure, due to the new data concept is going to be created, service hierarchy relative data models are
going to be added. If the user is using Elasticsearch and BanyanDB, this should be compatible, they just need to
re-run init-mode OAP to extend the existing models. But for SQL database users(MySQL, PostgreSQL), this could require
new tables.

GraphQL query protocol

New query protocol hierarchy.graphqls is going to be added.

graphql
type HierarchyRelatedService {
  # The related service ID.
  id: ID!
  # The literal name of the #id.
  name: String!
  # The related service's Layer name.
  layer: String!
  normal: Boolean!
}

type HierarchyRelatedInstance {
  # The related instance ID.
  id: ID!
  # The literal name of the #id. Instance Name.
  name: String!
  # Service id
  serviceId: ID!
  # The literal name of the #serviceId.
  serviceName: String!
  # The service's Layer name.
  # Service could have multiple layers, this is the layer of the service that the instance belongs to.
  layer: String!
  normal: Boolean!
}

type HierarchyServiceRelation {
  upperService: HierarchyRelatedService!
  lowerService: HierarchyRelatedService!
}

type HierarchyInstanceRelation {
  upperInstance: HierarchyRelatedInstance!
  lowerInstance: HierarchyRelatedInstance!
}

type ServiceHierarchy {
  relations: [HierarchyServiceRelation!]!
}

type InstanceHierarchy {
  relations: [HierarchyInstanceRelation!]!
}

type LayerLevel {
  # The layer name.
  layer: String!
  # The layer level.
  # The level of the upper service should greater than the level of the lower service.
  level: Int!
}

extend type Query {
  # Query the service hierarchy, based on the given service. Will recursively return all related layers services in the hierarchy.
  getServiceHierarchy(serviceId: ID!, layer: String!): ServiceHierarchy!
  # Query the instance hierarchy, based on the given instance. Will return all direct related layers instances in the hierarchy, no recursive.
  getInstanceHierarchy(instanceId: ID!, layer: String!): InstanceHierarchy!
  # List layer hierarchy levels. The layer levels are defined in the `hierarchy-definition.yml`.
  listLayerLevels: [LayerLevel!]!
}

New data models

  • service_hierarchy_relation

    Column name Data type Description
    id String serviceId.servicelayer-relatedServiceId.relatedServiceLayer
    service_id String upper service id
    service_layer int upper service layer value
    related_service_id String lower service id
    related_service_layer int lower service layer value
    time_bucket long

  • instance_hierarchy_relation

    Column name Data type Description
    id String instanceId.servicelayer-relateInstanceId.relatedServiceLayer
    instance_id String upper instance id
    service_layer int upper service layer value
    related_instance_id String lower instance id
    related_service_layer int lower service layer value
    time_bucket long

Internal APIs

Internal APIs should be exposed in the Core module to support building the hierarchy relationship.

java
public void toServiceHierarchyRelation(String upperServiceName, Layer upperServiceLayer, String lowerServiceName, Layer lowerServiceLayer);
public void toInstanceHierarchyRelation(String upperInstanceName, String upperServiceName, Layer upperServiceLayer, String lowerInstanceName, String lowerServiceName, Layer lowerServiceLayer);

Hierarchy Definition

All layers hierarchy relations are defined in the hierarchy-definition.yml file.
OAP will check the hierarchy relations before building and use the matching rules to auto match the relations. Here is an example:

yaml
# Define the hierarchy of service layers, the layers under the specific layer are related lower of the layer.
# The relation could have a matching rule for auto matching, which are defined in the `auto-matching-rules` section.
# All the layers are defined in the file `org.apache.skywalking.oap.server.core.analysis.Layers.java`.

hierarchy:
  MESH:
    MESH_DP: name
    K8S_SERVICE: short-name

  MESH_DP:
    K8S_SERVICE: short-name

  GENERAL:
    K8S_SERVICE: lower-short-name-remove-ns

  MYSQL:
    K8S_SERVICE: ~

  VIRTUAL_DATABASE:
    MYSQL: ~

# Use Groovy script to define the matching rules, the input parameters are the upper service(u) and the lower service(l) and the return value is a boolean.
# which are used to match the relation between the upper service(u) and the lower service(l) on the different layers.
auto-matching-rules:
  # the name of the upper service is equal to the name of the lower service
  name: "{ (u, l) -> u.name == l.name }"
  # the short name of the upper service is equal to the short name of the lower service
  short-name: "{ (u, l) -> u.shortName == l.shortName }"
  # remove the namespace from the lower service short name
  lower-short-name-remove-ns: "{ (u, l) -> u.shortName == l.shortName.substring(0, l.shortName.lastIndexOf('.')) }"

# The hierarchy level of the service layer, the level is used to define the order of the service layer for UI presentation,
# The level of the upper service should greater than the level of the lower service in `hierarchy` section.
layer-levels:
  MESH: 3
  GENERAL: 3
  VIRTUAL_DATABASE: 3
  MYSQL: 2
  MESH_DP: 1
  K8S_SERVICE: 0

General usage docs

This proposal doesn't impact the end user in any way of using SkyWalking. The remarkable change will be in the UI. On
the service dashboard and topology map, the user should be able to see the hierarchy relationship, which means other
services in other layers are logically the same as the current one. UI would provide the link to jump to the relative
service's dashboard.

No Goal

This proposal doesn't cover all the logic about how to detect the service hierarchy structure. All those should be in a
separate SWIP.

File: docs/en/swip/SWIP-2.md

Collecting and Gathering Kubernetes Monitoring Data

Motivation

SkyWalking has provided an access log collector based on the Agent layer and Service Mesh layer,
and can generate corresponding topology maps and metrics based on the data. However, the Kubernetes Layer still lacks
corresponding access log collector and analysis work.

This proposal is dedicated to collecting and analyzing network access logs in Kubernetes.

Architecture Graph

There is no significant architecture-level change. Still using the Rover project to collect data and report it to
SkyWalking OAP using the gRPC protocol.

Propose Changes

Based on the content in Motivation, if we want to ignore the application types(different program languages) and
only monitor network logs, using eBPF is a good choice. It mainly reflects in the following aspects:

  1. Non-intrusive: When monitoring network access logs with eBPF, the application do not need to make any changes to be monitored.
  2. Language-unrestricted: Regardless of which programming language is used in the application, network data will ultimately be accessed through Linux Syscalls. Therefore, we can monitor network data by attaching eBPF to the syscalls layer, thus ignoring programming languages.
  3. Kernel interception: Since eBPF can attach to the kernel methods, it can obtain the execution status of each packet at L2-L4 layers and generate more detailed metrics.

Based on these reasons and collected data, they can be implemented in SkyWalking Rover and collected and monitored based on the following steps:

  1. Monitor the network execution status of all processes in Kubernetes when the Rover system starts.
  2. Periodically report data content via gRPC protocol to SkyWalking OAP.
  3. SkyWalking OAP parses network access logs and generates corresponding network topology, metrics, etc.

Limitation

For content that uses TLS for data transmission, Rover will detect whether the current language uses libraries
such as OpenSSL. If it is used, it will asynchronously intercept relevant OpenSSL methods
when the process starts to perceive the original data content.

However, this approach is not feasible for Java because Java does not use the OpenSSL library but performs encryption/decryption
through Java code. Currently, eBPF cannot intercept Java method calls. Therefore, it results in an inability to perceive
the TLS data protocol in Java.

Service with Istio sidecar scenario

If the Service is deployed in Istio sidecar, it will still monitor each process.
If the Service is a Java service and uses TLS, it can analyze the relevant traffic generated in the sidecar (envoy).

Imported Dependencies libs and their licenses.

No new library is planned to be added to the codebase.

Compatibility

About the protocol, there should be no breaking changes, but enhancements only:

  1. Rover: adding a new gRPC data collection protocol for reporting the access logs.
  2. OAP: It should have no protocol updates. The existing query protocols are already sufficient for querying Kubernetes topology and metric data.

Data Generation

Entity

  • service_traffic
column data type value description
name string kubernetes service name
short_name string same with name
service_id string base64(name).1
group string empty string
layer string KUBERNETES
  • instance_traffic
column data type value description
service_id string base64(service_name).1
name string pod name
last_ping long last access log message timestamp(millisecond)
properties json empty string
  • endpoint_traffic
column data type value description
service_id string base64(service_name).1
name string access log endpoint name(for HTTP1, is URI)

Entity Relation

All entity information is built on connections. If the target address is remote, the name will be resolved in the following order:

  1. If it is a pod IP, it will be resolved as pod information.
  2. If it is a service IP, it will be resolved as service information.
  3. If neither exists, only pod information will be displayed.

Different entities have different displays for remote addresses. Please refer to the following table.

table name remote info(display by following order)
service_relation service name, remote IP address
instance_relation pod name, remote IP address

NOTICE: If it is the internal data interaction within the pod, such as exchanging data between services and sidecar (envoy),
no corresponding traffic will be generated. We only generate and interact with external pods.

Limitation

If the service IP is used to send requests to the upstream, we will use eBPF to perceive the real target PodIP by
perceiving relevant conntrack records.

However, if conntrack technology is not used, it is difficult to perceive the real target IP address.
In this case, instance relation data of this kind will be dropped, but we will mark all discarded relationship
generation counts through a metric for better understanding of the situation.

Metrics

Integrate the data into the OAL system and generate corresponding metrics through predefined data combined with OAL statements.

General usage docs

This proposal will only add a module to Rover that explains the configuration of access logs, and changes in the Kubernetes module on the UI.

In the Kubernetes UI, users can see the following additions:

  1. Topology: A topology diagram showing the calling relationships between services, instances, and processes.
  2. Entity Metrics: Metric data for services, instances, and processes.
  3. Call Relationship Metrics: Metrics for call relationships between different entities.

File: docs/en/swip/SWIP-3.md

Support RocketMQ Monitoring

Motivation

RocketMQ is a cloud native messaging and streaming platform, making it simple to build event-driven applications. Now that Skywalking can monitor OpenTelemetry metrics, I want to add RocketMQ monitoring via the OpenTelemetry Collector, which fetches metrics from the RocketMQ Exporter

Architecture Graph

There is no significant architecture-level change.

Proposed Changes

rocketmq-exporter collects metrics from RocketMQ and transport the data to OpenTelemetry collector, using SkyWalking openTelemetry receiver to receive these metrics。
Provide cluster, broker, and topic dimensions monitoring.

RocketMQ Cluster Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
Messages Produced Today Count meter_rocketmq_cluster_messages_produced_today The number of cluster messages produced today. RocketMQ Exporter
Messages Consumed Today Count meter_rocketmq_cluster_messages_consumed_today The number of cluster messages consumed today. RocketMQ Exporter
Total Producer Tps Msg/sec meter_rocketmq_cluster_total_producer_tps The number of messages produced per second. RocketMQ Exporter
Total Consume Tps Msg/sec meter_rocketmq_cluster_total_consumer_tps The number of messages consumed per second. RocketMQ Exporter
Producer Message Size Bytes/sec meter_rocketmq_cluster_producer_message_size The max size of a message produced per second. RocketMQ Exporter
Consumer Message Size Bytes/sec meter_rocketmq_cluster_consumer_message_size The max size of the consumed message per second. RocketMQ Exporter
Messages Produced Until Yesterday Count meter_rocketmq_cluster_messages_produced_until_yesterday The total number of messages put until 12 o'clock last night. RocketMQ Exporter
Messages Consumed Until Yesterday Count meter_rocketmq_cluster_messages_consumed_until_yesterday The total number of messages read until 12 o'clock last night. RocketMQ Exporter
Max Consumer Latency ms meter_rocketmq_cluster_max_consumer_latency The max number of consumer latency. RocketMQ Exporter
Max CommitLog Disk Ratio % meter_rocketmq_cluster_max_commitLog_disk_ratio The max utilization ratio of the commit log disk. RocketMQ Exporter
CommitLog Disk Ratio % meter_rocketmq_cluster_commitLog_disk_ratio The utilization ratio of the commit log disk per broker IP. RocketMQ Exporter
Pull ThreadPool Queue Head Wait Time ms meter_rocketmq_cluster_pull_threadPool_queue_head_wait_time The wait time in milliseconds for pulling threadPool queue per broker IP. RocketMQ Exporter
Send ThreadPool Queue Head Wait Time ms meter_rocketmq_cluster_send_threadPool_queue_head_wait_time The wait time in milliseconds for sending threadPool queue per broker IP. RocketMQ Exporter
Topic Count Count meter_rocketmq_cluster_topic_count The number of topics that received messages from the producer. RocketMQ Exporter
Broker Count Count meter_rocketmq_cluster_broker_count The number of brokers that received messages from the producer. RocketMQ Exporter

RocketMQ Broker Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
Produce TPS Msg/sec meter_rocketmq_broker_produce_tps The number of broker produces messages per second. RocketMQ Exporter
Consume QPS Msg/sec meter_rocketmq_broker_consume_qps The number of broker consumes messages per second. RocketMQ Exporter
Producer Message Size Bytes/sec meter_rocketmq_broker_producer_message_size The max size of the messages produced per second. RocketMQ Exporter
Consumer Message Size Bytes/sec meter_rocketmq_broker_consumer_message_size The max size of the messages consumed per second. RocketMQ Exporter

RocketMQ Topic Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
Max Producer Message Size Byte meter_rocketmq_topic_max_producer_message_size The maximum number of messages produced. RocketMQ Exporter
Max Consumer Message Size Byte meter_rocketmq_topic_max_consumer_message_size The maximum number of messages consumed. RocketMQ Exporter
Consumer Latency ms meter_rocketmq_topic_consumer_latency Consumption delay time of a consumer group. RocketMQ Exporter
Producer Tps Msg/sec meter_rocketmq_topic_producer_tps The number of messages produced per second. RocketMQ Exporter
Consumer Group Tps Msg/sec meter_rocketmq_topic_consumer_group_tps The number of messages consumed per second per consumer group. RocketMQ Exporter
Producer Offset Count meter_rocketmq_topic_producer_offset The max progress of a topic's production message. RocketMQ Exporter
Consumer Group Offset Count meter_rocketmq_topic_consumer_group_offset The max progress of a topic's consumption message per consumer group. RocketMQ Exporter
Producer Message Size Byte/sec meter_rocketmq_topic_producer_message_size The max size of messages produced per second. RocketMQ Exporter
Consumer Message Size Byte/sec meter_rocketmq_topic_consumer_message_size The max size of messages consumed per second. RocketMQ Exporter
Consumer Group_Count Count meter_rocketmq_topic_consumer_group_count The number of consumer groups. RocketMQ Exporter
Broker Count Count meter_rocketmq_topic_broker_count The number of topics that received messages from the producer. RocketMQ Exporter

Imported Dependencies libs and their licenses.

No new dependency.

Compatibility

no breaking changes.

General usage docs

This feature is out of the box.

File: docs/en/swip/SWIP-4.md

Support available layers of service in the topology.

Motivation

UI could jump to the service dashboard and query service hierarchy from the topology node.
For now topology node includes name and ID but without layer, as the service could have multiple layers,
the limitation is that it is only works on the current layer which the topology represents:

  1. UI could not jump into another layer's dashboard of the service.
  2. UI could not query the service hierarchy from the topology node if the node is not in current layer.

Here are typical use cases:
should have a chance to jump into another layer's dashboard of the service:

  1. In the mesh topology, mesh(layer MESH) and mesh-dp(layer MESH_DP) share a similar topology, one node will have two layers.
  2. In the mesh topology, agent(layer GENERAL) + virtual database(layer VIRTUAL_DATABASE), the node is in different layers.

Both of these two cases have hybrid layer topology. If we could support that, we could have a better x-layer interaction.

Architecture Graph

There is no significant architecture-level change.

Propose Changes

Add the layers info into topology node:

  1. When building the topology node fetch the layers info from the service according to the service id.
  2. Return layers info in the Node when query the topology.

Imported Dependencies libs and their licenses.

No new library is planned to be added to the codebase.

Compatibility

About the protocol, there should be no breaking changes, but enhancements only. New field layers is going to be added to the
Node in the query protocol topology.graphqls.

graphql
type Node {
  # The service ID of the node.
  id: ID!
  # The literal name of the #id.
  name: String!
  # The type name may be
  # 1. The service provider/middleware tech, such as: Tomcat, SpringMVC
  # 2. Conjectural Service, e.g. MySQL, Redis, Kafka
  type: String
  # It is a conjecture node or real node, to represent a service or endpoint.
  isReal: Boolean!
  # The layers of the service.
  layers: [String!]!
}

General usage docs

This proposal doesn't impact the end user in any way of using SkyWalking. The remarkable change will be in the UI topology map,
users could jump into the proper layer's service dashboard and query the service hierarchy from the topology node.

File: docs/en/swip/SWIP-5.md

Support ClickHouse Monitoring

Motivation

ClickHouse is a high-performance, column-oriented SQL database management system (DBMS) for online analytical processing (OLAP). It is available as both an open-source software and a cloud offering.

Now I want to add ClickHouse monitoring via the OpenTelemetry Collector which fetches metrics from it's own HTTP endpoint to expose metrics data for Prometheus (since ClickHouse v20.1.2.4). Clickhouse Exporter used only for old ClickHouse versions, modern versions have embedded prometheus endpoint.

Architecture Graph

There is no significant architecture-level change.

Proposed Changes

ClickHouse expose own metrics via HTTP endpoint to opentelemetry collector, using skyWalking openTelemetry receiver to fetch these metrics.

The exposed metrics are from the system.metrics table / the system.events table / the system.asynchronous_metrics table.

ClickHouse Instance Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
CpuUsage count meter_clickhouse_instance_cpu_usage CPU time spent seen by OS per second(according to ClickHouse.system.dashboard.CPU Usage (cores)). ClickHouse
MemoryUsage percentage meter_clickhouse_instance_memory_usage Total amount of memory (bytes) allocated by the server/ total amount of OS memory. ClickHouse
MemoryAvailable percentage meter_clickhouse_instance_memory_available Total amount of memory (bytes) available for program / total amount of OS memory. ClickHouse
Uptime sec meter_clickhouse_instance_uptime The server uptime in seconds. It includes the time spent for server initialization before accepting connections. ClickHouse
Version string meter_clickhouse_instance_version Version of the server in a single integer number in base-1000. ClickHouse
FileOpen count meter_clickhouse_instance_file_open Number of files opened. ClickHouse

ClickHouse Network Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
TcpConnections count meter_clickhouse_instance_tcp_connectionsmeter_clickhouse_tcp_connections Number of connections to TCP server. ClickHouse
MysqlConnections count meter_clickhouse_instance_mysql_connectionsmeter_clickhouse_mysql_connections Number of client connections using MySQL protocol. ClickHouse
HttpConnections count meter_clickhouse_instance_http_connectionsmeter_clickhouse_mysql_connections Number of connections to HTTP server. ClickHouse
InterserverConnections count meter_clickhouse_instance_interserver_connectionsmeter_clickhouse_interserver_connections Number of connections from other replicas to fetch parts. ClickHouse
PostgresqlConnections count meter_clickhouse_instance_postgresql_connectionsmeter_clickhouse_postgresql_connections Number of client connections using PostgreSQL protocol. ClickHouse
ReceiveBytes bytes meter_clickhouse_instance_network_receive_bytesmeter_clickhouse_network_receive_bytes Total number of bytes received from network. ClickHouse
SendBytes bytes meter_clickhouse_instance_network_send_bytesmeter_clickhouse_network_send_bytes Total number of bytes send to network. ClickHouse

ClickHouse Query Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
QueryCount count meter_clickhouse_instance_querymeter_clickhouse_query Number of executing queries. ClickHouse
SelectQueryCount count meter_clickhouse_instance_query_selectmeter_clickhouse_query_select Number of executing queries, but only for SELECT queries. ClickHouse
InsertQueryCount count meter_clickhouse_instance_query_insertmeter_clickhouse_query_insert Number of executing queries, but only for INSERT queries. ClickHouse
SelectQueryRate count/sec meter_clickhouse_instance_query_select_ratemeter_clickhouse_query_select_rate Number of SELECT queries per second. ClickHouse
InsertQueryRate count/sec meter_clickhouse_instance_query_insert_ratemeter_clickhouse_query_insert_rate Number of INSERT queries per second. ClickHouse
Querytime microsec meter_clickhouse_instance_querytime_microsecondsmeter_clickhouse_querytime_microseconds Total time of all queries. ClickHouse
SelectQuerytime microsec meter_clickhouse_instance_querytime_select_microsecondsmeter_clickhouse_querytime_select_microseconds Total time of SELECT queries. ClickHouse
InsertQuerytime microsec meter_clickhouse_instance_querytime_insert_microsecondsmeter_clickhouse_querytime_insert_microseconds Total time of INSERT queries. ClickHouse
OtherQuerytime microsec meter_clickhouse_instance_querytime_other_microsecondsmeter_clickhouse_querytime_other_microseconds Total time of queries that are not SELECT or INSERT. ClickHouse
QuerySlowCount count meter_clickhouse_instance_query_slowmeter_clickhouse_query_slow Number of reads from a file that were slow. ClickHouse

ClickHouse Insertion Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
InsertQueryCount count meter_clickhouse_instance_query_insertmeter_clickhouse_query_insert Number of executing queries, but only for INSERT queries. ClickHouse
InsertedRowCount count meter_clickhouse_instance_inserted_rowsmeter_clickhouse_inserted_rows Number of rows INSERTed to all tables. ClickHouse
InsertedBytes bytes meter_clickhouse_instance_inserted_bytesmeter_clickhouse_inserted_bytes Number of bytes INSERTed to all tables. ClickHouse
DelayedInsertCount count meter_clickhouse_instance_delayed_insertmeter_clickhouse_delayed_insert Number of times the INSERT of a block to a MergeTree table was throttled due to high number of active data parts for partition. ClickHouse

ClickHouse Replicas Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
ReplicatedChecks count meter_clickhouse_instance_replicated_checksmeter_clickhouse_replicated_checks Number of data parts checking for consistency. ClickHouse
ReplicatedFetch count meter_clickhouse_instance_replicated_fetchmeter_clickhouse_replicated_fetch Number of data parts being fetched from replica. ClickHouse
ReplicatedSend count meter_clickhouse_instance_replicated_sendmeter_clickhouse_replicated_send Number of data parts being sent to replicas. ClickHouse

ClickHouse MergeTree Supported Metrics

Monitoring Panel Unit Metric Name Description Data Source
BackgroundMergeCount count meter_clickhouse_instance_background_mergemeter_clickhouse_background_merge Number of executing background merges. ClickHouse
MergeRows count meter_clickhouse_instance_merge_rowsmeter_clickhouse_merge_rows Rows read for background merges. This is the number of rows before merge. ClickHouse
MergeUncompressedBytes bytes meter_clickhouse_instance_merge_uncompressed_bytesmeter_clickhouse_merge_uncompressed_bytes Uncompressed bytes (for columns as they stored in memory) that was read for background merges. This is the number before merge. ClickHouse
MoveCount count meter_clickhouse_instance_movemeter_clickhouse_move Number of currently executing moves. ClickHouse
PartsActive Count meter_clickhouse_instance_parts_activemeter_clickhouse_parts_active Active data part, used by current and upcoming SELECTs. ClickHouse
MutationsCount count meter_clickhouse_instance_mutationsmeter_clickhouse_mutations Number of mutations (ALTER DELETE/UPDATE). ClickHouse

ClickHouse Kafka Table Engine Supported Metrics

When table engine works
with Apache Kafka.

Kafka lets you:

  • Publish or subscribe to data flows.
  • Organize fault-tolerant storage.
  • Process streams as they become available.
Monitoring Panel Unit Metric Name Description Data Source
KafkaMessagesRead count meter_clickhouse_instance_kafka_messages_readmeter_clickhouse_kafka_messages_read Number of Kafka messages already processed by ClickHouse. ClickHouse
KafkaWrites count meter_clickhouse_instance_kafka_writesmeter_clickhouse_kafka_writes Number of writes (inserts) to Kafka tables. ClickHouse
KafkaConsumers count meter_clickhouse_instance_kafka_consumersmeter_clickhouse_kafka_consumers Number of active Kafka consumers. ClickHouse
KafkaProducers count meter_clickhouse_instance_kafka_producersmeter_clickhouse_kafka_producers Number of active Kafka producer created. ClickHouse

ClickHouse ZooKeeper Supported Metrics

ClickHouse uses ZooKeeper for storing metadata of replicas when using replicated tables. If replicated tables are not
used, this section of parameters can be omitted.

Monitoring Panel Unit Metric Name Description Data Source
ZookeeperSession count meter_clickhouse_instance_zookeeper_sessionmeter_clickhouse_zookeeper_session Number of sessions (connections) to ZooKeeper. ClickHouse
ZookeeperWatch count meter_clickhouse_instance_zookeeper_watchmeter_clickhouse_zookeeper_watch Number of watches (event subscriptions) in ZooKeeper. ClickHouse
ZookeeperBytesSent bytes meter_clickhouse_instance_zookeeper_bytes_sentmeter_clickhouse_zookeeper_bytes_sent Number of bytes send over network while communicating with ZooKeeper. ClickHouse
ZookeeperBytesReceive bytes meter_clickhouse_instance_zookeeper_bytes_receivedmeter_clickhouse_zookeeper_bytes_received Number of bytes send over network while communicating with ZooKeeper. ClickHouse

ClickHouse Keeper Supported Metrics

ClickHouse Keeper provides the coordination system for data replication and distributed DDL queries execution. ClickHouse Keeper is compatible with ZooKeeper.

ClickHouse Keeper can work in embedded mode or standalone cluster mode, the metrics below are for embedded mode.

Monitoring Panel Unit Metric Name Description Data Source
KeeperAliveConnections count meter_clickhouse_instance_keeper_connections_alivemeter_clickhouse_keeper_connections_alive Number of alive connections for embedded ClickHouse Keeper. ClickHouse
KeeperOutstandingRequets count meter_clickhouse_instance_keeper_outstanding_requestsmeter_clickhouse_keeper_outstanding_requests Number of outstanding requests for embedded ClickHouse Keeper. ClickHouse

Imported Dependencies libs and their licenses.

No new dependency.

Compatibility

no breaking changes.

General usage docs

2. Official Technical Reference & Guides (apache/docs)

File: README.md

Apache HoraeDB (incubating) Website

[!IMPORTANT]
Apache HoraeDB (incubating) is an effort undergoing incubation at the Apache
Software Foundation (ASF), sponsored by the Apache Incubator PMC.

Please read the DISCLAIMER and a full explanation of "incubating".

This repository stores all the source files of HoraeDB website, it's built with hugo and docsy.

How to run

bash
# Start a local server for preview
hugo serve

By default hugo will listen on http://localhost:1313

Contributing

Welcome any contributions from the community. You can

  • Open an issue with any suggestions, or
  • Submit a pull request to improve the website (trivial fixes are welcome).

File: content/en/blog/2024/release-2.0.0.md

title: Release 2.0.0 date: 2024-04-23

Upgrade from 1.x.x to 2.0.0

The transition from CeresDB to Apache HoraeDB introduces several breaking changes. To facilitate upgrading from older versions to v2.0.0, specific alterations are necessary.

Upgrade Steps

  1. Setup required envs
text
export HORAEDB_DEFAULT_CATALOG=ceresdb
  1. Update config

Etcd's root should be configured both in horaedb and horaemeta

For horaedb

text
[cluster_deployment.etcd_client]
server_addrs = ['127.0.0.1:2379']
root_path = "/rootPath"

For horaemeta

text
storage-root-path = "/rootPath"
  1. Upgrade horaemeta

Horaedb will throw following errors, which is expected

text
2024-01-23 14:37:57.726 ERRO [src/cluster/src/cluster_impl.rs:136] Send heartbeat to meta failed, err:Failed to send heartbeat, cluster:defaultCluster, err:status: Unimplemented, message: "unknown service meta_service.MetaRpcService", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
  1. Upgrade horaedb

After all server upgraded, the cluster should be ready for read/write, and old data could be queried like before.

What's Changed

Breaking Changes

Features

Refactor

Fixed

Docs

Chore

New Contributors

Full Changelog: https://github.com/apache/horaedb/compare/v1.2.7...v2.0.0

File: content/en/blog/2024/release-2.1.0.md

title: Release 2.1.0
date: 2024-11-18
tags:

  • release

The Apache HoraeDB(incubating) team are pleased to announce that v2.1.0 is released, which has closed over 60 issues, including two major features:

1. New WAL implementation based on local disk.

In previous version, there is a RocksDB-based WAL. Although it works well in most cases, it has following issues:

  • Compiling from source can be a challenging task, especially since RocksDB is primarily written in C++.
  • For WAL, RocksDB can be somewhat overkill. If you are not familiar with RocksDB, tuning it can be very challenging.

With this new WAL, the above two problems are solved very well, and in performance test, the new WAL slightly outperforms the previous implementation, giving a solid foundation for future optimizations.

Interested readers can refer to the design documentation [here]({{< ref "wal_on_disk" >}}) for more details on this feature.

How to enable

text
[analytic.wal]
type = "Local"
data_dir = "/path/to/local/wal"

2. Access object store with [Apache OpenDAL](https://github.com/apache/opendal)

OpenDAL (Open Data Access Layer) is a project that provides a unified API for accessing various data storage backends.
It offers several advantages for developers and organizations. Here are some key benefits:

  • Unified API. OpenDAL provides a consistent and unified API for accessing different storage backends, such as AWS S3, Azure Blob Storage, and local file systems.
  • Optimized for Efficiency: OpenDAL is built with performance in mind. It includes optimizations to ensure efficient data access and manipulation, making it suitable for high-performance applications.
  • Comprehensive Documentation: The project provides detailed documentation, making it easier for developers to get started and understand how to use the library effectively.

In newer versions of OpenDAL, object_store integration is provided, which is very beneficial for HoraeDB code migration, as the APIs used by the upper layers remain virtually unchanged, and only the object store part needs to be abstracted to a unified OpenDAL operator:

rust
// Create a new operator
let operator = Operator::new(S3::default())?.finish();

// Create a new object store
let object_store = Arc::new(OpendalStore::new(operator));

Additionally, since the Apache OpenDAL implementation of object_store is based on the latest version of the object_store, which has breaking changes from the previous version that HoraeDB is using, we've chosen to make it compatible in order to keep the scope of this upgrade as manageable as possible.

In the process of adapting to the new API, the put_multipart interface has changed the most, so the main adaptation logic is also here, HoraeDB's approach is to encapsulate the underlying put_multipart interface to ensure that the upper layer code is not modified, the details can be found in the reference:

https://github.com/apache/horaedb/blob/v2.1.0/src/components/object_store/src/multi_part.rs

Note: The adaptation logic is only practical when parquet version < 52.0.0.

Download

Go to download pages.

Conclusion

Other bug fixes and improvements can be seen here:

As always, we warmly welcome you to join our community and share your insights.

File: content/en/blog/_index.md

title: "Blog"
weight: 1
simple_list: true
menu:
main:
weight: 30
pre:

File: content/en/community/_index.md

title: "Community"
params:
contributingUrl: https://github.com/apache/horaedb/blob/main/CONTRIBUTING.md
menu:
main:
weight: 40

{{% blocks/section color="white" %}}

How to Subscribe Mail List

Mailing lists are a form of communication used by the Apache community. Generally speaking, many things in the Apache community are hosted by mailing lists, For example: project q&A, technical discussion, transaction decision, release vote, etc，By subscribing, you can get the first update of the HoraeDB community and keep up with the community. The following list is the main list we are using:

In order to subscribe to the [email protected] mailing list, the steps are as follows：

  1. Send an email without any content or subject: [email protected]
  2. Wait until you receive an email with the subject line confirm subscribe to [email protected] (if you have not received it for a long time, please confirm whether the email is blocked by your email, if you have not been blocked and will receive a reply for a long time, return to step 1)
  3. Reply directly to the email without changing the subject line or adding the email content.
  4. Wait until you receive an email with the subject line WELCOME to [email protected] .
  5. If you receive an email from dev, you have successfully subscribed to the email. To initiate a discussion, you can send an email directly to [email protected], which will be sent to everyone who subscribed to the mailing list.

Unsubscribe from the mail list

The steps for unsubscribing to a mailing list are similar to those for subscribing to a mailing list:

  1. Send an email without any content or subject to: [email protected]
  2. Wait until you receive an email with the subject line confirm unsubscribe from [email protected]
  3. Reply directly to the email without changing the subject line or adding the email content
  4. Wait until you receive an email with the subject line GOODBYE from [email protected]
  5. Unsubscribe success

{{% /blocks/section %}}

File: content/en/docs/design/architecture.md

title: "Introduction to HoraeDB's Architecture"

Target

  • Provide the overview of HoraeDB to the developers who want to know more about HoraeDB but have no idea where to start.
  • Make a brief introduction to the important modules of HoraeDB and the connections between these modules but details about their implementations are not be involved.

Motivation

HoraeDB is a timeseries database (TSDB). However, HoraeDB's goal is to handle both timeseries and analytic workloads compared with the classic TSDB, which usually have a poor performance in handling analytic workloads.

In the classic timeseries database, the Tag columns (InfluxDB calls them Tag and Prometheus calls them Label) are normally indexed by generating an inverted index. However, it is found that the cardinality of Tag varies in different scenarios. And in some scenarios the cardinality of Tag is very high (we name this case after analytic workload), and it takes a very high cost to store and retrieve the inverted index. On the other hand, it is observed that scanning+pruning often used by the analytical databases can do a good job to handle such analytic workload.

The basic design idea of HoraeDB is to adopt a hybrid storage format and the corresponding query method for a better performance in processing both timeseries and analytic workloads.

Architecture

plaintext
┌──────────────────────────────────────────┐
│       RPC Layer (HTTP/gRPC/MySQL)        │
└──────────────────────────────────────────┘
┌──────────────────────────────────────────┐
│                 SQL Layer                │
│ ┌─────────────────┐  ┌─────────────────┐ │
│ │     Parser      │  │     Planner     │ │
│ └─────────────────┘  └─────────────────┘ │
└──────────────────────────────────────────┘
┌───────────────────┐  ┌───────────────────┐
│    Interpreter    │  │      Catalog      │
└───────────────────┘  └───────────────────┘
┌──────────────────────────────────────────┐
│               Query Engine               │
│ ┌─────────────────┐  ┌─────────────────┐ │
│ │    Optimizer    │  │    Executor     │ │
│ └─────────────────┘  └─────────────────┘ │
└──────────────────────────────────────────┘
┌──────────────────────────────────────────┐
│         Pluggable Table Engine           │
│  ┌────────────────────────────────────┐  │
│  │              Analytic              │  │
│  │┌────────────────┐┌────────────────┐│  │
│  ││      Wal       ││    Memtable    ││  │
│  │└────────────────┘└────────────────┘│  │
│  │┌────────────────┐┌────────────────┐│  │
│  ││     Flush      ││   Compaction   ││  │
│  │└────────────────┘└────────────────┘│  │
│  │┌────────────────┐┌────────────────┐│  │
│  ││    Manifest    ││  Object Store  ││  │
│  │└────────────────┘└────────────────┘│  │
│  └────────────────────────────────────┘  │
│  ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           Another Table Engine        │  │
│  └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
└──────────────────────────────────────────┘

The figure above shows the architecture of HoraeDB stand-alone service and the details of some important modules will be described in the following part.

RPC Layer

module path: https://github.com/apache/incubator-horaedb/tree/main/server

The current RPC supports multiple protocols including HTTP, gRPC, MySQL.

Basically, HTTP and MySQL are used to debug HoraeDB, query manually and perform DDL operations (such as creating, deleting tables, etc.). And gRPC protocol can be regarded as a customized protocol for high-performance, which is suitable for massive reading and writing operations.

SQL Layer

module path: https://github.com/apache/incubator-horaedb/tree/main/query_frontend

SQL layer takes responsibilities for parsing sql and generating the query plan.

Based on sqlparser a sql dialect, which introduces some key concepts including Tag and Timestamp, is provided for processing timeseries data. And by utilizing DataFusion the planner is able to generate both regular logical plans and tailored ones which is used to implement the special operators defined by timeseries queries, e.g PromQL.

Interpreter

module path: https://github.com/apache/incubator-horaedb/tree/main/interpreters

The Interpreter module encapsulates the SQL CRUD operations. In the query procedure, a sql received by HoraeDB is parsed, converted into the query plan and then executed in some specific interpreter, such as SelectInterpreter, InsertInterpreter and etc.

Catalog

module path: https://github.com/apache/incubator-horaedb/tree/main/catalog_impls

Catalog is actually the module managing metadata and the levels of metadata adopted by HoraeDB is similar to PostgreSQL: Catalog > Schema > Table, but they are only used as namespace.

At present, Catalog and Schema have two different kinds of implementation for standalone and distributed mode because some strategies to generate ids and ways to persist metadata differ in different mode.

Query Engine

module path: https://github.com/apache/incubator-horaedb/tree/main/query_engine

Query Engine is responsible for optimizing and executing query plan given a basic SQL plan provided by SQL layer and now such work is mainly delegated to DataFusion.

In addition to the basic functions of SQL, HoraeDB also defines some customized query protocols and optimization rules for some specific query plans by utilizing the extensibility provided by DataFusion. For example, the implementation of PromQL is implemented in this way and read it if you are interested.

Pluggable Table Engine

module path: https://github.com/apache/incubator-horaedb/tree/main/table_engine

Table Engine is actually a storage engine for managing tables in HoraeDB and the pluggability of Table Engine is a core design of HoraeDB which matters in achieving our long-term target, e.g supporting handle log or tracing workload by implementing new storage engines. HoraeDB will have multiple kinds of Table Engine for different workloads and the most appropriate one should be chosen as the storage engine according to the workload pattern.

Now the requirements for a Table Engine are:

  • Manage all the shared resources under the engine:
    • Memory
    • Storage
    • CPU
  • Manage metadata of tables such as table schema and table options;
  • Provide Table instances which provides read and write methods;
  • Take responsibilities for creating, opening, dropping and closing Table instance;
  • ....

Actually the things that a Table Engine needs to process are a little complicated. And now in HoraeDB only one Table Engine called Analytic is provided and does a good job in processing analytical workload, but it is not ready yet to handle the timeseries workload (we plan to enhance it for a better performance by adding some indexes which help handle timeseries workload).

The following part gives a description about details of Analytic Table Engine.

WAL

module path: https://github.com/apache/incubator-horaedb/tree/main/wal

The model of HoraeDB processing data is WAL + MemTable that the recent written data is written to WAL first and then to MemTable and after a certain amount of data in MemTable is accumulated, the data will be organized in a query-friendly form to persistent devices.

Now three implementations of WAL are provided for standalone and distributed mode:

  • For standalone mode, WAL is based on RocksDB and data is persisted on the local disk.
  • For distributed mode, WAL is required as a distributed component and to be responsible for durability of the newly written data, so now we provide an implementation based on OceanBase.
  • For distributed mode, in addition to OceanBase, we also provide a more lightweight implementation based on Apache Kafka.

MemTable

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/memtable

For WAL can't provide efficient data retrieval, the newly written data is also stored in Memtable for efficient data retrieval, after a certain amount of data is reached, HoraeDB organizes the data in MemTable into a query-friendly storage format (SST) and stores it to the persistent device.

The current implementation of MemTable is based on agatedb's skiplist. It allows concurrent reads and writes and can control memory usage based on Arena.

Flush

module path: https://github.com/apache/incubator-horaedb/blob/main/analytic_engine/src/instance/flush_compaction.rs

What Flush does is that when the memory usage of MemTable reaches the threshold, some MemTables are selected for flushing into query-friendly SSTs saved on persistent device.

During the flushing procedure, the data will be divided by a certain time range (which is configured by table option Segment Duration), and any SST is ensured that the timestamps of the data in it are in the same Segment. Actually this is also a common operation in most timeseries databases which organizes data in the time dimension to speed up subsequent time-related operations, such as querying data over a time range and assisting purge data outside the TTL.

Compaction

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/compaction

The data of MemTable is flushed as SSTs, but the file size of recently flushed SST may be very small. And too small or too many SSTs lead to the poor query performance. Therefore, Compaction is then introduced to rearrange the SSTs so that the multiple smaller SST files can be compacted into a larger SST file.

Manifest

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/meta

Manifest takes responsibilities for managing tables' metadata of Analytic Engine including:

  • Table schema and table options;
  • The sequence number where the newest flush finishes;
  • The information of all the SSTs belonging to the table.

Now the Manifest is based on WAL and Object Storage. The newly written updates on the Manifest are persisted as logs in WAL, and in order to avoid infinite expansion of Manifest (actually every Flush leads to an update), Snapshot is also introduced to clean up the history of metadata updates, and the generated Snapshot will be saved to Object Storage.

Object Storage

module path: https://github.com/apache/incubator-horaedb/tree/main/components/object_store

The SST generated by Flush needs to be persisted and the abstraction of the persistent storage device is ObjectStore including multiple implementations:

The distributed architecture of HoraeDB separates storage and computing, which requires Object Store needs to be a highly available and reliable service independent of HoraeDB. Therefore, storage systems like Amazon S3, Alibaba Cloud OSS is a good choice and in the future implementations on storage systems of some other cloud service providers is planned to provide.

SST

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/sst

SST is actually an abstraction that can have multiple specific implementations. The current implementation is based on Parquet, which is a column-oriented data file format designed for efficient data storage and retrieval.

The format of SST is very critical for retrieving data and is also the most important part to perform well in handling both timeseries and analytic workloads. At present, our Parquet-based implementation is good at processing analytic workload but is poor at processing timeseries workload. In our roadmap, we will explore more storage formats in order to achieve a good performance in both workloads.

Space

module path: https://github.com/apache/incubator-horaedb/blob/main/analytic_engine/src/space.rs

In Analytic Engine, there is a concept called space and here is an explanation for it to resolve some ambiguities when read source code. Actually Analytic Engine does not have the concept of catalog and schema and only provides two levels of relationship: space and table. And in the implementation, the schema id (which should be unique across all catalogs) on the upper layer is actually mapped to space id.

The space in Analytic Engine serves mainly for isolation of resources for different tenants, such as the usage of memory.

Critical Path

After a brief introduction to some important modules of HoraeDB, we will give a description for some critical paths in code, hoping to provide interested developers with a guide for reading the code.

Query

plaintext
┌───────┐      ┌───────┐      ┌───────┐
│       │──1──▶│       │──2──▶│       │
│Server │      │  SQL  │      │Catalog│
│       │◀─10──│       │◀─3───│       │
└───────┘      └───────┘      └───────┘
                │    ▲
               4│   9│
                │    │
                ▼    │
┌─────────────────────────────────────┐
│                                     │
│             Interpreter             │
│                                     │
└─────────────────────────────────────┘
                           │    ▲
                          5│   8│
                           │    │
                           ▼    │
                   ┌──────────────────┐
                   │                  │
                   │   Query Engine   │
                   │                  │
                   └──────────────────┘
                           │    ▲
                          6│   7│
                           │    │
                           ▼    │
 ┌─────────────────────────────────────┐
 │                                     │
 │            Table Engine             │
 │                                     │
 └─────────────────────────────────────┘

Take SELECT SQL as an example. The figure above shows the query procedure and the numbers in it indicates the order of calling between the modules.

Here are the details:

  • Server module chooses a proper rpc module (it may be HTTP, gRPC or mysql) to process the requests according the protocol used by the requests;
  • Parse SQL in the request by the parser;
  • With the parsed sql and the information provided by catalog/schema module, DataFusion can generate the logical plan;
  • With the logical plan, the corresponding Interpreter is created and logical plan will be executed by it;
  • For the logical plan of normal Select SQL, it will be executed through SelectInterpreter;
  • In the SelectInterpreter the specific query logic is executed by the Query Engine:
    • Optimize the logical plan;
    • Generate the physical plan;
    • Optimize the physical plan;
    • Execute the physical plan;
  • The execution of physical plan involves Analytic Engine:
    • Data is obtained by read method of Table instance provided by Analytic Engine;
    • The source of the table data is SST and Memtable, and the data can be filtered by the pushed down predicates;
    • After retrieving the table data, Query Engine will complete the specific computation and generate the final results;
  • SelectInterpreter gets the results and feeds them to the protocol module;
  • After the protocol layer converts the results, the server module responds to the client with them.

The following is the flow of function calls in version v1.2.2:

text
/* Detailed source-code truncated for AI context efficiency. */
  1. The received request will be forwarded to handle_sql after various protocol conversions, and since the request may not be processed by this node, it may need to be forwarded to maybe_forward_sql_query to handle the forwarding logic.
  2. After constructing the ForwardRequest in maybe_forward_sql_query, call forward
  3. After constructing the RouteRequest in forward, call route
  4. Use route to get the destination node endpoint and return to forward.
  5. Call forward_with_endpoint to forward the request
  6. return forward
  7. return maybe_forward_sql_query
  8. return handle_sql
  9. If this is a Local request, call fetch_sql_query_output to process it
  10. Call parse_sql to parse sql into Statment
  11. return fetch_sql_query_output
  12. Call statement_to_plan with Statment
  13. Construct Planner with ctx and Statment, and call the statement_to_plan method of Planner
  14. The planner will call the corresponding planner method for the requested category, at this point our sql is a query and will call sql_statement_to_plan
  15. Call sql_statement_to_datafusion_plan , which will generate the datafusion object, and then call SqlToRel::sql_statement_to_plan
  16. The generated logical plan is returned from SqlToRel::sql_statement_to_plan
  17. return
  18. return
  19. return
  20. Call execute_plan_involving_partition_table (in the default configuration) for subsequent optimization and execution of this logical plan
  21. Call build_interpreter to generate Interpreter
  22. return
  23. Call Interpreter's interpreter_execute_plan method for logical plan execution.
  24. The corresponding execute function is called, at this time the sql is a query, so the execute of the SelectInterpreter will be called
  25. call execute_logical_plan , which will call build_df_session_ctx to generate the optimizer
  26. build_df_session_ctx will use the config information to generate the corresponding context, first using datafusion and some custom optimization rules (in logical_optimize_rules()) to generate the logical plan optimizer, using apply_adapters_for_physical_optimize_rules to generate the physical plan optimizer
  27. return optimizer
  28. Call optimize_plan, using the optimizer just generated to first optimize the logical plan and then the physical plan
  29. Return to optimized physical plan
  30. execute physical plan
  31. returned after execution
  32. After collecting the results of all slices, return
  33. return
  34. return
  35. return
  36. Return to the upper layer for network protocol conversion and finally return to the request sender

Write

plaintext
┌───────┐      ┌───────┐      ┌───────┐
│       │──1──▶│       │──2──▶│       │
│Server │      │  SQL  │      │Catalog│
│       │◀─8───│       │◀─3───│       │
└───────┘      └───────┘      └───────┘
                │    ▲
               4│   7│
                │    │
                ▼    │
┌─────────────────────────────────────┐
│                                     │
│             Interpreter             │
│                                     │
└─────────────────────────────────────┘
      │    ▲
      │    │
      │    │
      │    │
      │    │       ┌──────────────────┐
      │    │       │                  │
     5│   6│       │   Query Engine   │
      │    │       │                  │
      │    │       └──────────────────┘
      │    │
      │    │
      │    │
      ▼    │
 ┌─────────────────────────────────────┐
 │                                     │
 │            Table Engine             │
 │                                     │
 └─────────────────────────────────────┘

Take INSERT SQL as an example. The figure above shows the query procedure and the numbers in it indicates the order of calling between the modules.

Here are the details:

  • Server module chooses a proper rpc module (it may be HTTP, gRPC or mysql) to process the requests according the protocol used by the requests;
  • Parse SQL in the request by the parser;
  • With the parsed sql and the catalog/schema module, DataFusion can generate the logical plan;
  • With the logical plan, the corresponding Interpreter is created and logical plan will be executed by it;
  • For the logical plan of normal INSERT SQL, it will be executed through InsertInterpreter;
  • In the InsertInterpreter, write method of Table provided Analytic Engine is called:
    • Write the data into WAL first;
    • Write the data into MemTable then;
  • Before writing to MemTable, the memory usage will be checked. If the memory usage is too high, the flush process will be triggered:
    • Persist some old MemTables as SSTs;
    • Store updates about the new SSTs and the flushed sequence number of WAL to Manifest;
    • Delete the corresponding WAL entries;
  • Server module responds to the client with the execution result.

File: content/en/docs/design/clustering.md

title: "Introduction to Architecture of HoraeDB Cluster"
---Note: Some of the features mentioned in the article have not yet been implemented.

Overview

text
/* Detailed source-code truncated for AI context efficiency. */

The diagram above describes the architecture of a HoraeDB cluster, where some key concepts need to be explained:

  • HoraeMeta Cluster: Takes responsibilities for managing the metadata and resource scheduling of the cluster;
  • Shard(L)/Shard(F): Leader shard and follower shard consisting of multiple tables;
  • HoraeDB: One HoraeDB instance consisting of multiple shards;
  • WAL Service: Write-ahead log service for storing new-written real-time data;
  • Object Storage: Object storage service for storing SST converted from memtable;

From the architecture diagram above, it can be concluded that the compute and storage are separated in the HoraeDB cluster, which makes it easy to implement useful distributed features, such as elastic autoscaling of compute/storage resources, high availability, load balancing, and so on.

Let's dive into some of the key components mentioned above before explaining how these features are implemented.

Shard

Shard is the basic scheduling unit in the cluster, which consists of a group of tables. And the tables in a shard share the same region for better storage locality in the WAL Service, and because of this, it is efficient to recover the data of all tables in the shard by scanning the entire WAL region. For most of implementations of WAL Service, without the shard concept, it costs a lot to recover the table data one by one due to massive random IO, and this case will deteriorate sharply when the number of tables grows to a certain level.

A specific role, Leader or Follower, should be assigned to a shard. A pair of leader-follower shards share the same set of tables, and the leader shard can serve the write and query requests from the client while the follower shard can only serve the read-only requests, and must synchronize the newly written data from the WAL service in order to provide the latest snapshot for data retrieval. Actually, the follower is not needed if the high availability is not required, while with at least one follower, it takes only a short time to resume service by simply switching the Follower to Leader when the HoraeDB instance on which the leader shard exists crashes.

The diagram below concludes the relationship between HoraeDB instance, Shard, Table. As shown in the diagram, the leader and follower shards are interleaved on the HoraeDB instance.

plaintext
┌─HoraeDB Instance0──────┐     ┌─HoraeDB Instance1──────┐
│  ┌─Shard0(L)────────┐  │     │  ┌─Shard0(F)────────┐  │
│  │ ┌────┬────┬────┐ │  │     │  │ ┌────┬────┬────┐ │  │
│  │ │ T0 │ T1 │ T2 │ │  │     │  │ │ T0 │ T1 │ T2 │ │  │
│  │ └────┴────┴────┘ │  │     │  │ └────┴────┴────┘ │  │
│  └──────────────────┘  │     │  └──────────────────┘  │
│                        │     │                        │
│  ┌─Shard1(F)────────┐  │     │  ┌─Shard1(L)────────┐  │
│  │ ┌────┬────┬────┐ │  │     │  │ ┌────┬────┬────┐ │  │
│  │ │ T0 │ T1 │ T2 │ │  │     │  │ │ T0 │ T1 │ T2 │ │  │
│  │ └────┴────┴────┘ │  │     │  │ └────┴────┴────┘ │  │
│  └──────────────────┘  │     │  └──────────────────┘  │
└────────────────────────┘     └────────────────────────┘

Since Shard is the basic scheduling unit, it is natural to introduce some basic shard operations:

  • Create/Drop table to/from a shard;
  • Open/Close a shard;
  • Split one shard into two shards;
  • Merge two shards into one shard;
  • Switch the role of a shard;

With these basic shard operations, some complex scheduling logic can be implemented, e.g. perform an expansion by splitting one shard into two shards and migrating one of them to the new HoraeDB instance.

HoraeMeta

HoraeMeta is implemented by embedding an ETCD inside to ensure consistency and takes responsibilities for cluster metadata management and scheduling.

The cluster metadata includes:

  • Table information, such as table name, table ID, and which cluster the table belongs to;
  • The mapping between table and shard and between shard and HoraeDB instance;
  • ...

As for the cluster scheduling work, it mainly includes:

  • Receiving the heartbeats from the HoraeDB instances and determining the online status of these registered instances;
  • Assigning specific role shards to the registered HoraeDB instances;
  • Participating in table creation by assigning a unique table ID and the most appropriate shard to the table;
  • Performing load balancing through shard operations according to the load information sent with the heartbeats;
  • Performing expansion through shard operations when new instances are registered;
  • Initiating failover through shard operations when old instances go offline;

Route

In order to avoid the overhead of forwarding requests, the communication between clients and the HoraeDB instances is peer-to-peer, that is to say, the client should retrieve routing information from the server before sending any specific write/query requests.

Actually, the routing information is decided by the HoraeMeta, but clients are only allowed the access to it through the HoraeDB instances rather than HoraeMeta, to avoid potential performance issues on the HoraeMeta.

WAL Service & Object Storage

In the HoraeDB cluster, WAL Service and Object Storage exist as separate distributed systems featured with HA, data replication and scalability. Current distributed implementations for WAL Service includes Kafka and OBKV (access OceanBase by its table api), and the implementations for Object Storage include popular object storage services, such as AWS S3, Azure object storage and Aliyun OSS.

The two components are similar in that they are introduced to serve as the underlying storage layer for separating compute and storage, while the difference between two components is obvious that WAL Service is used to store the newly written data from the real-time write requests whose individual size is small but quantity is large, and Object Storage is used to store the read-friendly data files (SST) organized in the background, whose individual size is large and aggregate size is much larger.

The two components make it much easier to implement the horaedb cluster, which features horizontal scalability, high availability and load balancing.

Scalability

Scalability is an important feature for a distributed system. Let's take a look at to how the horizontal scalability of the HoraeDB cluster is achieved.

First, the two storage components (WAL Service and Object Storage) should be horizontally scalable when deciding on the actual implementations for them, so the two storage services can be expanded separately if the storage capacity is not sufficient.

It will be a little bit complex when discussing the scalability of the compute service. Basically, these cases will bring the capacity problem:

  • Massive queries on massive tables;
  • Massive queries on a single large table;
  • Massive queries on a normal table;

For the first case, it is easy to achieve horizontal scalability just by assigning shards that are created or split from old shards to expanded HoraeDB instances.

For the second case, the table partitioning is proposed and after partitioning, massive queries are distributed across multiple HoraeDB instances.

And the last case is the most important and the most difficult. Actually, the follower shard can handle part of the queries, but the number of follower shards is limited by the throughput threshold of the synchronization from the WAL regions. As shown in the diagram below, a pure compute shard can be introduced if the followers are not enough to handle the massive queries. Such a shard is not required to synchronize data with the leader shard, and retrieves the newly written data from the leader/follower shard only when the query comes. As for the SSTs required by the query, they will be downloaded from Object Storage and cached afterwards. With the two parts of the data, the compute resources are fully utilized to execute the CPU-intensive query plan. As we can see, such a shard can be added with only a little overhead (retrieving some data from the leader/follower shard when it needs), so to some extent, the horizontal scalability is achieved.

plaintext
┌HoraeDB─────┬┬─┐
                            ┌──newly written─│  │  │TableN││ │
                            ▼                └──Shard(L/F)┴┴─┘
┌───────┐  Query  ┌HoraeDB─────┬┬─┐
│client │────────▶│  │  │TableN││ │
└───────┘         └──Shard─────┴┴─┘          ┌───────────────┐
                            ▲                │    Object     │
                            └───old SST──────│    Storage    │
                                             └───────────────┘

High Availability

Assuming that WAL service and Object Storage are highly available, the high availability of the HoraeDB cluster can be achieved by such a procedure:

  • When detecting that the heartbeat is broken, HoraeMeta determines that the HoraeDB instance is offline;
  • The follower shards whose paired leader shards exist on the offline instance are switched to leader shards for fast failover;
  • A slow failover can be achieved by opening the crashed shards on another instance if such follower shards don't exist.
text
/* Detailed source-code truncated for AI context efficiency. */

Load Balancing

HoraeMeta collects the instance load information contained in the received heartbeats to create a load overview of the whole cluster, according to which the load balancing can be implemented as an automatic mechanism:

  • Pick a shard on a low-load instance for the newly created table;
  • Migrate a shard from a high-load instance load to another low-load instance;
  • Split the large shard on the high-load instance and migrate the split shards to other low-load instances;

File: content/en/docs/design/compaction_offload.md

title: "Compaction Offload"

Note: This feature is still in development.

This chapter discusses compaction offload, which is designed to separate the compaction workload from the local horaedb nodes and delegate it to external compaction nodes.

Overview

plaintext
┌─────────────────────────────────────────────────────────────────────────┐
│                                                                         │
│                           HoraeMeta Cluster                             │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
    ▲                ▲                           |                   |
    │                │1.Fetch compaction         │(Monitor compaction│
    │                │  node  info               │node)              │
    |                │                           ▼                   ▼
┌────────────┐  ┌────────────┐              ┌────────────┐   ┌────────────┐
│            │  │            │2.Offload Task│            │   │            │
│  HoraeDB   │  │  HoraeDB   │  ─────────▶  │ Compaction │   │ Compaction │
│            │  │            │  ◀─────────  │ Node       │   │ Node       │
└────────────┘  └────────────┘ 4.Ret Result └────────────┘   └────────────┘
    |                |                            |                 |
    │  5.Update the  │                            │    3.Compact    │
    │    SSTable     │                            │                 │
    ▼                ▼                            ▼                 ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                                                ┌─────────────────────┐  │
│                            Object Storage      │ Temporary Workspace │  │
│                                                └─────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────┘

The diagram above describes the architecture of cluster for compaction offload, where some key concepts need to be explained:

  • Compaction Node: Takes responsibility to handle offloaded compaction tasks. The compaction node receives the compaction task and performs the actual merging of SSTables, then sends back the task result to HoraeDB.
  • HoraeMeta Cluster: HoraeMeta acts as a compaction nodes manager in the compaction offload scenario. It monitors the compaction nodes cluster and schedule the compaction nodes.

The procedure of remote compaction based above architecture diagram is:

  1. HoraeDB servers fetch the information of suitable compaction nodes from the HoraeMeta.
  2. HoraeDB submit the compaction task to the remote compaction node, according to the information fetched from HoraeMeta.
  3. Compaction node executes the task and write results to the temporary workspace.
  4. Compaction node sends compaction results back to HoraeDB.
  5. HoraeDB receives the result, installs the data in temporary workspace and purges compaction input files.

The architecture above makes it easy to implement some wonderful features like load balancing and high availability. Let's dive into the key components in the architecture and talking about how these features are implemented.

Compaction Node

Compaction Node runs the main logic of compaction. It is implemented based on HoraeDB and distinguished by:

  • NodeType: A config parameter used to distinguished the HoraeDB and CompactionNode. This info would be sent to HoraeMeta through heartbeat.

The compaction service is implemented as grpc service.

HoraeMeta

HoraeMeta manages the compaction nodes cluster with CompactionNodeManager, which takes responsibilities for compaction nodes metadata management and scheduling.

The compaction nodes metadata includes:

  • Compaction node information, such as node name, node state;
  • A compaction node name list, used as the key to access compaction node info, for better scheduling with round-robin strategy;
  • ...

As for the compaction nodes scheduling work, it mainly includes:

  • Receiving the heartbeats from the compaction node and determining the online status of these registered nodes.
  • Performing load balancing according to the compaction nodes cluster info.
  • Providing the info of suitable compaction node for HoraeDB when remote compaction execution is needed.

Load Balancing

Load Balancing is critical for compaction nodes cluster to make their overall processing more efficient. The effect of load balancing mainly based on the schedule algorithm for compaction nodes impl in CompactionNodeManager.

(ps: The current implementation of schedule algorithm is round-robin strategy for easiness.)

The main process for the schedule algorithm based on real load is:

  • HoraeMeta collects the compaction nodes load information through the heartbeats to create a load overview of the compaction nodes cluster.
  • Pick a compaction node with low load according to the load overview.

High Availability

The fault tolerance of above architecture can be achieved by such a procedure:

  • When detecting that the heartbeat is broken, HoraeMeta determines that the compaction node is offline.
  • When HoraeMeta can not provide suitable compaction node for HoraeDB or compaction node doesn't return the task result successfully, HoraeDB would switches to run compaction task locally.

File: content/en/docs/design/storage.md

title: "Storage"

The storage engine mainly provides the following two functions：

  1. Persistence of data
  2. Under the premise of ensuring the correctness of the data, organize the data in the most reasonable way to meet the query needs of different scenarios.

This document will introduce the internal implementation of the storage engine in HoraeDB. Readers can refer to the content here to explore how to use HoraeDB efficiently.

Overall Structure

HoraeDB is a distributed storage system based on the share-nothing architecture.

Data between different servers is isolated from each other and does not affect each other. The storage engine in each stand-alone machine is a variant of log-structured merge-tree, which is optimized for time-series scenarios. The following figure shows its core components:

Write Ahead Log (WAL)

A write request will be written to

  1. memtable in memory
  2. WAL in durable storage

Since memtable is not persisted to the underlying storage system in real time, so WAL is required to ensure the reliability of the data in memtable.

On the other hand, due to the design of the distributed architecture, WAL itself is required to be highly available. Now there are following implementations in HoraeDB:

Memtable

Memtable is a memory data structure used to hold recently written table data. Different tables have its corresponding memtable.

Memtable is read-write by default (aka active), and when the write reaches some threshold, it will become read-only and be replaced by a new memtable.

The read-only memtable will be flushed to the underlying storage system in SST format by background thread. After flush is completed, the read-only memtable can be destroyed, and the corresponding data in WAL can also be deleted.

Sorted String Table（SST）

SST is a persistent format for data, which is stored in the order of primary keys of table. Currently, HoraeDB uses parquet format for this.

For HoraeDB, SST has an important option: segment_duration, only SST within the same segment can be merged, which is benefical for time-series data. And it is also convenient to eliminate expired data.

In addition to storing the original data, the statistical information of the data will also be stored in the SST to speed up the query, such as the maximum value, the minimum value, etc.

Compactor

Compactor can merge multiple small SST files into one, which is used to solve the problem of too many small files. In addition, Compactor will also delete expired data and duplicate data during the compaction. In future, compaction maybe add more task, such as downsample.

The current compaction strategy in HoraeDB reference Cassandra:

Manifest

Manifest records metadata of table, SST file, such as: the minimum and maximum timestamps of the data in an SST.

Due to the design of the distributed architecture, the manifest itself is required to be highly available. Now in HoraeDB, there are mainly the following implementations：

  • WAL
  • ObjectStore

ObjectStore

ObjectStore is place where data (i.e. SST) is persisted.

Generally speaking major cloud vendors should provide corresponding services, such as Alibaba Cloud's OSS and AWS's S3.

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#apm Browse apm catalog → #dapper Browse dapper catalog → #distributed-tracing Browse distributed-tracing catalog → #ebpf Browse ebpf catalog → #logging Browse logging catalog → #metrics Browse metrics catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →