apisix Documentation

The Cloud-Native API Gateway and AI Gateway

16,947 stars Lua Lua 1 file · ~27,125 tokens #ai-gateway#api#api-gateway#api-management#apisix#cloud-native#devops#iot
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy apisix on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name apisix -p 6379:6379 --restart always redis:latest
Configured Services & Ports
⚙️ apisix_redis
image: redis:latest
🔌 Ports: 6379:6379, 6380:6380
⚙️ etcd_old
image: bitnamilegacy/etcd:3.3.8
🔌 Ports: 3379:2379, 3380:2380
⚙️ etcd
image: bitnamilegacy/etcd:3.5.4
🔌 Ports: 2379:2379, 2380:2380
⚙️ etcd_tls
image: bitnamilegacy/etcd:3.5.4
🔌 Ports: 12379:12379, 12380:12380
⚙️ etcd_mtls
image: bitnamilegacy/etcd:3.5.4
🔌 Ports: 22379:22379, 22380:22380
⚙️ redis-cluster
image: vishnunair/docker-redis-cluster:latest
🔌 Ports: 5000:6379, 5002:6380, 5003:6381, 5004:6382, 5005:6383, 5006:6384
⚙️ vault
image: vault:1.9.0
🔌 Ports: 8200:8200
⚙️ localstack
image: localstack/localstack:4.12
🔌 Ports: 127.0.0.1:4566:4566
⚙️ httpbin
image: kennethreitz/httpbin
🔌 Ports: 8280:80, 8281:80
docker-compose.yml
yaml (docker-compose.yml) 
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

version: "3.8"

services:
  ## Redis
  apisix_redis:
    # The latest image is the latest stable version
    image: redis:latest
    restart: unless-stopped
    volumes:
      - ./t/certs:/certs
    command: "--tls-port 6380 \
            --tls-cert-file /certs/mtls_server.crt \
            --tls-key-file /certs/mtls_server.key \
            --tls-ca-cert-file /certs/mtls_ca.crt \
            --tls-auth-clients no \
            --user alice on +@all ~* \\&* \\>somepassword"
    ports:
      - "6379:6379"
      - "6380:6380"

  ## Etcd
  etcd_old:
    image: bitnamilegacy/etcd:3.3.8
    restart: unless-stopped
    env_file:
      - ci/pod/etcd/env/common.env
    environment:
      ETCD_ADVERTISE_CLIENT_URLS: http://0.0.0.0:2379
    ports:
      - "3379:2379"
      - "3380:2380"

  etcd:
    image: bitnamilegacy/etcd:3.5.4
    restart: unless-stopped
    env_file:
      - ci/pod/etcd/env/common.env
    environment:
        ETCD_ADVERTISE_CLIENT_URLS: http://0.0.0.0:2379
    ports:
      - "2379:2379"
      - "2380:2380"

  etcd_tls:
    image: bitnamilegacy/etcd:3.5.4
    restart: unless-stopped
    env_file:
      - ci/pod/etcd/env/common.env
    environment:
      ETCD_ADVERTISE_CLIENT_URLS: https://0.0.0.0:12379
      ETCD_LISTEN_CLIENT_URLS: https://0.0.0.0:12379
      ETCD_CERT_FILE: /certs/etcd.pem
      ETCD_KEY_FILE: /certs/etcd.key
    ports:
      - "12379:12379"
      - "12380:12380"
    volumes:
      - ./t/certs:/certs

  etcd_mtls:
    image: bitnamilegacy/etcd:3.5.4
    restart: unless-stopped
    env_file:
      - ci/pod/etcd/env/common.env
    environment:
      ETCD_ADVERTISE_CLIENT_URLS: https://0.0.0.0:22379
      ETCD_LISTEN_CLIENT_URLS: https://0.0.0.0:22379
      ETCD_CERT_FILE: /certs/mtls_server.crt
      ETCD_KEY_FILE: /certs/mtls_server.key
      ETCD_CLIENT_CERT_AUTH: "true"
      ETCD_TRUSTED_CA_FILE: /certs/mtls_ca.crt
    ports:
      - "22379:22379"
      - "22380:22380"
    volumes:
      - ./t/certs:/certs


  ## Redis cluster
  redis-cluster:
    image: vishnunair/docker-redis-cluster:latest
    restart: unless-stopped
    ports:
      - "5000:6379"
      - "5002:6380"
      - "5003:6381"
      - "5004:6382"
      - "5005:6383"
      - "5006:6384"


  ## HashiCorp Vault
  vault:
    image: vault:1.9.0
    container_name: vault
    restart: unless-stopped
    ports:
      - "8200:8200"
    cap_add:
      - IPC_LOCK
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: root
      VAULT_DEV_LISTEN_ADDRESS: 0.0.0.0:8200
    command: [ "vault", "server", "-dev" ]


  ## LocalStack
  localstack:
    image: localstack/localstack:4.12
    container_name: localstack
    restart: unless-stopped
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway


  ## httpbin - HTTP Request & Response Service
  httpbin:
    image: kennethreitz/httpbin
    container_name: httpbin
    restart: unless-stopped
    ports:
      - "8280:80"
      - "8281:80"

1. Project Overview & Quickstart (apache/apisix)

File: README.md

Apache APISIX API Gateway ｜ AI Gateway








Apache APISIX is a dynamic, real-time, high-performance API Gateway.

APISIX API Gateway provides rich traffic management features such as load balancing, dynamic upstream, canary release, circuit breaking, authentication, observability, and more. You can use it to handle traditional north-south traffic, as well as east-west traffic between services. It can also be used as a Kubernetes ingress controller.

Table of Contents

Why APISIX

APISIX is built on top of NGINX and etcd. Compared with traditional API gateways, APISIX has dynamic routing and hot-loading of plugins, which is especially suitable for API management under a microservice architecture.

The technical architecture of Apache APISIX:

AI Gateway

APISIX can serve as an AI Gateway through its flexible plugin system, providing:

  • AI proxying to route traffic to different LLM providers through a unified interface.
  • Load balancing, retries, and fallbacks across multiple LLMs to ensure the efficiency and reliability of AI agents.
  • Token-based rate limiting to control cost and protect upstream models.
  • Robust security for authentication, authorization, and traffic control on AI workloads.

APISIX also provides the mcp-bridge plugin to seamlessly convert stdio-based MCP servers to scalable HTTP SSE services.

Get Started

Install and run APISIX with a single command using the quickstart script (requires Docker):

shell
curl -sL https://run.api7.ai/apisix/quickstart | sh

This starts APISIX (listening on port 9080) together with its etcd configuration store. Verify it is running:

shell
curl "http://127.0.0.1:9080" --head | grep Server

Create your first route via the Admin API (port 9180) to proxy requests to an upstream service:

shell
curl -i "http://127.0.0.1:9180/apisix/admin/routes/1" -X PUT -d '
{
  "uri": "/get",
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "httpbin.org:80": 1
    }
  }
}'

Send a request through APISIX to confirm the route works:

shell
curl "http://127.0.0.1:9080/get"

To learn more, follow the Getting Started guide and the installation documentation for other deployment methods. To extend APISIX, see the plugin development guide, the plugin concept, and the REST Admin API reference.

For more documents, please refer to the Apache APISIX Documentation site.

Features

You can use APISIX API Gateway as a traffic entrance to process all business data, including dynamic routing, dynamic upstream, dynamic certificates,
A/B testing, canary release, blue-green deployment, limit rate, defense against malicious attacks, metrics, monitoring alarms, service observability, service governance, etc.

  • All platforms

    • Cloud-Native: Platform agnostic, No vendor lock-in, APISIX API Gateway can run from bare-metal to Kubernetes.
    • Supports ARM64: Don't worry about the lock-in of the infra technology.

  • Multi protocols

  • Full Dynamic

    • Hot Updates And Hot Plugins: Continuously updates its configurations and plugins without restarts!
    • Proxy Rewrite: Support rewrite the host, uri, schema, method, headers of the request before send to upstream.
    • Response Rewrite: Set customized response status code, body and header to the client.
    • Dynamic Load Balancing: Round-robin load balancing with weight.
    • Hash-based Load Balancing: Load balance with consistent hashing sessions.
    • Health Checks: Enable health check on the upstream node and will automatically filter unhealthy nodes during load balancing to ensure system stability.
    • Circuit-Breaker: Intelligent tracking of unhealthy upstream services.
    • Proxy Mirror: Provides the ability to mirror client requests.
    • Traffic Split: Allows users to incrementally direct percentages of traffic between various upstreams.

  • Fine-grained routing

  • Security

  • OPS friendly

    • Zipkin tracing: Zipkin
    • Open source APM: support Apache SkyWalking
    • Works with external service discovery: In addition to the built-in etcd, it also supports Consul, Consul_kv, Nacos, Eureka and Zookeeper (CP).
    • Monitoring And Metrics: Prometheus
    • Clustering: APISIX nodes are stateless, creates clustering of the configuration center, please refer to etcd Clustering Guide.
    • High availability: Support to configure multiple etcd addresses in the same cluster.
    • Dashboard
    • Version Control: Supports rollbacks of operations.
    • CLI: start\stop\reload APISIX through the command line.
    • Standalone: Supports to load route rules from local YAML file, it is more friendly such as under the kubernetes(k8s).
    • Global Rule: Allows to run any plugin for all request, eg: limit rate, IP filter etc.
    • High performance: The single-core QPS reaches 18k with an average delay of fewer than 0.2 milliseconds.
    • Fault Injection
    • REST Admin API: Using the REST Admin API to control Apache APISIX, which only allows 127.0.0.1 access by default, you can modify the allow_admin field in conf/config.yaml to specify a list of IPs that are allowed to call the Admin API. Also, note that the Admin API uses key auth to verify the identity of the caller.
    • External Loggers: Export access logs to external log management tools. (HTTP Logger, TCP Logger, Kafka Logger, UDP Logger, RocketMQ Logger, SkyWalking Logger, Alibaba Cloud Logging(SLS), Google Cloud Logging, Splunk HEC Logging, File Logger, SolarWinds Loggly Logging, TencentCloud CLS).
    • ClickHouse: push logs to ClickHouse.
    • Elasticsearch: push logs to Elasticsearch.
    • Datadog: push custom metrics to the DogStatsD server, comes bundled with Datadog agent, over the UDP protocol. DogStatsD basically is an implementation of StatsD protocol which collects the custom metrics for Apache APISIX agent, aggregates it into a single data point and sends it to the configured Datadog server.
    • Helm charts
    • HashiCorp Vault: Support secret management solution for accessing secrets from Vault secure storage backed in a low trust environment. Currently, RS256 keys (public-private key pairs) or secret keys can be linked from vault in jwt-auth authentication plugin using APISIX Secret resource.

  • Highly scalable

  • Multi-Language support

    • Apache APISIX is a multi-language gateway for plugin development and provides support via RPC and Wasm.

    • The RPC way, is the current way. Developers can choose the language according to their needs and after starting an independent process with the RPC, it exchanges data with APISIX through local RPC communication. Till this moment, APISIX has support for Java, Golang, Python and Node.js.

    • The Wasm or WebAssembly, is an experimental way. APISIX can load and run Wasm bytecode via APISIX wasm plugin written with the Proxy Wasm SDK. Developers only need to write the code according to the SDK and then compile it into a Wasm bytecode that runs on Wasm VM with APISIX.

  • Serverless

    • Lua functions: Invoke functions in each phase in APISIX.
    • AWS Lambda: Integration with AWS Lambda function as a dynamic upstream to proxy all requests for a particular URI to the AWS API gateway endpoint. Supports authorization via api key and AWS IAM access secret.
    • Azure Functions: Seamless integration with Azure Serverless Function as a dynamic upstream to proxy all requests for a particular URI to the Microsoft Azure cloud.
    • Apache OpenWhisk: Seamless integration with Apache OpenWhisk as a dynamic upstream to proxy all requests for a particular URI to your own OpenWhisk cluster.

Benchmark

Using AWS's eight-core server, APISIX's QPS reaches 140,000 with a latency of only 0.2 ms.

Benchmark script has been open sourced, welcome to try and contribute.

APISIX also works perfectly in AWS graviton3 C7g.

Community

User Stories

Who Uses APISIX API Gateway?

A wide variety of companies and organizations use APISIX API Gateway for research, production and commercial product, below are some of them:

  • Airwallex
  • Bilibili
  • CVTE
  • European eFactory Platform
  • European Copernicus Reference System
  • Geely
  • HONOR
  • Horizon Robotics
  • iQIYI
  • Lenovo
  • NASA JPL
  • Nayuki
  • OPPO
  • QingCloud
  • Swisscom
  • Tencent Game
  • Travelsky
  • vivo
  • Sina Weibo
  • WeCity
  • WPS
  • XPENG
  • Zoom

Logos

Acknowledgments

Inspired by Kong and Orange.

License

Apache 2.0 License

File: docs/zh/latest/architecture-design/apisix.md

title: 软件架构
keywords:

  • 网关
  • Apache APISIX
  • APISIX 架构

Apache APISIX 是一个动态、实时、高性能的云原生 API 网关。它构建于 NGINX + ngx_lua 的技术基础之上，充分利用了 LuaJIT 所提供的强大性能。 为什么 Apache APISIX 选择 NGINX+Lua 技术栈？。

APISIX 主要分为两个部分：

  1. APISIX 核心：包括 Lua 插件、多语言插件运行时（Plugin Runner）、Wasm 插件运行时等；
  2. 功能丰富的各种内置插件：包括可观测性、安全、流量控制等。

APISIX 在其核心中，提供了路由匹配、负载均衡、服务发现、API 管理等重要功能，以及配置管理等基础性模块。除此之外，APISIX 插件运行时也包含其中，提供原生 Lua 插件的运行框架和多语言插件的运行框架，以及实验性的 Wasm 插件运行时等。APISIX 多语言插件运行时提供多种开发语言的支持，比如 Golang、Java、Python、JS 等。

APISIX 目前也内置了各类插件，覆盖了 API 网关的各种领域，如认证鉴权、安全、可观测性、流量管理、多协议接入等。当前 APISIX 内置的插件使用原生 Lua 实现，关于各个插件的介绍与使用方式，可以查看相关插件文档。

插件加载流程

插件内部结构

File: docs/zh/latest/discovery/control-plane-service-discovery.md

title: 控制面服务发现
keywords:

  • API 网关
  • APISIX
  • ZooKeeper
  • Nacos
  • APISIX-Seed

本文档介绍了如何在 APISIX 控制面通过 Nacos 和 Zookeeper 实现服务发现。

APISIX-Seed 架构

Apache APISIX 在早期已经支持了数据面服务发现，现在 APISIX 也通过 APISIX-Seed 项目实现了控制面服务发现，下图为 APISIX-Seed 架构图。

图中的数字代表的具体信息如下：

  1. 通过 Admin API 向 APISIX 注册上游并指定服务发现类型。APISIX-Seed 将监听 etcd 中的 APISIX 资源变化，过滤服务发现类型并获取服务名称（如 ZooKeeper）；
  2. APISIX-Seed 将在服务注册中心（如 ZooKeeper）订阅指定的服务名称，以监控和更新对应的服务信息；
  3. 客户端向服务注册中心注册服务后，APISIX-Seed 会获取新的服务信息，并将更新后的服务节点写入 etcd；
  4. 当 APISIX-Seed 在 etcd 中更新相应的服务节点信息时，APISIX 会将最新的服务节点信息同步到内存中。

:::note

引入 APISIX-Seed 后，如果注册中心的服务变化频繁，etcd 中的数据也会频繁变化。因此，需要在启动 etcd 时设置 --auto-compaction 选项，用来定期压缩历史记录，避免耗尽 etcd 存储空间。详细信息请参考 revisions。

:::

为什么需要 APISIX-Seed？

  • 网络拓扑变得更简单

    APISIX 不需要与每个注册中心保持网络连接，只需要关注 etcd 中的配置信息即可。这将大大简化网络拓扑。

  • 上游服务总数据量变小

    由于 registry 的特性，APISIX 可能会在 Worker 中存储全量的 registry 服务数据，例如 Consul_KV。通过引入 APISIX-Seed，APISIX 的每个进程将不需要额外缓存上游服务相关信息。

  • 更容易管理

    服务发现配置需要为每个 APISIX 实例配置一次。通过引入 APISIX-Seed，APISIX 将对服务注册中心的配置变化无感知。

支持的服务发现类型

目前已经支持了 ZooKeeper 和 Nacos，后续还将支持更多的服务注册中心，更多信息请参考：APISIX Seed。

File: docs/zh/latest/discovery/dns.md

title: DNS

基于 DNS 的服务发现

某些服务发现系统如 Consul，支持通过 DNS 提供系统信息。我们可以使用这种方法直接实现服务发现，七层与四层均支持。

首先我们需要配置 DNS 服务器的地址：

yaml
# 添加到 config.yaml
discovery:
   dns:
     servers:
       - "127.0.0.1:8600"          # 使用 DNS 服务器的真实地址

与在 Upstream 的 nodes 对象中配置域名不同的是，DNS 服务发现将返回所有的记录。例如按照以下的 upstream 配置：

json
{
    "id": 1,
    "discovery_type": "dns",
    "service_name": "test.consul.service",
    "type": "roundrobin"
}

之后 test.consul.service 将被解析为 1.1.1.1 和 1.1.1.2，这个结果等同于：

json
{
    "id": 1,
    "type": "roundrobin",
    "nodes": [
        {"host": "1.1.1.1", "weight": 1},
        {"host": "1.1.1.2", "weight": 1}
    ]
}

注意所有来自 test.consul.service 的 IP 都有相同的权重。

解析的记录将根据它们的 TTL 来进行缓存。对于记录不在缓存中的服务，我们将默认按照 SRV -> A -> AAAA -> CNAME 的顺序进行查询，刷新缓存记录时，我们将从上次成功的类型开始尝试。也可以通过修改配置文件来自定义 DNS 的解析顺序。

yaml
# 添加到 config.yaml
discovery:
   dns:
     servers:
       - "127.0.0.1:8600"          # 使用 DNS 服务器的真实地址
     order:                        # DNS 解析的顺序
       - last                      # "last" 表示从上次成功的类型开始
       - SRV
       - A
       - AAAA
       - CNAME

如果你想指定 upstream 服务器的端口，可以把以下内容添加到 service_name：

json
{
    "id": 1,
    "discovery_type": "dns",
    "service_name": "test.consul.service:1980",
    "type": "roundrobin"
}

另一种方法是通过 SRV 记录，见如下。

SRV 记录

通过使用 SRV 记录你可以指定一个服务的端口和权重。

假设你有一条这样的 SRV 记录：

text
; under the section of blah.service
A       300 IN      A     1.1.1.1
B       300 IN      A     1.1.1.2
B       300 IN      A     1.1.1.3

; name  TTL         type    priority    weight  port
srv     86400 IN    SRV     10          60      1980 A
srv     86400 IN    SRV     20          20      1981 B

Upstream 配置是这样的：

json
{
    "id": 1,
    "discovery_type": "dns",
    "service_name": "srv.blah.service",
    "type": "roundrobin"
}

效果等同于：

json
{
    "id": 1,
    "type": "roundrobin",
    "nodes": [
        {"host": "1.1.1.1", "port": 1980, "weight": 60, "priority": -10},
        {"host": "1.1.1.2", "port": 1981, "weight": 10, "priority": -20},
        {"host": "1.1.1.3", "port": 1981, "weight": 10, "priority": -20}
    ]
}

注意 B 域名的两条记录均分权重。
对于 SRV 记录，低优先级的节点被先选中，所以最后一项的优先级是负数。

关于 0 权重的 SRV 记录，在 RFC 2782 中是这么描述的：

当没有任何候选服务器时，域管理员应使用权重为 0 的，使 RR 更为易读（噪音更少）。当存在权重大于 0 的记录时，权重为 0 的记录被选中的可能性很小。

我们把权重为 0 的记录当作权重为 1，因此节点“被选中的可能性很小”，这也是处理此类记录的常用方法。

对于端口为 0 的 SRV 记录，我们会使用上游协议的默认端口。
你也可以在“service_name”字段中直接指定端口，比如“srv.blah.service:8848”。

File: docs/zh/latest/discovery/eureka.md

title: eureka

Apache APISIX 支持使用 Eureka 做服务发现。
详情请阅读 支持的服务注册发现 。

File: docs/zh/latest/discovery/kubernetes.md

title: Kubernetes
keywords:

  • Kubernetes
  • Apache APISIX
  • 服务发现
  • 集群
  • API 网关

基于 Kubernetes 的服务发现

Kubernetes 服务发现以 List-Watch 方式监听 Kubernetes 集群 Endpoints 资源的实时变化，并将其值存储到 ngx.shared.DICT 中。

同时遵循 APISIX Discovery 规范 提供了节点查询接口。

Kubernetes 服务发现的使用

目前 Kubernetes 服务发现支持单集群和多集群模式，分别适用于待发现的服务分布在单个或多个 Kubernetes 的场景。

单集群模式 Kubernetes 服务发现的配置格式

单集群模式 Kubernetes 服务发现的完整配置如下：

text
/* Detailed source-code truncated for AI context efficiency. */

如果 Kubernetes 服务发现运行在 Pod 内，你可以使用如下最简配置：

yaml
discovery:
  kubernetes: { }

如果 Kubernetes 服务发现运行在 Pod 外，你需要新建或选取指定的 ServiceAccount, 获取其 Token 值，然后使用如下配置：

yaml
discovery:
  kubernetes:
    service:
      schema: https
      host: # enter apiserver host value here
      port: # enter apiServer port value here
    client:
      token: # enter serviceaccount token value here
      #token_file: # enter token file path here

单集群模式 Kubernetes 服务发现的查询接口

单集群模式 Kubernetes 服务发现遵循 APISIX Discovery 规范 提供节点查询接口。

函数：
nodes(service_name)

说明：
service_name 必须满足格式：[namespace]/[name]:[portName]

  • namespace: Endpoints 所在的命名空间

  • name: Endpoints 的资源名

  • portName: Endpoints 定义包含的 ports.name 值，如果 Endpoints 没有定义 ports.name，请依次使用 targetPort, port 代替。设置了 ports.name 的情况下，不能使用后两者。

返回值：
以如下 Endpoints 为例：

yaml
apiVersion: v1
kind: Endpoints
metadata:
  name: plat-dev
  namespace: default
subsets:
  - addresses:
      - ip: "10.5.10.109"
      - ip: "10.5.10.110"
    ports:
      - port: 3306
        name: port

nodes("default/plat-dev:port") 调用会得到如下的返回值：

text
{
     {
         host="10.5.10.109",
         port= 3306,
         weight= 50,
     },
     {
         host="10.5.10.110",
         port= 3306,
         weight= 50,
     },
 }

多集群模式 Kubernetes 服务发现的配置格式

多集群模式 Kubernetes 服务发现的完整配置如下：

text
/* Detailed source-code truncated for AI context efficiency. */

多集群模式 Kubernetes 服务发现没有为 service 和 client 域填充默认值，你需要根据集群配置情况自行填充。

多集群模式 Kubernetes 服务发现的查询接口

多集群模式 Kubernetes 服务发现遵循 APISIX Discovery 规范 提供节点查询接口。

函数：
nodes(service_name)

说明：
service_name 必须满足格式：[id]/[namespace]/[name]:[portName]

  • id: Kubernetes 服务发现配置中定义的集群 id 值

  • namespace: Endpoints 所在的命名空间

  • name: Endpoints 的资源名

  • portName: Endpoints 定义包含的 ports.name 值，如果 Endpoints 没有定义 ports.name，请依次使用 targetPort, port 代替。设置了 ports.name 的情况下，不能使用后两者。

返回值：
以如下 Endpoints 为例：

yaml
apiVersion: v1
kind: Endpoints
metadata:
  name: plat-dev
  namespace: default
subsets:
  - addresses:
      - ip: "10.5.10.109"
      - ip: "10.5.10.110"
    ports:
      - port: 3306
        name: port

nodes("release/default/plat-dev:port") 调用会得到如下的返回值：

text
{
     {
         host="10.5.10.109",
         port= 3306,
         weight= 50,
     },
     {
         host="10.5.10.110",
         port= 3306,
         weight= 50,
     },
 }

Q&A

Q: 为什么只支持配置 token 来访问 Kubernetes APIServer?

A: 一般情况下，我们有三种方式可以完成与 Kubernetes APIServer 的认证：

  • mTLS
  • Token
  • Basic authentication

因为 lua-resty-http 目前不支持 mTLS, Basic authentication 不被推荐使用，所以当前只实现了 Token 认证方式。

Q: APISIX 继承了 NGINX 的多进程模型，是否意味着每个 APISIX 工作进程都会监听 Kubernetes Endpoints？

A: Kubernetes 服务发现只使用特权进程监听 Kubernetes Endpoints，然后将其值存储到 ngx.shared.DICT 中，工作进程通过查询 ngx.shared.DICT 来获取结果。

Q: ServiceAccount 需要的权限有哪些？

A: ServiceAccount 需要集群级 [ get,list,watch ] endpoints 和 endpointslices 资源的的权限，其声明式定义如下：

yaml
kind: ServiceAccount
apiVersion: v1
metadata:
 name: apisix-test
 namespace: default
---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: apisix-test
rules:
  - apiGroups: [ "" ]
    resources: [ endpoints]
    verbs: [ get,list,watch ]
  - apiGroups: [ "discovery.k8s.io" ]
    resources: [ endpointslices ]
    verbs: [ get,list,watch ]
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: apisix-test
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: apisix-test
subjects:
 - kind: ServiceAccount
   name: apisix-test
   namespace: default

Q: 怎样获取指定 ServiceAccount 的 Token 值？

A: 假定你指定的 ServiceAccount 资源名为“kubernetes-discovery“, 命名空间为“apisix”, 请按如下步骤获取其 Token 值。

  1. 获取 Secret 资源名。执行以下命令，输出的第一列内容就是目标 Secret 资源名：
shell
kubectl -n apisix get secrets | grep kubernetes-discovery
  1. 获取 Token 值。假定你获取到的 Secret 资源名为 "kubernetes-discovery-token-c64cv", 执行以下命令，输出内容就是目标 Token 值：
shell
kubectl -n apisix get secret kubernetes-discovery-token-c64cv -o jsonpath={.data.token} | base64 -d

调试 API

它还提供了用于调试的控制 api。

内存 Dump API

shell
GET /v1/discovery/kubernetes/dump

例子

shell
# curl http://127.0.0.1:9090/v1/discovery/kubernetes/dump | jq
{
  "endpoints": [
    {
      "endpoints": [
        {
          "value": "{\"https\":[{\"host\":\"172.18.164.170\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":6443,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":6443,\"weight\":50}]}",
          "name": "default/kubernetes"
        },
        {
          "value": "{\"metrics\":[{\"host\":\"172.18.164.170\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.171\",\"port\":2379,\"weight\":50},{\"host\":\"172.18.164.172\",\"port\":2379,\"weight\":50}]}",
          "name": "kube-system/etcd"
        },
        {
          "value": "{\"http-85\":[{\"host\":\"172.64.89.2\",\"port\":85,\"weight\":50}]}",
          "name": "test-ws/testing"
        }
      ],
      "id": "first"
    }
  ],
  "config": [
    {
      "default_weight": 50,
      "id": "first",
      "client": {
        "token": "xxx"
      },
      "service": {
        "host": "172.18.164.170",
        "port": "6443",
        "schema": "https"
      },
      "shared_size": "64m"
    }
  ]
}

File: docs/zh/latest/discovery/nacos.md

title: nacos

基于 [Nacos](https://nacos.io/zh-cn/docs/what-is-nacos.html) 的服务发现

当前模块的性能有待改进：

  1. 并行发送请求。

Nacos 配置

在文件 conf/config.yaml 中添加以下配置到：

yaml
discovery:
  nacos:
    host:
      - "http://${username}:${password}@${host1}:${port1}"
    prefix: "/nacos/v1/"
    fetch_interval: 30    # default 30 sec
    weight: 100           # default 100
    timeout:
      connect: 2000       # default 2000 ms
      send: 2000          # default 2000 ms
      read: 5000          # default 5000 ms

也可以这样简洁配置（未配置项使用默认值）：

yaml
discovery:
  nacos:
    host:
      - "http://192.168.33.1:8848"

Upstream 设置

七层

例如，转发 URI 匹配 "/nacos/*" 的请求到一个上游服务，
该服务在 Nacos 中的服务名是 APISIX-NACOS，查询地址是 http://192.168.33.1:8848/nacos/v1/ns/instance/list?serviceName=APISIX-NACOS，创建路由时指定服务发现类型为 nacos。

:::note

您可以这样从 config.yaml 中获取 admin_key 并存入环境变量：

bash
admin_key=$(yq '.deployment.admin.admin_key[0].key' conf/config.yaml | sed 's/"//g')

:::

shell
$ curl http://127.0.0.1:9180/apisix/admin/routes/1 -H "X-API-KEY: $admin_key" -X PUT -i -d '
{
    "uri": "/nacos/*",
    "upstream": {
        "service_name": "APISIX-NACOS",
        "type": "roundrobin",
        "discovery_type": "nacos"
    }
}'

响应如下：

json
{
  "node": {
    "key": "\/apisix\/routes\/1",
    "value": {
      "id": "1",
      "create_time": 1615796097,
      "status": 1,
      "update_time": 1615799165,
      "upstream": {
        "hash_on": "vars",
        "pass_host": "pass",
        "scheme": "http",
        "service_name": "APISIX-NACOS",
        "type": "roundrobin",
        "discovery_type": "nacos"
      },
      "priority": 0,
      "uri": "\/nacos\/*"
    }
  }
}

四层

nacos 服务发现也支持在四层中使用，配置方式与七层的类似。

shell
$ curl http://127.0.0.1:9180/apisix/admin/stream_routes/1 -H "X-API-KEY: $admin_key" -X PUT -i -d '
{
    "remote_addr": "127.0.0.1",
    "upstream": {
        "scheme": "tcp",
        "discovery_type": "nacos",
        "service_name": "APISIX-NACOS",
        "type": "roundrobin"
    }
}'

参数

名字 类型 可选项 默认值 有效值 说明
namespace_id string 可选 public 服务所在的命名空间
group_name string 可选 DEFAULT_GROUP 服务所在的组

指定命名空间

例如，转发 URI 匹配 "/nacosWithNamespaceId/*" 的请求到一个上游服务，
该服务在 Nacos 中的服务名是 APISIX-NACOS，命名空间是 test_ns，查询地址是 http://192.168.33.1:8848/nacos/v1/ns/instance/list?serviceName=APISIX-NACOS&namespaceId=test_ns，创建路由时指定服务发现类型为 nacos。

shell
$ curl http://127.0.0.1:9180/apisix/admin/routes/2 -H "X-API-KEY: $admin_key" -X PUT -i -d '
{
    "uri": "/nacosWithNamespaceId/*",
    "upstream": {
        "service_name": "APISIX-NACOS",
        "type": "roundrobin",
        "discovery_type": "nacos",
        "discovery_args": {
          "namespace_id": "test_ns"
        }
    }
}'

响应如下：

json
{
  "node": {
    "key": "\/apisix\/routes\/2",
    "value": {
      "id": "2",
      "create_time": 1615796097,
      "status": 1,
      "update_time": 1615799165,
      "upstream": {
        "hash_on": "vars",
        "pass_host": "pass",
        "scheme": "http",
        "service_name": "APISIX-NACOS",
        "type": "roundrobin",
        "discovery_type": "nacos",
        "discovery_args": {
          "namespace_id": "test_ns"
        }
      },
      "priority": 0,
      "uri": "\/nacosWithNamespaceId\/*"
    }
  }
}

指定组

例如，转发 URI 匹配 "/nacosWithGroupName/*" 的请求到一个上游服务，
该服务在 Nacos 中的服务名是 APISIX-NACOS，组名是 test_group，查询地址是 http://192.168.33.1:8848/nacos/v1/ns/instance/list?serviceName=APISIX-NACOS&groupName=test_group，创建路由时指定服务发现类型为 nacos。

shell
$ curl http://127.0.0.1:9180/apisix/admin/routes/3 -H "X-API-KEY: $admin_key" -X PUT -i -d '
{
    "uri": "/nacosWithGroupName/*",
    "upstream": {
        "service_name": "APISIX-NACOS",
        "type": "roundrobin",
        "discovery_type": "nacos",
        "discovery_args": {
          "group_name": "test_group"
        }
    }
}'

响应如下：

json
{
  "node": {
    "key": "\/apisix\/routes\/3",
    "value": {
      "id": "3",
      "create_time": 1615796097,
      "status": 1,
      "update_time": 1615799165,
      "upstream": {
        "hash_on": "vars",
        "pass_host": "pass",
        "scheme": "http",
        "service_name": "APISIX-NACOS",
        "type": "roundrobin",
        "discovery_type": "nacos",
        "discovery_args": {
          "group_name": "test_group"
        }
      },
      "priority": 0,
      "uri": "\/nacosWithGroupName\/*"
    }
  }
}

同时指定命名空间和组

例如，转发 URI 匹配 "/nacosWithNamespaceIdAndGroupName/*" 的请求到一个上游服务，
该服务在 Nacos 中的服务名是 APISIX-NACOS，命名空间是 test_ns，组名是 test_group，查询地址是 http://192.168.33.1:8848/nacos/v1/ns/instance/list?serviceName=APISIX-NACOS&namespaceId=test_ns&groupName=test_group，创建路由时指定服务发现类型为 nacos。

shell
$ curl http://127.0.0.1:9180/apisix/admin/routes/4 -H "X-API-KEY: $admin_key" -X PUT -i -d '
{
    "uri": "/nacosWithNamespaceIdAndGroupName/*",
    "upstream": {
        "service_name": "APISIX-NACOS",
        "type": "roundrobin",
        "discovery_type": "nacos",
        "discovery_args": {
          "namespace_id": "test_ns",
          "group_name": "test_group"
        }
    }
}'

响应如下：

json
{
  "node": {
    "key": "\/apisix\/routes\/4",
    "value": {
      "id": "4",
      "create_time": 1615796097,
      "status": 1,
      "update_time": 1615799165,
      "upstream": {
        "hash_on": "vars",
        "pass_host": "pass",
        "scheme": "http",
        "service_name": "APISIX-NACOS",
        "type": "roundrobin",
        "discovery_type": "nacos",
        "discovery_args": {
          "namespace_id": "test_ns",
          "group_name": "test_group"
        }
      },
      "priority": 0,
      "uri": "\/nacosWithNamespaceIdAndGroupName\/*"
    }
  }
}

File: docs/zh/latest/getting-started/configure-routes.md

title: 配置路由 slug: /getting-started/configure-routes

本教程由 API7.ai 编写。

Apache APISIX 使用 routes 来提供灵活的网关管理功能，在一个请求中，routes 包含了访问路径和上游目标等信息。

本教程将引导你创建一个 route 并验证它，你可以参考以下步骤：

  1. 创建一个指向 httpbin.org的 upstream。
  2. 使用 cURL 发送一个请求，了解 APISIX 的代理和转发请求机制。

Route 是什么

Route（也称之为路由）是访问上游目标的路径，在 Apache APISIX 中，Route 首先通过预定的规则来匹配客户端请求，然后加载和执行相应的插件，最后将请求转发至特定的 Upstream。

在 APISIX 中，一个最简单的 Route 仅由匹配路径和 Upstream 地址两个信息组成。

Upstream 是什么

Upstream（也称之为上游）是一组具备相同功能的节点集合，它是对虚拟主机的抽象。Upstream 可以通过预先配置的规则对多个服务节点进行负载均衡。

前置条件

  1. 参考入门指南完成 APISIX 的安装。

创建路由

你可以创建一个路由，将客户端的请求转发至 httpbin.org（这个网站能测试 HTTP 请求和响应的各种信息）。

通过下面的命令，你将创建一个路由，把请求http://127.0.0.1:9080/ip 转发至 httpbin.org/ip：

shell
curl -i "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '
{
  "id": "getting-started-ip",
  "uri": "/ip",
  "upstream": {
    "type": "roundrobin",
    "nodes": {
      "httpbin.org:80": 1
    }
  }
}'

如果配置成功，将会返回 HTTP/1.1 201 Created。

验证

shell
curl "http://127.0.0.1:9080/ip"

你将会得到类似下面的返回：

text
{
  "origin": "183.94.122.205"
}

下一步

本教程创建的路由仅对应一个上游目标。在下个教程中，你将会学习如何配置多个上游目标的负载均衡。

File: docs/zh/latest/getting-started/key-authentication.md

title: 密钥验证 slug: /getting-started/key-authentication

本教程由 API7.ai 编写。

API 网关主要作用是连接 API 消费者和提供者。出于安全考虑，在访问内部资源之前，应先对消费者进行身份验证和授权。

APISIX 拥有灵活的插件扩展系统，目前有很多可用于用户身份验证和授权的插件。例如：

本教程中，你将创建一个带有 密钥验证 插件的 消费者，并学习如何启用和停用身份验证插件。

Consumer 是什么

Consumer（也称之为消费者）是指使用 API 的应用或开发人员。

在 APISIX 中，消费者需要一个全局唯一的 名称，并从上面的列表中选择一个身份验证 插件。

Key Authentication 是什么

Key Authentication（也称之为密钥验证）是一个相对比较简单但是应用广泛的身份验证方法，它的设计思路如下：

  1. 管理员为路由添加一个身份验证密钥（API 密钥）。
  2. API 消费者在发送请求时，在查询字符串或者请求头中添加密钥。

启用 Key Authentication

前置条件

  1. 参考快入门指南完成 APISIX 的安装。
  2. 完成配置路由。

创建消费者

创建一个名为 tom 的消费者，并启用 key-auth 插件，密钥设置为 secret-key。所有携带密钥 secret-key 的请求都会被识别为消费者 tom。

:::caution

生产环境请使用复杂的密钥。

:::

shell
curl -i "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT -d '
{
  "username": "tom",
  "plugins": {
    "key-auth": {
      "key": "secret-key"
    }
  }
}'

如果消费者创建成功，你将得到返回 HTTP/1.1 201 Created。

启用 Authentication

在教程配置路由中，我们已经创建了路由 getting-started-ip，我们通过 PATCH 方法为该路由增加 key-auth 插件：

shell
curl -i "http://127.0.0.1:9180/apisix/admin/routes/getting-started-ip" -X PATCH -d '
{
  "plugins": {
    "key-auth": {}
  }
}'

如果增加插件成功，你将得到返回 HTTP/1.1 201 Created。

验证

我们可以在以下场景中进行验证：

1. 发送不带任何密钥的请求

发送一个不带请求头 apikey 的请求。

shell
curl -i "http://127.0.0.1:9080/ip"

如果你已经启用了密钥身份验证，你将会得到返回 HTTP/1.1 401 Unauthorized，即未授权。

text
HTTP/1.1 401 Unauthorized
Date: Wed, 08 Feb 2023 09:38:36 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: APISIX/3.1.0

2. 发送携带错误密钥的请求

发送一个携带错误密钥（比如 wrong-key）的请求。

shell
curl -i "http://127.0.0.1:9080/ip" -H 'apikey: wrong-key'

如果密钥错误，你也将得到返回 HTTP/1.1 401 Unauthorized，即未授权。

text
HTTP/1.1 401 Unauthorized
Date: Wed, 08 Feb 2023 09:38:27 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: APISIX/3.1.0

3. 发送携带正确密钥的请求

发送一个携带正确密钥（secret-key）的请求。

shell
curl -i "http://127.0.0.1:9080/ip" -H 'apikey: secret-key'

你将会得到返回 HTTP/1.1 200 OK。

text
HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 44
Connection: keep-alive
Date: Thu, 09 Feb 2023 03:27:57 GMT
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true
Server: APISIX/3.1.0

禁用 Authentication

将参数设置 _meta.disable 为 true，即可禁用密钥验证插件。

shell
curl "http://127.0.0.1:9180/apisix/admin/routes/getting-started-ip" -X PATCH -d '
{
  "plugins": {
    "key-auth": {
      "_meta": {
        "disable": true
      }
    }
  }
}'

你可以发送一个不带任何密钥的请求来验证：

shell
curl -i "http://127.0.0.1:9080/ip"

因为你已经禁用了密钥验证插件，所以你将会得到返回 HTTP/1.1 200 OK。

下一步

你已经学习了如何为路由配置密钥验证。在下个教程中，你将学习如何配置限速。

File: docs/zh/latest/getting-started/load-balancing.md

title: 负载均衡 slug: /getting-started/load-balancing

本教程由 API7.ai 编写。

负载均衡管理客户端和服务端之间的流量。它决定由哪个服务来处理特定的请求，从而提高性能、可扩展性和可靠性。在设计需要处理大量流量的系统时，负载均衡是一个关键的考虑因素。

Apache APISIX 支持加权负载均衡算法，传入的流量按照预定顺序轮流分配给一组服务器的其中一个。

在本教程中，你将创建一个具有两个上游服务的路由，并且启用负载均衡来测试在两个服务之间的切换情况。

前置条件

  1. 参考入门指南完成 APISIX 的安装。
  2. 了解 APISIX 中路由及上游的概念。

启用负载均衡

创建一个具有两个上游服务的路由，访问 /headers 将被转发到 httpbin.org 和 mock.api7.ai 这两个上游服务，并且会返回请求头。

shell
curl -i "http://127.0.0.1:9180/apisix/admin/routes" -X PUT -d '
{
  "id": "getting-started-headers",
  "uri": "/headers",
  "upstream" : {
    "type": "roundrobin",
    "nodes": {
      "httpbin.org:443": 1,
      "mock.api7.ai:443": 1
    },
    "pass_host": "node",
    "scheme": "https"
  }
}'

如果路由创建成功，你将会收到返回 HTTP/1.1 201 Created。

:::info

  1. 将 pass_host 字段设置为 node，将传递请求头给上游。
  2. 将 scheme 字段设置为 https，向上游发送请求时将启用 TLS。

:::

验证

这两个服务返回不同的数据。

httpbin.org 返回：

json
{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.58.0",
    "X-Amzn-Trace-Id": "Root=1-63e34b15-19f666602f22591b525e1e80",
    "X-Forwarded-Host": "localhost"
  }
}

mock.api7.ai 返回：

json
{
  "headers": {
    "accept": "*/*",
    "host": "mock.api7.ai",
    "user-agent": "curl/7.58.0",
    "content-type": "application/json",
    "x-application-owner": "API7.ai"
  }
}

我们生成 100 个请求来测试负载均衡的效果：

shell
hc=$(seq 100 | xargs -I {} curl "http://127.0.0.1:9080/headers" -sL | grep "httpbin" | wc -l); echo httpbin.org: $hc, mock.api7.ai: $((100 - $hc))

结果显示，请求几乎平均分配给这两个上游服务：

text
httpbin.org: 51, mock.api7.ai: 49

下一步

你已经学习了如何配置负载均衡。在下个教程中，你将学习如何配置身份验证。

2. Official Technical Reference & Guides (apache/docs)

File: README.md

Apache HoraeDB (incubating) Website

[!IMPORTANT]
Apache HoraeDB (incubating) is an effort undergoing incubation at the Apache
Software Foundation (ASF), sponsored by the Apache Incubator PMC.

Please read the DISCLAIMER and a full explanation of "incubating".

This repository stores all the source files of HoraeDB website, it's built with hugo and docsy.

How to run

bash
# Start a local server for preview
hugo serve

By default hugo will listen on http://localhost:1313

Contributing

Welcome any contributions from the community. You can

  • Open an issue with any suggestions, or
  • Submit a pull request to improve the website (trivial fixes are welcome).

File: content/en/blog/2024/release-2.0.0.md

title: Release 2.0.0 date: 2024-04-23

Upgrade from 1.x.x to 2.0.0

The transition from CeresDB to Apache HoraeDB introduces several breaking changes. To facilitate upgrading from older versions to v2.0.0, specific alterations are necessary.

Upgrade Steps

  1. Setup required envs
text
export HORAEDB_DEFAULT_CATALOG=ceresdb
  1. Update config

Etcd's root should be configured both in horaedb and horaemeta

For horaedb

text
[cluster_deployment.etcd_client]
server_addrs = ['127.0.0.1:2379']
root_path = "/rootPath"

For horaemeta

text
storage-root-path = "/rootPath"
  1. Upgrade horaemeta

Horaedb will throw following errors, which is expected

text
2024-01-23 14:37:57.726 ERRO [src/cluster/src/cluster_impl.rs:136] Send heartbeat to meta failed, err:Failed to send heartbeat, cluster:defaultCluster, err:status: Unimplemented, message: "unknown service meta_service.MetaRpcService", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc"} }
  1. Upgrade horaedb

After all server upgraded, the cluster should be ready for read/write, and old data could be queried like before.

What's Changed

Breaking Changes

Features

Refactor

Fixed

Docs

Chore

New Contributors

Full Changelog: https://github.com/apache/horaedb/compare/v1.2.7...v2.0.0

File: content/en/blog/2024/release-2.1.0.md

title: Release 2.1.0
date: 2024-11-18
tags:

  • release

The Apache HoraeDB(incubating) team are pleased to announce that v2.1.0 is released, which has closed over 60 issues, including two major features:

1. New WAL implementation based on local disk.

In previous version, there is a RocksDB-based WAL. Although it works well in most cases, it has following issues:

  • Compiling from source can be a challenging task, especially since RocksDB is primarily written in C++.
  • For WAL, RocksDB can be somewhat overkill. If you are not familiar with RocksDB, tuning it can be very challenging.

With this new WAL, the above two problems are solved very well, and in performance test, the new WAL slightly outperforms the previous implementation, giving a solid foundation for future optimizations.

Interested readers can refer to the design documentation [here]({{< ref "wal_on_disk" >}}) for more details on this feature.

How to enable

text
[analytic.wal]
type = "Local"
data_dir = "/path/to/local/wal"

2. Access object store with [Apache OpenDAL](https://github.com/apache/opendal)

OpenDAL (Open Data Access Layer) is a project that provides a unified API for accessing various data storage backends.
It offers several advantages for developers and organizations. Here are some key benefits:

  • Unified API. OpenDAL provides a consistent and unified API for accessing different storage backends, such as AWS S3, Azure Blob Storage, and local file systems.
  • Optimized for Efficiency: OpenDAL is built with performance in mind. It includes optimizations to ensure efficient data access and manipulation, making it suitable for high-performance applications.
  • Comprehensive Documentation: The project provides detailed documentation, making it easier for developers to get started and understand how to use the library effectively.

In newer versions of OpenDAL, object_store integration is provided, which is very beneficial for HoraeDB code migration, as the APIs used by the upper layers remain virtually unchanged, and only the object store part needs to be abstracted to a unified OpenDAL operator:

rust
// Create a new operator
let operator = Operator::new(S3::default())?.finish();

// Create a new object store
let object_store = Arc::new(OpendalStore::new(operator));

Additionally, since the Apache OpenDAL implementation of object_store is based on the latest version of the object_store, which has breaking changes from the previous version that HoraeDB is using, we've chosen to make it compatible in order to keep the scope of this upgrade as manageable as possible.

In the process of adapting to the new API, the put_multipart interface has changed the most, so the main adaptation logic is also here, HoraeDB's approach is to encapsulate the underlying put_multipart interface to ensure that the upper layer code is not modified, the details can be found in the reference:

https://github.com/apache/horaedb/blob/v2.1.0/src/components/object_store/src/multi_part.rs

Note: The adaptation logic is only practical when parquet version < 52.0.0.

Download

Go to download pages.

Conclusion

Other bug fixes and improvements can be seen here:

As always, we warmly welcome you to join our community and share your insights.

File: content/en/blog/_index.md

title: "Blog"
weight: 1
simple_list: true
menu:
main:
weight: 30
pre:

File: content/en/community/_index.md

title: "Community"
params:
contributingUrl: https://github.com/apache/horaedb/blob/main/CONTRIBUTING.md
menu:
main:
weight: 40

{{% blocks/section color="white" %}}

How to Subscribe Mail List

Mailing lists are a form of communication used by the Apache community. Generally speaking, many things in the Apache community are hosted by mailing lists, For example: project q&A, technical discussion, transaction decision, release vote, etc，By subscribing, you can get the first update of the HoraeDB community and keep up with the community. The following list is the main list we are using:

In order to subscribe to the [email protected] mailing list, the steps are as follows：

  1. Send an email without any content or subject: [email protected]
  2. Wait until you receive an email with the subject line confirm subscribe to [email protected] (if you have not received it for a long time, please confirm whether the email is blocked by your email, if you have not been blocked and will receive a reply for a long time, return to step 1)
  3. Reply directly to the email without changing the subject line or adding the email content.
  4. Wait until you receive an email with the subject line WELCOME to [email protected] .
  5. If you receive an email from dev, you have successfully subscribed to the email. To initiate a discussion, you can send an email directly to [email protected], which will be sent to everyone who subscribed to the mailing list.

Unsubscribe from the mail list

The steps for unsubscribing to a mailing list are similar to those for subscribing to a mailing list:

  1. Send an email without any content or subject to: [email protected]
  2. Wait until you receive an email with the subject line confirm unsubscribe from [email protected]
  3. Reply directly to the email without changing the subject line or adding the email content
  4. Wait until you receive an email with the subject line GOODBYE from [email protected]
  5. Unsubscribe success

{{% /blocks/section %}}

File: content/en/docs/design/architecture.md

title: "Introduction to HoraeDB's Architecture"

Target

  • Provide the overview of HoraeDB to the developers who want to know more about HoraeDB but have no idea where to start.
  • Make a brief introduction to the important modules of HoraeDB and the connections between these modules but details about their implementations are not be involved.

Motivation

HoraeDB is a timeseries database (TSDB). However, HoraeDB's goal is to handle both timeseries and analytic workloads compared with the classic TSDB, which usually have a poor performance in handling analytic workloads.

In the classic timeseries database, the Tag columns (InfluxDB calls them Tag and Prometheus calls them Label) are normally indexed by generating an inverted index. However, it is found that the cardinality of Tag varies in different scenarios. And in some scenarios the cardinality of Tag is very high (we name this case after analytic workload), and it takes a very high cost to store and retrieve the inverted index. On the other hand, it is observed that scanning+pruning often used by the analytical databases can do a good job to handle such analytic workload.

The basic design idea of HoraeDB is to adopt a hybrid storage format and the corresponding query method for a better performance in processing both timeseries and analytic workloads.

Architecture

plaintext
┌──────────────────────────────────────────┐
│       RPC Layer (HTTP/gRPC/MySQL)        │
└──────────────────────────────────────────┘
┌──────────────────────────────────────────┐
│                 SQL Layer                │
│ ┌─────────────────┐  ┌─────────────────┐ │
│ │     Parser      │  │     Planner     │ │
│ └─────────────────┘  └─────────────────┘ │
└──────────────────────────────────────────┘
┌───────────────────┐  ┌───────────────────┐
│    Interpreter    │  │      Catalog      │
└───────────────────┘  └───────────────────┘
┌──────────────────────────────────────────┐
│               Query Engine               │
│ ┌─────────────────┐  ┌─────────────────┐ │
│ │    Optimizer    │  │    Executor     │ │
│ └─────────────────┘  └─────────────────┘ │
└──────────────────────────────────────────┘
┌──────────────────────────────────────────┐
│         Pluggable Table Engine           │
│  ┌────────────────────────────────────┐  │
│  │              Analytic              │  │
│  │┌────────────────┐┌────────────────┐│  │
│  ││      Wal       ││    Memtable    ││  │
│  │└────────────────┘└────────────────┘│  │
│  │┌────────────────┐┌────────────────┐│  │
│  ││     Flush      ││   Compaction   ││  │
│  │└────────────────┘└────────────────┘│  │
│  │┌────────────────┐┌────────────────┐│  │
│  ││    Manifest    ││  Object Store  ││  │
│  │└────────────────┘└────────────────┘│  │
│  └────────────────────────────────────┘  │
│  ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
│           Another Table Engine        │  │
│  └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─   │
└──────────────────────────────────────────┘

The figure above shows the architecture of HoraeDB stand-alone service and the details of some important modules will be described in the following part.

RPC Layer

module path: https://github.com/apache/incubator-horaedb/tree/main/server

The current RPC supports multiple protocols including HTTP, gRPC, MySQL.

Basically, HTTP and MySQL are used to debug HoraeDB, query manually and perform DDL operations (such as creating, deleting tables, etc.). And gRPC protocol can be regarded as a customized protocol for high-performance, which is suitable for massive reading and writing operations.

SQL Layer

module path: https://github.com/apache/incubator-horaedb/tree/main/query_frontend

SQL layer takes responsibilities for parsing sql and generating the query plan.

Based on sqlparser a sql dialect, which introduces some key concepts including Tag and Timestamp, is provided for processing timeseries data. And by utilizing DataFusion the planner is able to generate both regular logical plans and tailored ones which is used to implement the special operators defined by timeseries queries, e.g PromQL.

Interpreter

module path: https://github.com/apache/incubator-horaedb/tree/main/interpreters

The Interpreter module encapsulates the SQL CRUD operations. In the query procedure, a sql received by HoraeDB is parsed, converted into the query plan and then executed in some specific interpreter, such as SelectInterpreter, InsertInterpreter and etc.

Catalog

module path: https://github.com/apache/incubator-horaedb/tree/main/catalog_impls

Catalog is actually the module managing metadata and the levels of metadata adopted by HoraeDB is similar to PostgreSQL: Catalog > Schema > Table, but they are only used as namespace.

At present, Catalog and Schema have two different kinds of implementation for standalone and distributed mode because some strategies to generate ids and ways to persist metadata differ in different mode.

Query Engine

module path: https://github.com/apache/incubator-horaedb/tree/main/query_engine

Query Engine is responsible for optimizing and executing query plan given a basic SQL plan provided by SQL layer and now such work is mainly delegated to DataFusion.

In addition to the basic functions of SQL, HoraeDB also defines some customized query protocols and optimization rules for some specific query plans by utilizing the extensibility provided by DataFusion. For example, the implementation of PromQL is implemented in this way and read it if you are interested.

Pluggable Table Engine

module path: https://github.com/apache/incubator-horaedb/tree/main/table_engine

Table Engine is actually a storage engine for managing tables in HoraeDB and the pluggability of Table Engine is a core design of HoraeDB which matters in achieving our long-term target, e.g supporting handle log or tracing workload by implementing new storage engines. HoraeDB will have multiple kinds of Table Engine for different workloads and the most appropriate one should be chosen as the storage engine according to the workload pattern.

Now the requirements for a Table Engine are:

  • Manage all the shared resources under the engine:
    • Memory
    • Storage
    • CPU
  • Manage metadata of tables such as table schema and table options;
  • Provide Table instances which provides read and write methods;
  • Take responsibilities for creating, opening, dropping and closing Table instance;
  • ....

Actually the things that a Table Engine needs to process are a little complicated. And now in HoraeDB only one Table Engine called Analytic is provided and does a good job in processing analytical workload, but it is not ready yet to handle the timeseries workload (we plan to enhance it for a better performance by adding some indexes which help handle timeseries workload).

The following part gives a description about details of Analytic Table Engine.

WAL

module path: https://github.com/apache/incubator-horaedb/tree/main/wal

The model of HoraeDB processing data is WAL + MemTable that the recent written data is written to WAL first and then to MemTable and after a certain amount of data in MemTable is accumulated, the data will be organized in a query-friendly form to persistent devices.

Now three implementations of WAL are provided for standalone and distributed mode:

  • For standalone mode, WAL is based on RocksDB and data is persisted on the local disk.
  • For distributed mode, WAL is required as a distributed component and to be responsible for durability of the newly written data, so now we provide an implementation based on OceanBase.
  • For distributed mode, in addition to OceanBase, we also provide a more lightweight implementation based on Apache Kafka.

MemTable

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/memtable

For WAL can't provide efficient data retrieval, the newly written data is also stored in Memtable for efficient data retrieval, after a certain amount of data is reached, HoraeDB organizes the data in MemTable into a query-friendly storage format (SST) and stores it to the persistent device.

The current implementation of MemTable is based on agatedb's skiplist. It allows concurrent reads and writes and can control memory usage based on Arena.

Flush

module path: https://github.com/apache/incubator-horaedb/blob/main/analytic_engine/src/instance/flush_compaction.rs

What Flush does is that when the memory usage of MemTable reaches the threshold, some MemTables are selected for flushing into query-friendly SSTs saved on persistent device.

During the flushing procedure, the data will be divided by a certain time range (which is configured by table option Segment Duration), and any SST is ensured that the timestamps of the data in it are in the same Segment. Actually this is also a common operation in most timeseries databases which organizes data in the time dimension to speed up subsequent time-related operations, such as querying data over a time range and assisting purge data outside the TTL.

Compaction

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/compaction

The data of MemTable is flushed as SSTs, but the file size of recently flushed SST may be very small. And too small or too many SSTs lead to the poor query performance. Therefore, Compaction is then introduced to rearrange the SSTs so that the multiple smaller SST files can be compacted into a larger SST file.

Manifest

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/meta

Manifest takes responsibilities for managing tables' metadata of Analytic Engine including:

  • Table schema and table options;
  • The sequence number where the newest flush finishes;
  • The information of all the SSTs belonging to the table.

Now the Manifest is based on WAL and Object Storage. The newly written updates on the Manifest are persisted as logs in WAL, and in order to avoid infinite expansion of Manifest (actually every Flush leads to an update), Snapshot is also introduced to clean up the history of metadata updates, and the generated Snapshot will be saved to Object Storage.

Object Storage

module path: https://github.com/apache/incubator-horaedb/tree/main/components/object_store

The SST generated by Flush needs to be persisted and the abstraction of the persistent storage device is ObjectStore including multiple implementations:

The distributed architecture of HoraeDB separates storage and computing, which requires Object Store needs to be a highly available and reliable service independent of HoraeDB. Therefore, storage systems like Amazon S3, Alibaba Cloud OSS is a good choice and in the future implementations on storage systems of some other cloud service providers is planned to provide.

SST

module path: https://github.com/apache/incubator-horaedb/tree/main/analytic_engine/src/sst

SST is actually an abstraction that can have multiple specific implementations. The current implementation is based on Parquet, which is a column-oriented data file format designed for efficient data storage and retrieval.

The format of SST is very critical for retrieving data and is also the most important part to perform well in handling both timeseries and analytic workloads. At present, our Parquet-based implementation is good at processing analytic workload but is poor at processing timeseries workload. In our roadmap, we will explore more storage formats in order to achieve a good performance in both workloads.

Space

module path: https://github.com/apache/incubator-horaedb/blob/main/analytic_engine/src/space.rs

In Analytic Engine, there is a concept called space and here is an explanation for it to resolve some ambiguities when read source code. Actually Analytic Engine does not have the concept of catalog and schema and only provides two levels of relationship: space and table. And in the implementation, the schema id (which should be unique across all catalogs) on the upper layer is actually mapped to space id.

The space in Analytic Engine serves mainly for isolation of resources for different tenants, such as the usage of memory.

Critical Path

After a brief introduction to some important modules of HoraeDB, we will give a description for some critical paths in code, hoping to provide interested developers with a guide for reading the code.

Query

plaintext
┌───────┐      ┌───────┐      ┌───────┐
│       │──1──▶│       │──2──▶│       │
│Server │      │  SQL  │      │Catalog│
│       │◀─10──│       │◀─3───│       │
└───────┘      └───────┘      └───────┘
                │    ▲
               4│   9│
                │    │
                ▼    │
┌─────────────────────────────────────┐
│                                     │
│             Interpreter             │
│                                     │
└─────────────────────────────────────┘
                           │    ▲
                          5│   8│
                           │    │
                           ▼    │
                   ┌──────────────────┐
                   │                  │
                   │   Query Engine   │
                   │                  │
                   └──────────────────┘
                           │    ▲
                          6│   7│
                           │    │
                           ▼    │
 ┌─────────────────────────────────────┐
 │                                     │
 │            Table Engine             │
 │                                     │
 └─────────────────────────────────────┘

Take SELECT SQL as an example. The figure above shows the query procedure and the numbers in it indicates the order of calling between the modules.

Here are the details:

  • Server module chooses a proper rpc module (it may be HTTP, gRPC or mysql) to process the requests according the protocol used by the requests;
  • Parse SQL in the request by the parser;
  • With the parsed sql and the information provided by catalog/schema module, DataFusion can generate the logical plan;
  • With the logical plan, the corresponding Interpreter is created and logical plan will be executed by it;
  • For the logical plan of normal Select SQL, it will be executed through SelectInterpreter;
  • In the SelectInterpreter the specific query logic is executed by the Query Engine:
    • Optimize the logical plan;
    • Generate the physical plan;
    • Optimize the physical plan;
    • Execute the physical plan;
  • The execution of physical plan involves Analytic Engine:
    • Data is obtained by read method of Table instance provided by Analytic Engine;
    • The source of the table data is SST and Memtable, and the data can be filtered by the pushed down predicates;
    • After retrieving the table data, Query Engine will complete the specific computation and generate the final results;
  • SelectInterpreter gets the results and feeds them to the protocol module;
  • After the protocol layer converts the results, the server module responds to the client with them.

The following is the flow of function calls in version v1.2.2:

text
/* Detailed source-code truncated for AI context efficiency. */
  1. The received request will be forwarded to handle_sql after various protocol conversions, and since the request may not be processed by this node, it may need to be forwarded to maybe_forward_sql_query to handle the forwarding logic.
  2. After constructing the ForwardRequest in maybe_forward_sql_query, call forward
  3. After constructing the RouteRequest in forward, call route
  4. Use route to get the destination node endpoint and return to forward.
  5. Call forward_with_endpoint to forward the request
  6. return forward
  7. return maybe_forward_sql_query
  8. return handle_sql
  9. If this is a Local request, call fetch_sql_query_output to process it
  10. Call parse_sql to parse sql into Statment
  11. return fetch_sql_query_output
  12. Call statement_to_plan with Statment
  13. Construct Planner with ctx and Statment, and call the statement_to_plan method of Planner
  14. The planner will call the corresponding planner method for the requested category, at this point our sql is a query and will call sql_statement_to_plan
  15. Call sql_statement_to_datafusion_plan , which will generate the datafusion object, and then call SqlToRel::sql_statement_to_plan
  16. The generated logical plan is returned from SqlToRel::sql_statement_to_plan
  17. return
  18. return
  19. return
  20. Call execute_plan_involving_partition_table (in the default configuration) for subsequent optimization and execution of this logical plan
  21. Call build_interpreter to generate Interpreter
  22. return
  23. Call Interpreter's interpreter_execute_plan method for logical plan execution.
  24. The corresponding execute function is called, at this time the sql is a query, so the execute of the SelectInterpreter will be called
  25. call execute_logical_plan , which will call build_df_session_ctx to generate the optimizer
  26. build_df_session_ctx will use the config information to generate the corresponding context, first using datafusion and some custom optimization rules (in logical_optimize_rules()) to generate the logical plan optimizer, using apply_adapters_for_physical_optimize_rules to generate the physical plan optimizer
  27. return optimizer
  28. Call optimize_plan, using the optimizer just generated to first optimize the logical plan and then the physical plan
  29. Return to optimized physical plan
  30. execute physical plan
  31. returned after execution
  32. After collecting the results of all slices, return
  33. return
  34. return
  35. return
  36. Return to the upper layer for network protocol conversion and finally return to the request sender

Write

plaintext
┌───────┐      ┌───────┐      ┌───────┐
│       │──1──▶│       │──2──▶│       │
│Server │      │  SQL  │      │Catalog│
│       │◀─8───│       │◀─3───│       │
└───────┘      └───────┘      └───────┘
                │    ▲
               4│   7│
                │    │
                ▼    │
┌─────────────────────────────────────┐
│                                     │
│             Interpreter             │
│                                     │
└─────────────────────────────────────┘
      │    ▲
      │    │
      │    │
      │    │
      │    │       ┌──────────────────┐
      │    │       │                  │
     5│   6│       │   Query Engine   │
      │    │       │                  │
      │    │       └──────────────────┘
      │    │
      │    │
      │    │
      ▼    │
 ┌─────────────────────────────────────┐
 │                                     │
 │            Table Engine             │
 │                                     │
 └─────────────────────────────────────┘

Take INSERT SQL as an example. The figure above shows the query procedure and the numbers in it indicates the order of calling between the modules.

Here are the details:

  • Server module chooses a proper rpc module (it may be HTTP, gRPC or mysql) to process the requests according the protocol used by the requests;
  • Parse SQL in the request by the parser;
  • With the parsed sql and the catalog/schema module, DataFusion can generate the logical plan;
  • With the logical plan, the corresponding Interpreter is created and logical plan will be executed by it;
  • For the logical plan of normal INSERT SQL, it will be executed through InsertInterpreter;
  • In the InsertInterpreter, write method of Table provided Analytic Engine is called:
    • Write the data into WAL first;
    • Write the data into MemTable then;
  • Before writing to MemTable, the memory usage will be checked. If the memory usage is too high, the flush process will be triggered:
    • Persist some old MemTables as SSTs;
    • Store updates about the new SSTs and the flushed sequence number of WAL to Manifest;
    • Delete the corresponding WAL entries;
  • Server module responds to the client with the execution result.

File: content/en/docs/design/clustering.md

title: "Introduction to Architecture of HoraeDB Cluster"
---Note: Some of the features mentioned in the article have not yet been implemented.

Overview

text
/* Detailed source-code truncated for AI context efficiency. */

The diagram above describes the architecture of a HoraeDB cluster, where some key concepts need to be explained:

  • HoraeMeta Cluster: Takes responsibilities for managing the metadata and resource scheduling of the cluster;
  • Shard(L)/Shard(F): Leader shard and follower shard consisting of multiple tables;
  • HoraeDB: One HoraeDB instance consisting of multiple shards;
  • WAL Service: Write-ahead log service for storing new-written real-time data;
  • Object Storage: Object storage service for storing SST converted from memtable;

From the architecture diagram above, it can be concluded that the compute and storage are separated in the HoraeDB cluster, which makes it easy to implement useful distributed features, such as elastic autoscaling of compute/storage resources, high availability, load balancing, and so on.

Let's dive into some of the key components mentioned above before explaining how these features are implemented.

Shard

Shard is the basic scheduling unit in the cluster, which consists of a group of tables. And the tables in a shard share the same region for better storage locality in the WAL Service, and because of this, it is efficient to recover the data of all tables in the shard by scanning the entire WAL region. For most of implementations of WAL Service, without the shard concept, it costs a lot to recover the table data one by one due to massive random IO, and this case will deteriorate sharply when the number of tables grows to a certain level.

A specific role, Leader or Follower, should be assigned to a shard. A pair of leader-follower shards share the same set of tables, and the leader shard can serve the write and query requests from the client while the follower shard can only serve the read-only requests, and must synchronize the newly written data from the WAL service in order to provide the latest snapshot for data retrieval. Actually, the follower is not needed if the high availability is not required, while with at least one follower, it takes only a short time to resume service by simply switching the Follower to Leader when the HoraeDB instance on which the leader shard exists crashes.

The diagram below concludes the relationship between HoraeDB instance, Shard, Table. As shown in the diagram, the leader and follower shards are interleaved on the HoraeDB instance.

plaintext
┌─HoraeDB Instance0──────┐     ┌─HoraeDB Instance1──────┐
│  ┌─Shard0(L)────────┐  │     │  ┌─Shard0(F)────────┐  │
│  │ ┌────┬────┬────┐ │  │     │  │ ┌────┬────┬────┐ │  │
│  │ │ T0 │ T1 │ T2 │ │  │     │  │ │ T0 │ T1 │ T2 │ │  │
│  │ └────┴────┴────┘ │  │     │  │ └────┴────┴────┘ │  │
│  └──────────────────┘  │     │  └──────────────────┘  │
│                        │     │                        │
│  ┌─Shard1(F)────────┐  │     │  ┌─Shard1(L)────────┐  │
│  │ ┌────┬────┬────┐ │  │     │  │ ┌────┬────┬────┐ │  │
│  │ │ T0 │ T1 │ T2 │ │  │     │  │ │ T0 │ T1 │ T2 │ │  │
│  │ └────┴────┴────┘ │  │     │  │ └────┴────┴────┘ │  │
│  └──────────────────┘  │     │  └──────────────────┘  │
└────────────────────────┘     └────────────────────────┘

Since Shard is the basic scheduling unit, it is natural to introduce some basic shard operations:

  • Create/Drop table to/from a shard;
  • Open/Close a shard;
  • Split one shard into two shards;
  • Merge two shards into one shard;
  • Switch the role of a shard;

With these basic shard operations, some complex scheduling logic can be implemented, e.g. perform an expansion by splitting one shard into two shards and migrating one of them to the new HoraeDB instance.

HoraeMeta

HoraeMeta is implemented by embedding an ETCD inside to ensure consistency and takes responsibilities for cluster metadata management and scheduling.

The cluster metadata includes:

  • Table information, such as table name, table ID, and which cluster the table belongs to;
  • The mapping between table and shard and between shard and HoraeDB instance;
  • ...

As for the cluster scheduling work, it mainly includes:

  • Receiving the heartbeats from the HoraeDB instances and determining the online status of these registered instances;
  • Assigning specific role shards to the registered HoraeDB instances;
  • Participating in table creation by assigning a unique table ID and the most appropriate shard to the table;
  • Performing load balancing through shard operations according to the load information sent with the heartbeats;
  • Performing expansion through shard operations when new instances are registered;
  • Initiating failover through shard operations when old instances go offline;

Route

In order to avoid the overhead of forwarding requests, the communication between clients and the HoraeDB instances is peer-to-peer, that is to say, the client should retrieve routing information from the server before sending any specific write/query requests.

Actually, the routing information is decided by the HoraeMeta, but clients are only allowed the access to it through the HoraeDB instances rather than HoraeMeta, to avoid potential performance issues on the HoraeMeta.

WAL Service & Object Storage

In the HoraeDB cluster, WAL Service and Object Storage exist as separate distributed systems featured with HA, data replication and scalability. Current distributed implementations for WAL Service includes Kafka and OBKV (access OceanBase by its table api), and the implementations for Object Storage include popular object storage services, such as AWS S3, Azure object storage and Aliyun OSS.

The two components are similar in that they are introduced to serve as the underlying storage layer for separating compute and storage, while the difference between two components is obvious that WAL Service is used to store the newly written data from the real-time write requests whose individual size is small but quantity is large, and Object Storage is used to store the read-friendly data files (SST) organized in the background, whose individual size is large and aggregate size is much larger.

The two components make it much easier to implement the horaedb cluster, which features horizontal scalability, high availability and load balancing.

Scalability

Scalability is an important feature for a distributed system. Let's take a look at to how the horizontal scalability of the HoraeDB cluster is achieved.

First, the two storage components (WAL Service and Object Storage) should be horizontally scalable when deciding on the actual implementations for them, so the two storage services can be expanded separately if the storage capacity is not sufficient.

It will be a little bit complex when discussing the scalability of the compute service. Basically, these cases will bring the capacity problem:

  • Massive queries on massive tables;
  • Massive queries on a single large table;
  • Massive queries on a normal table;

For the first case, it is easy to achieve horizontal scalability just by assigning shards that are created or split from old shards to expanded HoraeDB instances.

For the second case, the table partitioning is proposed and after partitioning, massive queries are distributed across multiple HoraeDB instances.

And the last case is the most important and the most difficult. Actually, the follower shard can handle part of the queries, but the number of follower shards is limited by the throughput threshold of the synchronization from the WAL regions. As shown in the diagram below, a pure compute shard can be introduced if the followers are not enough to handle the massive queries. Such a shard is not required to synchronize data with the leader shard, and retrieves the newly written data from the leader/follower shard only when the query comes. As for the SSTs required by the query, they will be downloaded from Object Storage and cached afterwards. With the two parts of the data, the compute resources are fully utilized to execute the CPU-intensive query plan. As we can see, such a shard can be added with only a little overhead (retrieving some data from the leader/follower shard when it needs), so to some extent, the horizontal scalability is achieved.

plaintext
┌HoraeDB─────┬┬─┐
                            ┌──newly written─│  │  │TableN││ │
                            ▼                └──Shard(L/F)┴┴─┘
┌───────┐  Query  ┌HoraeDB─────┬┬─┐
│client │────────▶│  │  │TableN││ │
└───────┘         └──Shard─────┴┴─┘          ┌───────────────┐
                            ▲                │    Object     │
                            └───old SST──────│    Storage    │
                                             └───────────────┘

High Availability

Assuming that WAL service and Object Storage are highly available, the high availability of the HoraeDB cluster can be achieved by such a procedure:

  • When detecting that the heartbeat is broken, HoraeMeta determines that the HoraeDB instance is offline;
  • The follower shards whose paired leader shards exist on the offline instance are switched to leader shards for fast failover;
  • A slow failover can be achieved by opening the crashed shards on another instance if such follower shards don't exist.
text
/* Detailed source-code truncated for AI context efficiency. */

Load Balancing

HoraeMeta collects the instance load information contained in the received heartbeats to create a load overview of the whole cluster, according to which the load balancing can be implemented as an automatic mechanism:

  • Pick a shard on a low-load instance for the newly created table;
  • Migrate a shard from a high-load instance load to another low-load instance;
  • Split the large shard on the high-load instance and migrate the split shards to other low-load instances;

File: content/en/docs/design/compaction_offload.md

title: "Compaction Offload"

Note: This feature is still in development.

This chapter discusses compaction offload, which is designed to separate the compaction workload from the local horaedb nodes and delegate it to external compaction nodes.

Overview

plaintext
┌─────────────────────────────────────────────────────────────────────────┐
│                                                                         │
│                           HoraeMeta Cluster                             │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘
    ▲                ▲                           |                   |
    │                │1.Fetch compaction         │(Monitor compaction│
    │                │  node  info               │node)              │
    |                │                           ▼                   ▼
┌────────────┐  ┌────────────┐              ┌────────────┐   ┌────────────┐
│            │  │            │2.Offload Task│            │   │            │
│  HoraeDB   │  │  HoraeDB   │  ─────────▶  │ Compaction │   │ Compaction │
│            │  │            │  ◀─────────  │ Node       │   │ Node       │
└────────────┘  └────────────┘ 4.Ret Result └────────────┘   └────────────┘
    |                |                            |                 |
    │  5.Update the  │                            │    3.Compact    │
    │    SSTable     │                            │                 │
    ▼                ▼                            ▼                 ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                                                ┌─────────────────────┐  │
│                            Object Storage      │ Temporary Workspace │  │
│                                                └─────────────────────┘  │
└─────────────────────────────────────────────────────────────────────────┘

The diagram above describes the architecture of cluster for compaction offload, where some key concepts need to be explained:

  • Compaction Node: Takes responsibility to handle offloaded compaction tasks. The compaction node receives the compaction task and performs the actual merging of SSTables, then sends back the task result to HoraeDB.
  • HoraeMeta Cluster: HoraeMeta acts as a compaction nodes manager in the compaction offload scenario. It monitors the compaction nodes cluster and schedule the compaction nodes.

The procedure of remote compaction based above architecture diagram is:

  1. HoraeDB servers fetch the information of suitable compaction nodes from the HoraeMeta.
  2. HoraeDB submit the compaction task to the remote compaction node, according to the information fetched from HoraeMeta.
  3. Compaction node executes the task and write results to the temporary workspace.
  4. Compaction node sends compaction results back to HoraeDB.
  5. HoraeDB receives the result, installs the data in temporary workspace and purges compaction input files.

The architecture above makes it easy to implement some wonderful features like load balancing and high availability. Let's dive into the key components in the architecture and talking about how these features are implemented.

Compaction Node

Compaction Node runs the main logic of compaction. It is implemented based on HoraeDB and distinguished by:

  • NodeType: A config parameter used to distinguished the HoraeDB and CompactionNode. This info would be sent to HoraeMeta through heartbeat.

The compaction service is implemented as grpc service.

HoraeMeta

HoraeMeta manages the compaction nodes cluster with CompactionNodeManager, which takes responsibilities for compaction nodes metadata management and scheduling.

The compaction nodes metadata includes:

  • Compaction node information, such as node name, node state;
  • A compaction node name list, used as the key to access compaction node info, for better scheduling with round-robin strategy;
  • ...

As for the compaction nodes scheduling work, it mainly includes:

  • Receiving the heartbeats from the compaction node and determining the online status of these registered nodes.
  • Performing load balancing according to the compaction nodes cluster info.
  • Providing the info of suitable compaction node for HoraeDB when remote compaction execution is needed.

Load Balancing

Load Balancing is critical for compaction nodes cluster to make their overall processing more efficient. The effect of load balancing mainly based on the schedule algorithm for compaction nodes impl in CompactionNodeManager.

(ps: The current implementation of schedule algorithm is round-robin strategy for easiness.)

The main process for the schedule algorithm based on real load is:

  • HoraeMeta collects the compaction nodes load information through the heartbeats to create a load overview of the compaction nodes cluster.
  • Pick a compaction node with low load according to the load overview.

High Availability

The fault tolerance of above architecture can be achieved by such a procedure:

  • When detecting that the heartbeat is broken, HoraeMeta determines that the compaction node is offline.
  • When HoraeMeta can not provide suitable compaction node for HoraeDB or compaction node doesn't return the task result successfully, HoraeDB would switches to run compaction task locally.

File: content/en/docs/design/storage.md

title: "Storage"

The storage engine mainly provides the following two functions：

  1. Persistence of data
  2. Under the premise of ensuring the correctness of the data, organize the data in the most reasonable way to meet the query needs of different scenarios.

This document will introduce the internal implementation of the storage engine in HoraeDB. Readers can refer to the content here to explore how to use HoraeDB efficiently.

Overall Structure

HoraeDB is a distributed storage system based on the share-nothing architecture.

Data between different servers is isolated from each other and does not affect each other. The storage engine in each stand-alone machine is a variant of log-structured merge-tree, which is optimized for time-series scenarios. The following figure shows its core components:

Write Ahead Log (WAL)

A write request will be written to

  1. memtable in memory
  2. WAL in durable storage

Since memtable is not persisted to the underlying storage system in real time, so WAL is required to ensure the reliability of the data in memtable.

On the other hand, due to the design of the distributed architecture, WAL itself is required to be highly available. Now there are following implementations in HoraeDB:

Memtable

Memtable is a memory data structure used to hold recently written table data. Different tables have its corresponding memtable.

Memtable is read-write by default (aka active), and when the write reaches some threshold, it will become read-only and be replaced by a new memtable.

The read-only memtable will be flushed to the underlying storage system in SST format by background thread. After flush is completed, the read-only memtable can be destroyed, and the corresponding data in WAL can also be deleted.

Sorted String Table（SST）

SST is a persistent format for data, which is stored in the order of primary keys of table. Currently, HoraeDB uses parquet format for this.

For HoraeDB, SST has an important option: segment_duration, only SST within the same segment can be merged, which is benefical for time-series data. And it is also convenient to eliminate expired data.

In addition to storing the original data, the statistical information of the data will also be stored in the SST to speed up the query, such as the maximum value, the minimum value, etc.

Compactor

Compactor can merge multiple small SST files into one, which is used to solve the problem of too many small files. In addition, Compactor will also delete expired data and duplicate data during the compaction. In future, compaction maybe add more task, such as downsample.

The current compaction strategy in HoraeDB reference Cassandra:

Manifest

Manifest records metadata of table, SST file, such as: the minimum and maximum timestamps of the data in an SST.

Due to the design of the distributed architecture, the manifest itself is required to be highly available. Now in HoraeDB, there are mainly the following implementations：

  • WAL
  • ObjectStore

ObjectStore

ObjectStore is place where data (i.e. SST) is persisted.

Generally speaking major cloud vendors should provide corresponding services, such as Alibaba Cloud's OSS and AWS's S3.

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#ai-gateway Browse ai-gateway catalog → #api Browse api catalog → #api-gateway Browse api-gateway catalog → #api-management Browse api-management catalog → #apisix Browse apisix catalog → #cloud-native Browse cloud-native catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →