General speaking, region is the minimum element of data distribution, and we can also use it as the unit to distribute computation. This can greatly simplify the routing logic of this distributed planner, by always schedule the computation to the node that currently opening the corresponding region. And is also easy to scale more node for computing since GreptimeDB's data is persisted on shared storage backend like S3. But this is a bit beyond the scope of this specific topic.
Background: Commutativity  Copy Chapter

Commutativity is an attribute that describes whether two operation can exchange their apply order: $P1(P2(R)) \Leftrightarrow P2(P1(R))$. If the equation keeps, we can transform one expression into another form without changing its result. This is useful on rewriting SQL expression, and is the theoretical basis of this RFC.
Take this SQL as an example

As we know projection and filter are commutative (todo: latex), it can be translated to the following two identical plan trees:

Merge Operation  Copy Chapter
This RFC proposes to add a new expression node MergeScan to merge result from several regions in the frontend. It wrap the abstraction of remote data and execution, and expose a TableScan interface to upper level.

This merge operation simply chains all the underlying remote data sources and return RecordBatch, just like a coalesce op. And each remote sources is a gRPC query to datanode via the substrait logical plan interface. The plan is transformed and divided from the original query that comes to frontend.
Commutativity of MergeScan  Copy Chapter
Obviously, The position of MergeScan is the key of the distributed plan. The more closer to the underlying TableScan, the less computation is taken by datanodes. Thus the goal is to pull the MergeScan up as more as possible. The word "pull up" means exchange MergeScan with its parent node in the plan tree, which means we should check the commutativity between the existing expression nodes and the MergeScan. Here I classify all the possibility into five categories:
- Commutative: $P1(P2(R)) \Leftrightarrow P2(P1(R))$
  - filter
  - projection
  - operations that match the partition key
- Partial Commutative: $P1(P2(R)) \Leftrightarrow P1(P2(P1(R)))$
  - $min(R) \rightarrow min(MERGE(min(R)))$
  - $max(R) \rightarrow max(MERGE(max(R)))$
- Conditional Commutative: $P1(P2(R)) \Leftrightarrow P3(P2(P1(R)))$
  - $count(R) \rightarrow sum(count(R))$
- Transformed Commutative: $P1(P2(R)) \Leftrightarrow P1(P3(R)) \Leftrightarrow P3(P1(R))$
  - $avg(R) \rightarrow sum(R)/count(R)$
- Non-commutative
  - sort
  - join
  - percentile
Steps to plan  Copy Chapter

After establishing the set of commutative relations for all expressions, we can begin transforming the logical plan. There are four steps:
  - Add a merge node before table scan
  - Evaluate commutativity in a bottom-up way, stop at the first non-commutative node
  - Divide the TableScan to scan over partitions
  - Execute
First insert the MergeScan on top of the bottom TableScan node. Then examine the commutativity start from the MergeScan node transform the plan tree based on the result. Stop this process on the first non-commutative node.

Then in the physical planning phase, convert the sub-tree below MergeScan into a remote query request and dispatch to all the regions. And let the MergeScan to receive the results and feed to it parent node.
To simplify the overall complexity, any error in the procedure will lead to a failure to the entire query, and cancel all other parts.
Alternatives

Spill  Copy Chapter

If only consider the ability of processing large dataset, we can enable DataFusion's spill ability to temporary persist intermediate data into disk, like the "swap" memory. But this will lead to a super slow performance and very large write amplification.
Future Work

As described in the Motivation section we can further explore the distributed planner on the physical execution level, by introducing mechanism like Spark's shuffle to improve parallelism and reduce intermediate pipeline breaker's stage.
---
Rfcs/2023 07 06 Table Engine Refactor  Copy Section
---
Feature Name: table-engine-refactor
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/1869
Date: 2023-07-06
Author: "Yingwen <[email protected]>"
---
Refactor Table Engine
----------------------
Summary

Refactor table engines to address several historical tech debts.
Motivation

Both Frontend and Datanode have to deal with multiple regions in a table. This results in code duplication and additional burden to the Datanode.
Before:

Datanodes can update the same manifest file for a table as regions are assigned to different nodes in the cluster. We also have to run procedures on Datanode to ensure the table manifest is consistent with region manifests. "Table" in a Datanode is a subset of the table's regions. The Datanode is much closer to RegionServer in HBase which only deals with regions.
In cluster mode, we store table metadata in etcd and table manifest. The table manifest becomes redundant. We can remove the table manifest if we refactor the table engines to region engines that only care about regions. What's more, we don't need to run those procedures on Datanode.
After:

This RFC proposes to refactor table engines into region engines as a first step to make the Datanode acts like a RegionServer.

Details

Overview  Copy Chapter
We plan to refactor the TableEngine trait into RegionEngine gradually. This RFC focuses on the mito engine as it is the default table engine and the most complicated engine.
Currently, we built MitoEngine upon StorageEngine that manages regions of the mito engine. Since MitoEngine becomes a region engine, we could combine StorageEngine with MitoEngine to simplify our code structure.
The chart below shows the overall architecture of the MitoEngine.

We replace the RegionWriter with RegionWorker to process write requests and DDL requests.

Metadata  Copy Chapter

We also merge region's metadata with table's metadata. It should make metadata much easier to maintain.

Drawback

This is a breaking change.
Future Work

- Rename TableEngine to RegionEngine
- Simplify schema relationship in the mito engine
- Refactor the Datanode into a RegionServer.
---
Rfcs/2023 07 10 Metric Engine  Copy Section
---
Feature Name: metric-engine
Tracking Issue: TBD
Date: 2023-07-10
Author: "Ruihang Xia <[email protected]>"
---
Summary
A new metric engine that can significantly enhance our ability to handle the tremendous number of small tables in scenarios like Prometheus metrics, by leveraging a synthetic wide table that offers storage and metadata multiplexing capabilities over the existing engine.
Motivation
The concept "Table" in GreptimeDB is a bit "heavy" compared to other time-series storage like Prometheus or VictoriaMetrics. This has lots of disadvantages in aspects from performance, footprint, and storage to cost.
Details
Top level description  Copy Chapter
- User Interface
    This feature will add a new type of storage engine. It might be available to be an option like with ENGINE=mito or an internal interface like auto create table on Prometheus remote write. From the user side, there is no difference from tables in mito engine. All the DDL like CREATE, ALTER and DML like SELECT should be supported.
- Implementation Overlook
    This new engine doesn't re-implement low level components like file R/W etc. It's a wrapper layer over the existing mito engine, with extra storage and metadata multiplexing capabilities. I.e., it expose multiple table based on one mito engine table like this:

The following parts will describe these implementation details:
- How to route these metric region tables and how those table are distributed
- How to maintain the schema and other metadata of the underlying mito engine table
- How to maintain the schema of metric engine table
- How the query goes
Routing  Copy Chapter
Before this change, the region route rule was based on a group of partition keys. Relation of physical table to region is one-to-many.

And for metric engine tables, the key difference is we split the concept of "physical table" and "logical table". Like the previous ASCII chart, multiple logical tables are based on one physical table. The relationship of logical table to region becomes many-to-many. Thus, we must include the table name (of logical table) into partition rules.
Consider the partition/route interface is a generic map of string array to region id, all we need to do is to insert logical table name into the request:

The next question is, where to do this conversion? The basic idea is to dispatch different routing behavior based on the engine type. Since we have all the necessary information in frontend, it's a good place to do that. And can leave meta server untouched. The essential change is to associate engine type with route rule.
Physical Region Schema  Copy Chapter
The idea "physical wide table" is to perform column-level multiplexing. I.e., map all logical columns to physical columns by their names.

This approach is very straightforward but has one problem. It only works when two columns have different semantic type (time index, tag or field) or data types but with the same name. E.g., CREATE TABLE t1 (c1 timestamp(3) TIME INDEX) and CREATE TABLE t2 (c1 STRING PRIMARY KEY).
One possible workaround is to prefix each column with its data type and semantic type, like _STRING_PK_c1. However, considering the primary goal at present is to support data from monitoring metrics like Prometheus remote write, it's acceptable not to support this at first because data types are often simple and limited here.

The next point is changing the physical table's schema. This is only needed when creating a new logical table or altering the existing table. Typically speaking, table creating and altering are explicit. We only need to emit an add column request to underlying physical table on processing logical table's DDL. GreptimeDB can create or alter table automatically on some protocols, but the internal logic is the same.
Also for simplicity, we don't support shrinking the underlying table at first. This can be achieved by introducing mechanism on the physical column.
Frontend needs not to keep physical table's schema.
Metadata of physical regions  Copy Chapter
Those metric engine regions need to store extra metadata like the schema of logical table or all logical table's name. That information is relatively simple and can be stored in a format like key-value pair. For now, we have to use another physical mito region for metadata. This involves an issue with region scheduling. Since we don't have the ability to perform affinity scheduling, the initial version will just assume the data region and metadata region are in the same instance. See alternatives - other storage for physical region's metadata for possible future improvement.
Here is the schema of metadata region and how we would use it. The CREATE TABLE clause of metadata region looks like the following. Notice that it wouldn't be actually created by SQL.

The ts field is just a placeholder -- for the constraints that a mito region must contain a time index field. It will always be 0. The other two fields key and value will be used as a k-v storage. It contains two group of key
    - __table_<TABLE_NAME> is used for marking table existence. It doesn't have value.
    - __column_<TABLE_NAME>_<COLUMN_NAME> is used for marking table existence, the value is column's semantic type.
Physical region implementation  Copy Chapter
This RFC proposes to add a new region implementation named "MetricRegion". As showed in the first chart, it's wrapped over the existing mito region. This section will describe the implementation details. Firstly, here is a chart shows how the region hierarchy looks like:

All upper levels only see the Metric Region. E.g., Meta Server schedules on this region, or Frontend routes requests to this Metrics Region's id. To be scheduled (open or close etc.), Metric Region needs to implement its own procedures. Most of those procedures can be simply assembled from underlying Mito Regions', but those related to data like alter or drop will have its own new logic.
Another point is region id. Since the region id is used widely from meta server to persisted state, it's better to keep it unchanged. This means we can't use the same id for two regions, but one for each. To achieve this, this RFC proposes a concept named "region id group". A region id group is a group of region ids that are bound for different purposes. Like the two underlying regions here. 
This preserves the first 8 bits of the u32 region number for grouping. Each group has one main id (the first one) and other sub ids (the rest non-zero ids). All components other than the region implementation itself doesn't aware of the existence of region id group. They only see the main id. The region implementation is in response of managing and using the region id group.

Routing in meta server  Copy Chapter
From previous sections, we can conclude the following points about routing:
- Each "logical table" has its own, universe unique table id.
- Logical table doesn't have physical region, they share the same physical region with other logical tables.
- Route rule of logical table's is a strict subset of physical table's.
To associate the logical table with physical region, we need to specify necessary information in the create table request. Specifically, the table type and its parent table. This require to change our gRPC proto's definition. And once meta recognize the table to create is a logical table, it will use the parent table's region to create route entry.
And to reduce the consumption of region failover (which need to update the physical table route info), we'd better to split the current route table structure into two parts:

By doing this on each failover the meta server only needs to update the second node_route map and leave the first one untouched.
Query  Copy Chapter
Like other existing components, a user query always starts in the frontend. In the planning phase, frontend needs to fetch related schemas of the queried table. This part is the same. I.e., changes in this RFC don't affect components above the Table abstraction.
Alternatives
Other routing method  Copy Chapter
We can also do this "special" route rule in the meta server. But there is no difference with the proposed method.
Other storage for physical region's metadata  Copy Chapter
Once we have implemented the "region family" that allows multiple physical schemas exist in one region, we can store the metadata and table data into one region.
Before that, we can also let the MetricRegion holds a KvBackend to access the storage layer directly. But this breaks the abstraction in some way.
Drawbacks
Since the physical storage is mixed together. It's hard to do fine-grained operations in table level. Like configuring TTL, memtable size or compaction strategy in table level. Or define different partition rules for different tables. For scenarios like this, it's better to move the table out of metrics engine and "upgrade" it to a normal mito engine table. This requires a migration process in a low cost. And we have to ensure data consistency during the migration, which may require a out-of-service period.
---
Rfcs/2023 08 04 Table Trait Refactor  Copy Section
---
Feature Name: table-trait-refactor
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/2065
Date: 2023-08-04
Author: "Ruihang Xia <[email protected]>"
---
Refactor Table Trait
--------------------
Summary

Refactor Table trait to adapt the new region server architecture and make code more straightforward.
Motivation

The Table is designed in the background of both frontend and datanode keeping the same concepts. And all the operations are served by a Table. However, in our practice, we found that not all the operations are suitable to be served by a Table. For example, the Table doesn't hold actual physical data itself, thus operations like write or alter are simply a proxy over underlying regions. And in the recent refactor to datanode (rfc table-engine-refactor), we are changing datanode to region server that is only aware of Region things. This also calls for a refactor to the Table trait.
Details
Definitions  Copy Chapter
The current Table trait contains the following methods:

We can divide those methods into three categories from the perspective of functionality:
|     Retrieve Metadata      | Manipulate Data |    Read Data     |
| :------------------------: | :-------------: | :--------------: |
|          schema          |    insert     | scan_to_stream |
|        table_info        |     alter     |                  |
|        table_type        |    delete     |                  |
| supports_filter_pushdown |     flush     |                  |
|       region_stats       |     close     |                  |
|     contains_region      |    compact    |                  |
|        statistics        |                 |                  |
And considering most of the access to metadata happens in frontend, like route or query; and all the persisted data are stored in regions; while only the query engine needs to read data. We can divide the Table trait into three concepts:
- struct Table provides metadata:

- Requests to region server
  - InsertRequest
  - AlterRequest
  - DeleteRequest
  - FlushRequest
  - CompactRequest
  - CloseRequest
- trait DataSource provides data (RecordBatch)

Use Table ')" title="Copy chapter prompt for LLMs"> Copy Chapter
Table will only be used in frontend. It's constructed from the OpenTableRequest or CreateTableRequest.
Table also provides a method to_data_source to generate a DataSource from itself. But this method is only for non-TableType::Base tables (i.e., TableType::View and TableType::Temporary) because TableType::Base table doesn't hold actual data itself. Its DataSource should be constructed from the Region directly (in other words, it's a remote query).
And it requires some extra information to construct a DataSource, named TableSourceProvider:

Use DataSource ')" title="Copy chapter prompt for LLMs"> Copy Chapter
DataSource will be adapted to the TableProvider from DataFusion that can be scan()ed in a TableScan plan.
In frontend this is done in the planning phase. And datanode will have one implementation for Region to generate record batch stream.
Interact with RegionServer  Copy Chapter
Previously, persisted state change operations were through the old Table trait, like said before. Now they will come from the action source, like the procedure or protocol handler directly to the region server. E.g., on alter table, the corresponding procedure will generate its AlterRequest and send it to regions. Or write request will be split in frontend handler, and sent to regions. Table only provides necessary metadata like route information if needed, but not the necessary part anymore.
Implement temporary table  Copy Chapter
Temporary table is a special table that doesn't revolves to any persistent physical region. Examples are:
- the Numbers table for testing, which produces a record batch that contains 0-100 integers.
- tables in information schema. It is an interface for querying catalog's metadata. The contents are generated on the fly with information from CatalogManager. The CatalogManager can be held in TableFactory.
- Function table that produces data generated by a formula or a function. Like something that always sin(current_timestamp()).
Relationship among those components  Copy Chapter
Here is a diagram to show the relationship among those components, and how they interact with each other.

Drawback

This is a breaking change.
---
Rfcs/2023 08 13 Metadata Txn  Copy Section
---
Feature Name: Update Metadata in single transaction
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/1715
Date: 2023-08-13
Author: "Feng Yangsen <[email protected]>, Xu Wenkang <[email protected]>"
---
Summary

Update Metadata in single transaction.
Motivation

Currently, multiple transactions are involved during the procedure. This implementation is inefficient, and it's hard to make data consistent. Therefore, We can update multiple metadata in a single transaction.
Details 

Now we have the following table metadata keys:
TableInfo

TableRoute

DatanodeTable

TableNameKey

These table metadata only updates in the following operations.
Region Failover  Copy Chapter

It needs to update TableRoute key and DatanodeTable keys. If the TableRoute equals the Snapshot of TableRoute submitting the Failover task, then we can safely update these keys.
After submitting Failover tasks to acquire locks for execution, the TableRoute may be updated by another task. After acquiring the lock, we can get the latest TableRoute again and then execute it if needed.
Create Table DDL  Copy Chapter

Creates all of the above keys. TableRoute, TableInfo, should be empty.
The TableNameKey's lock will be held by the procedure framework.
Drop Table DDL  Copy Chapter
TableInfoKey and NextTableRouteKey will be added with  __removed- prefix, and the other above keys will be deleted.  The transaction will not compare any keys.
Alter Table DDL  Copy Chapter
1. Rename table, updates TableInfo and TableName. Compares TableInfo, and the new TableNameKey should be empty, and TableInfo should equal the Snapshot when submitting DDL.
The old and new TableNameKey's lock will be held by the procedure framework.
2. Alter table, updates TableInfo. TableInfo should equal the Snapshot when submitting DDL.
---
Rfcs/2023 11 03 Inverted Index  Copy Section
---
Feature Name: Inverted Index for SST File
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/2705
Date: 2023-11-03
Author: "Zhong Zhenchi <[email protected]>"
---
Summary

This RFC proposes an optimization towards the storage engine by introducing an inverted indexing methodology aimed at optimizing label selection queries specifically pertaining to Metrics with tag columns as the target for optimization.
Introduction

In the current system setup, in the Mito Engine, the first column of Primary Keys has a Min-Max index, which significantly optimizes the outcome. However, there are limitations when it comes to other columns, primarily tags. This RFC suggests the implementation of an inverted index to provide enhanced filtering benefits to bridge these limitations and improve overall system performance.
Design Detail
Inverted Index  Copy Chapter
The primary aim of the proposed inverted index is to optimize tag columns in the SST Parquet Files within the Mito Engine. The mapping and construction of an inverted index, from Tag Values to Row Groups, enables efficient logical structures that provide faster and more flexible queries.
When scanning SST Files, pushed-down filters applied to a respective Tag's inverted index, determine the final Row Groups to be indexed and scanned, further bolstering the speed and efficiency of data retrieval processes.
Index Format  Copy Chapter
The Inverted Index for each SST file follows the format shown below:

The structure inside each Inverted Index is as followed:

The format is encapsulated by a footer:

The footer_payload is presented in protobuf encoding of InvertedIndexFooter.
The complete format is containerized in Puffin with the type defined as greptime-inverted-index-v1.
Protobuf Details  Copy Chapter
The InvertedIndexFooter is defined in the following protobuf structure:

Bitmap  Copy Chapter
Bitmaps are used to represent indices of fixed-size groups. Rows are divided into groups of a fixed size, defined in the InvertedIndexMeta as row_count_in_group.
For example, when row_count_in_group is 4096, it means each group has 4096 rows. If there are a total of 10000 rows, there will be 3 groups in total. The first two groups will have 4096 rows each, and the last group will have 1808 rows. If the indexed values are found in row 200 and 9000, they will correspond to groups 0 and 2, respectively. Therefore, the bitmap should show 0 and 2.
Bitmap is implemented using BitVec, selected due to its efficient representation of dense data arrays typical of indices of groups.

Finite State Transducer (FST)  Copy Chapter
FST is a highly efficient data structure ideal for in-memory indexing. It represents ordered sets or maps where the keys are bytes. The choice of the FST effectively balances the need for performance, space efficiency, and the ability to perform complex analyses such as regular expression matching.
The conventional usage of FST and u64 values has been adapted to facilitate indirect indexing to row groups. As the row groups are represented as Bitmaps, we utilize the u64 values split into bitmap's offset (higher 32 bits) and size (lower 32 bits) to represent the location of these Bitmaps. 
API Design  Copy Chapter
Two APIs InvertedIndexBuilder for building indexes and  InvertedIndexSearcher for querying indexes are designed:

---
Rfcs/2023 11 07 Region Migration  Copy Section
---
Feature Name: Region Migration Procedure
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/2700
Date: 2023-11-03
Author: "Xu Wenkang <[email protected]>"
---
Summary

This RFC proposes a way that brings the ability of Meta Server to move regions between the Datanodes.
Motivation

Typically, We need this ability in the following scenarios:
- Migrate hot-spot Regions to idle Datanode
- Move the failure Regions to an available Datanode
Details

Only the red nodes will persist state after it has succeeded, and other nodes won't persist state. (excluding the Start and End nodes).
Steps  Copy Chapter
The persistent context: It's shared in each step and available after recovering. It will only be updated/stored after the Red node has succeeded.
Values: 
- region_id: The target leader region.
- peer: The target datanode.
- close_old_leader: Indicates whether close the region. 
- leader_may_unreachable: It's used to support the failover procedure.
The Volatile context: It's shared in each step and available in executing (including retrying). It will be dropped if the procedure runner crashes.
Select Candidate  Copy Section
The Persistent state: Selected Candidate Region.
Update Metadata(Down)  Copy Section
The Persistent context:
- The (latest/updated) version of TableRouteValue, It will be used in the step of Update Metadata(Up).
Downgrade Leader  Copy Section

This step sends an instruction via heartbeat and performs:
1. Downgrades leader region.
2. Retrieves the last_entry_id (if available).
If the target leader region is not found: 
- Sets close_old_leader to true.
- Sets leader_may_unreachable to true.
If the target Datanode is unreachable:
- Waits for region lease expired.
- Sets close_old_leader to true.
- Sets leader_may_unreachable to true.
The Persistent context:
None
The Persistent state: 
- last_entry_id
*Passes to next step.

Upgrade Candidate  Copy Section

This step sends an instruction via heartbeat and performs:
1. Replays the WAL to latest(last_entry_id).
2. Upgrades the candidate region.
If the target region is not found: 
- Rollbacks.
- Notifies the failover detector if leader_may_unreachable == true.
- Exits procedure.
If the target Datanode is unreachable:
- Rollbacks.
- Notifies the failover detector if leader_may_unreachable == true.
- Exits procedure.
The Persistent context:
None
Update Metadata(Up)  Copy Section

This step performs
1. Switches Leader.
2. Removes Old Leader(Opt.).
3. Moves Old Leader to follower(Opt.).
The TableRouteValue version should equal the TableRouteValue's version in Persistent context. Otherwise, verifies whether TableRouteValue already updated.
The Persistent context:
None
Close Old Leader(Opt.)  Copy Section

This step sends a close region instruction via heartbeat.
If the target leader region is not found: 
- Ignore.
If the target Datanode is unreachable: 
- Ignore.
Open Candidate(Opt.)  Copy Section

This step sends an open region instruction via heartbeat and waits for conditions to be met (typically, the condition is that the last_entry_id of the Candidate Region is very close to that of the Leader Region or the latest).
If the target Datanode is unreachable: 
- Exits procedure.
---
Rfcs/2023 12 22 Enclose Column Id  Copy Section
---
Feature Name: Enclose Column Id
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/2982
Date: 2023-12-22
Author: "Ruihang Xia <[email protected]>"
---
Summary

This RFC proposes to enclose the usage of ColumnId into the region engine only.
Motivation

ColumnId is an identifier for columns. It's assigned by meta server, stored in TableInfo and RegionMetadata and used in region engine to distinguish columns.
At present, Both Frontend, Datanode and Metasrv are aware of ColumnId but it's only used in region engine. Thus this RFC proposes to remove it from Frontend (mainly used in TableInfo) and Metasrv.
Details
ColumnId is used widely on both read and write paths. Removing it from Frontend and Metasrv implies several things:
- A column may have different column id in different regions.
- A column is identified by its name in all components.
- Column order in the region engine is not restricted, i.e., no need to be in the same order with table info.
The first thing doesn't matter IMO. This concept doesn't exist anymore outside of region server, and each region is autonomous and independent -- the only guarantee it should hold is those columns exist. But if we consider region repartition, where the SST file would be re-assign to different regions, things would become a bit more complicated. A possible solution is store the relation between name and ColumnId in the manifest, but it's out of the scope of this RFC. We can likely give a workaround by introducing a indirection mapping layer of different version of partitions.
And more importantly, we can still assume columns have the same column ids across regions. We have procedure to maintain consistency between regions and the region engine should ensure alterations are idempotent. So it is possible that region repartition doesn't need to consider column ids or other region metadata in the future.
Users write and query column by their names, not by ColumnId or something else. The second point also means to change the column reference in ScanRequest from index to name. This change can hugely alleviate the misuse of the column index, which has given us many surprises.
And for the last one, column order only matters in table info. This order is used in user-faced table structure operation, like add column, describe column or as the default order of INSERT clause. None of them is connected with the order in storage.
Drawback

Firstly, this is a breaking change. Delivering this change requires a full upgrade of the cluster. Secondly, this change may introduce some performance regression. For example, we have to pass the full table name in the ScanRequest instead of the ColumnId. But this influence is very limited, since the column index is only used in the region engine.
Alternatives
There are two alternatives from the perspective of "what can be used as the column identifier":
- Index of column to the table schema
- ColumnId of that column
The first one is what we are using now. By choosing this way, it's required to keep the column order in the region engine the same as the table info. This is not hard to achieve, but it's a bit annoying. And things become tricky when there is internal column or different schemas like those stored in file format. And this is the initial purpose of this RFC, which is trying to decouple the table schema and region schema.
The second one, in other hand, requires the ColumnId should be identical in all regions and TableInfo. It has the same drawback with the previous alternative, that the TableInfo and RegionMetadata are tighted together. Another point is that the ColumnId is assigned by the Metasrv, who doesn't need it but have to maintain it. And this also limits the functionality of ColumnId, by taking the ability of assigning it from concrete region engine.
---
Rfcs/2024 01 17 Dataflow Framework  Copy Section
---
Feature Name: Dataflow Framework
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/3187
Date: 2024-01-17
Author: "Discord9 <[email protected]>"
---
Summary

This RFC proposes a Lightweight Module for executing continuous aggregation queries on a stream of data.
Motivation

Being able to do continuous aggregation is a very powerful tool. It allows you to do things like:
1. downsample data from i.e. 1 milliseconds to 1 second
2. calculate the average of a stream of data
3. Keeping a sliding window of data in memory
In order to do those things while maintaining a low memory footprint, you need to be able to manage the data in a smart way. Hence, we only store necessary data in memory, and send/recv data deltas to/from the client.
Details
System boundary / What it's and isn't  Copy Chapter

- GreptimeFlow provides a way to perform continuous aggregation over time-series data.
- It's not a complete streaming-processing system. Only a must subset functionalities are provided.
- Flow can process a configured range of fresh data. Data exceeding this range will be dropped directly. Thus it cannot handle random datasets (random on timestamp).
- Both sliding windows (e.g., latest 5m from present) and fixed windows (every 5m from some time) are supported. And these two are the major targeting scenarios.
- Flow can handle most aggregate operators within one table(i.e. Sum, avg, min, max and comparison operators). But others (join, trigger, txn etc.) are not the target feature.
Framework  Copy Chapter

- Greptime Flow's is built on top of Hydroflow.
- We have three choices for the Dataflow/Streaming process framework for our simple continuous aggregation feature:
1. Based on the timely/differential dataflow crate that materialize based on. Later, it's proved too obscure for a simple usage, and is hard to customize memory usage control.
2. Based on a simple dataflow framework that we write from ground up, like what arroyo or risingwave did, for example the core streaming logic of arroyo only takes up to 2000 line of codes. However, it means maintaining another layer of dataflow framework, which might seem easy in the beginning, but I fear it might be too burdensome to maintain once we need more features.
3. Based on a simple and lower level dataflow framework that someone else write, like hydroflow, this approach combines the best of both worlds. Firstly, it boasts ease of comprehension and customization. Secondly, the dataflow framework offers precisely the necessary features for crafting uncomplicated single-node dataflow programs while delivering decent performance.
Hence, we choose the third option, and use a simple logical plan that's anagonistic to the underlying dataflow framework, as it only describe how the dataflow graph should be doing, not how it do that. And we built operator in hydroflow to execute the plan. And the result hydroflow graph is wrapped in a engine that only support data in/out and tick event to flush and compute the result. This provide a thin middle layer that's easy to maintain and allow switching to other dataflow framework if necessary.
Deploy mode and protocol  Copy Chapter

- Greptime Flow is an independent streaming compute component. It can be used either within a standalone node or as a dedicated node at the same level as frontend in distributed mode.
- It accepts insert request Rows, which is used between frontend and datanode.
- New flow job is submitted in the format of modified SQL query like snowflake do, like: CREATE TASK avg_over_5m WINDOW_SIZE = "5m" AS SELECT avg(value) FROM table WHERE time > now() - 5m GROUP BY time(1m). Flow job then got stored in Metasrv.
- It also persists results in the format of Rows to frontend.
- The query plan uses Substrait as codec format. It's the same with GreptimeDB's query engine.
- Greptime Flow needs a WAL for recovering. It's possible to reuse datanode's.
The workflow is shown in the following diagram

Lifecycle of data  Copy Chapter

- New data is inserted into frontend like before. Frontend will mirror insert request to Flow node if there is configured flow job.
- Depending on the timestamp of incoming data, flow will either drop it (outdated data) or process it (fresh data).
- Greptime Flow will periodically write results back to the result table through frontend.
- Those result will then be written into a result table stored in datanode.
- A small table of intermediate state is kept in memory, which is used to calculate the result.
Supported operations  Copy Chapter

- Greptime Flow accepts a configurable "materialize window", data point exceeds that time window is discarded.
- Data within that "materialize window" is queryable and updateable.
- Greptime Flow can handle partitioning, if and only if the input query can be transformed to a fully partitioned plan according to the existing commutative rules. Otherwise the corresponding flow job has to be calculated in a single node.
- Notice that Greptime Flow has to see all the data belongs to one partition.
- Deletion and duplicate insertion are not supported at early stage.
Miscellaneous   Copy Chapter

- Greptime Flow can translate SQL to it's own plan, however only a selected few aggregate function is supported for now, like min/max/sum/count/avg
- Greptime Flow's operator is configurable in terms of the size of the materialize window, whether to allow delay of incoming data etc., so simplest operator can choose to not tolerate any delay to save memory.
Future Work

- Support UDF that can do one-to-one mapping. Preferably, we can reuse the UDF mechanism in GreptimeDB.
- Support join operator.
- Design syntax for config operator for different materialize window and delay tolerance.
- Support cross partition merge operator that allows complex query plan that not necessary accord with partitioning rule to communicate between nodes and create final materialize result.
- Duplicate insertion, which can be reverted easily within the current framework, so supporting it could be easy
- Deletion within "materialize window", this requires operators like min/max to store all inputs within materialize window, which might require further optimization.
---
Rfcs/2024 08 06 Json Datatype  Copy Section
---
Feature Name: Json Datatype
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/4230
Date: 2024-8-6
Author: "Yuhan Wang <[email protected]>"
---
Summary

This RFC proposes a method for storing and querying JSON data in the database.
Motivation

JSON is widely used across various scenarios. Direct support for writing and querying JSON can significantly enhance the database's flexibility.
Details
Storage and Query  Copy Chapter
GreptimeDB's type system is built on Arrow/DataFusion, where each data type in GreptimeDB corresponds to a data type in Arrow/DataFusion. The proposed JSON type will be implemented on top of the existing Binary type, leveraging the current datatype::value::Value and datatype::vectors::BinaryVector implementations, utilizing the JSONB format as the encoding of JSON data. JSON data is stored and processed similarly to binary data within the storage layer and query engine.
This approach brings problems when dealing with insertions and queries of JSON columns.
Insertion  Copy Chapter
Users commonly write JSON data as strings. Thus we need to make conversions between string and JSONB. There are 2 ways to do this:
1. MySQL and PostgreSQL servers provide auto-conversions between strings and JSONB. When a string is inserted into a JSON column, the server will try to parse the string as JSON and convert it to JSONB. The non-JSON strings will be rejected.
2. A function parse_json is provided to convert string to JSONB. If the string is not a valid JSON string, the function will return an error.
For example, in MySQL client:

Are both valid.
The dataflow of the insertion process is as follows:

Servers identify JSON column through column schema and perform auto-conversions. But when using prepared statements and binding parameters, the corresponding cached plans in datafusion generated by prepared statements cannot identify JSON columns. Under this circumstance, the servers identify JSON columns through the given parameters and perform auto-conversions.
The following is an example of inserting JSON data through prepared statements:

Query  Copy Chapter
Correspondingly, users prefer to display JSON data as strings. Thus we need to make conversions between JSON data and strings before presenting JSON data. There are also 2 ways to do this: auto-conversions on MySQL and PostgreSQL servers, and function json_to_string.
For example, in MySQL client:

Will both return the JSON as human-readable strings.
Specifically, to perform auto-conversions, we attach a message to JSON data in the metadata of Field in Arrow/Datafusion schema when scanning a JSON column. Frontend servers could identify JSON data and convert it to strings.
The dataflow of the query process is as follows:

However, if a function uses JSON type as its return type, the metadata method mentioned above is not applicable. Thus the functions of JSON type should specify the return type explicitly instead of returning a JSON type, such as json_get_int and json_get_float which return corresponding data of INT and FLOAT type respectively.
Functions  Copy Chapter

Similar to the common JSON type, JSON data can be queried with functions.
For example:

And more functions can be added in the future.
Drawbacks
As a general purpose JSON data type, JSONB may not be as efficient as specialized data types for specific scenarios.
The auto-conversion mechanism is not supported in all scenarios. We need to find workarounds for these scenarios.
Alternatives
Extract and flatten JSON schema to store in a structured format through pipeline. For nested data, we can provide nested types like STRUCT or ARRAY.
---
Rfcs/2025 02 06 Remote Wal Purge  Copy Section
---
Feature Name: Remote WAL Purge
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/5474
Date: 2025-02-06
Author: "Yuhan Wang <[email protected]>"
---
Summary
This RFC proposes a method for purging remote WAL in the database.
Motivation
Currently only local wal entries are purged when flushing, while remote wal does nothing.
Details

Steps  Copy Chapter
Before purge  Copy Section
Before purging remote WAL, metasrv needs to know:
1. last_entry_id of each region.
2. kafka_topic_last_entry_id which is the last entry id of the topic in use. Can be lazily updated and needed when region has empty memtable.
3. Kafka topics that each region uses.
The states are maintained through:
1. Heartbeat: Datanode sends last_entry_id to metasrv in heartbeat. As for regions with empty memtable, last_entry_id should equals to kafka_topic_last_entry_id.
2. Metasrv maintains a topic-region map to know which region uses which topic.
kafka_topic_last_entry_id will be maintained by the region itself. Region will update the value after k heartbeats if the memtable is empty.
Purge procedure  Copy Section
We can better handle locks utilizing current procedure. It's quite similar to the region migration procedure.
After a period of time, metasrv will submit a purge procedure to ProcedureManager. The purge will apply to all topics.
The procedure is divided into following stages:
1. Preparation:
   - Retrieve last_entry_id of each region kvbackend.
   - Choose regions that have a relatively small last_entry_id as candidate regions, which means we need to send a flush request to these regions.
2. Communication:
   - Send flush requests to candidate regions.
3. Purge:
   - Choose proper entry id to delete for each topic. The entry should be the smallest last_entry_id - 1 among all regions. 
   - Delete legacy entries in Kafka.
   - Store the last_purged_entry_id in kvbackend. It should be locked to prevent other regions from replaying the purged entries.
After purge  Copy Section
After purge, there may be some regions that have last_entry_id smaller than the entry we just deleted. It's legal since we only delete the entries that are not needed anymore.
When restarting a region, it should query the last_purged_entry_id from metasrv and replay from min(last_entry_id, last_purged_entry_id).
Error handling  Copy Section
No persisted states are needed since all states are maintained in kvbackend.
Retry when failed to retrieving metadata from kvbackend.
Alternatives
Purge time can depend on the size of the WAL entries instead of a fixed period of time, which may be more efficient.
---
Rfcs/2025 06 20 Repartition  Copy Section
---
Feature Name: Repartition
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/6558
Date: 2025-06-20
Author: "Ruihang Xia <[email protected]>"
---
Summary
This RFC proposes a method for repartitioning a table, to adjust the partition rule and data distribution.
Motivation
With time passing, the data distribution and skew pattern of a table might change. We need a way to repartition the table to suit the new pattern.
Details
Here is a rough workflow diagram of the entire repartition process, each step is described in detail below.

Preprocessing  Copy Chapter
This phase is for static analysis of the new partition rule. The server can know whether the repartitioning is possible, how to do the repartitioning, and how much resources are needed.
In theory, the input and output partition rules for repartitioning can be completely unrelated. But in practice, to avoid a very large change set, we'll only allow two simple kinds of change. One splits one region into two regions (region split) and another merges two regions into one (region merge).
After validating the new partition rule using the same validation logic as table creation, we compute the difference between the old and new partition rules. The resulting diff may contain several independent groups of changes. During subsequent processing, each group of changes can be handled independently and can succeed or fail without affecting other groups or creating non-idempotently retryable scenarios.
Next, we generate a repartition plan for each group of changes. Each plan contains this information for all regions involved in that particular plan. And one target region will only be referenced by a single plan.
With those plans, we can determine the resource requirements for the repartition operation, where resources here primarily refer to Regions. Metasrv will coordinate with PaaS layer to pre-allocate the necessary regions at this stage. These new regions start completely empty, and their metadata and manifests will be populated during subsequent modification steps.
Data Processing  Copy Chapter
This phase is primarily for region's change, including region's metadata (route table and the corresponding rule) and manifest.
Once we start processing one plan through a procedure, we'll first stop the region's compaction and snapshot. This is to avoid any states being removed due to compaction (which may removes old SST files) and snapshot (which may removes old manifest files).
Metasrv will trying to update the metadata of partition, or the region route table (related to PartitionRuleManager). This step is in the "no ingestion" scope, so no new data will be ingested. Since this won't take much time, the affection to the cluster is minimized. Metasrv will also update the region rule to corresponding regions on Datanodes. 
Every regions and all the ingestion requests to the region server will have a version of region rule, to identify under which rule the request is processed. The version can be something like hash(region_rule). Once the region rule on region server is updated, all ingestion request with old rule will be rejected, and all requests with new rule will be accepted but not visible. They can still be flushed to persisted storage, but their version change (new manifest) will be staged.
Then region 0 (or let metasrv to pick any operational region) will compute the new manifests for all target regions. This step is done by first reading all old manifests, and remapping the files with new partition rule, to get the content of new manifests. Notice this step only handles the manifests before region rule change on region server, and won't touch those staged manifests, as they are already with the new rule.
Those new manifest will be submitted to the corresponding target regions by region 0 via a RegionEdit request. If this request falls after a few retries, region 0 will try to rollback this change by directly overwriting the manifest on object storage. and report this failure to metasrv and let the entire repartition procedure to fail. And we can also optionally compute the new manifest for those staged version changes (like another repartition) and submit them to the target regions to make the also visible even if the repartition fails.
In the other hand, a successful RegionEdit request also acknowledges those staged version changes and make them visible.
After this step, the repartition is done in the data plane. We can start to process compaction and snapshot again.
Postprocessing  Copy Chapter
After the main processing is done, we can do some extra postprocessing to reduce the performance impact of repartition. Including reloading caches in frontend's route table, metasrv's kv cache and datanode's read/write/page cache etc.
We can also schedule an optional compaction to reorganize all the data file under the new partition rule to reduce potential fragmentation or read amplification.
Procedure  Copy Chapter
Here describe the repartition procedure step by step:
- <on frontend> Validating repartition request
- <on frontend> Initialize the repartition procedure
- Calculate rule diff and repartition plan group
- Allocate necessary new regions
- Lock the table key
- For each repartition subprocedure
  - Stop compaction and snapshot
  - Forbid new ingestion requests, update metadata, allow ingestion requests.
  - Update region rule to regions
  - Pick one region to calculate new manifest for all regions in this repartition group
  - Let that region to apply new manifest to each region via RegionEdit
  - If failed after some retries, revert this manifest change to other succeeded regions and mark this failure.
  - If all succeeded, acknowledge those staged version changes and make them visible.
  - Return result
- Collect results from subprocedure.
  - For those who failed, we need to restart those regions to force reconstruct their status from manifests
  - For those who succeeded, collect and merge their rule diff
- Unlock the table key
- Report the result to user.
- <in background> Reload cache
- <in background> Maybe trigger a special compaction
In addition of sequential step, rollback is also an important part of this procedure. There are three steps can be rolled back when unrecoverable failure occurs.
If the metadata update is not committed, we can overwrite the metadata to previous version. This step is scoped in the "no ingestion" period, so no new data will be ingested and the status of both datanode and metasrv will be consistent.
If the RegionEdit to other regions is not acknowledged, or partial acknowledged, we can directly overwrite the manifest on object storage from the central region (who computes the new manifest), and force region server to reload corresponding region to load its state from object storage to recover.
If the staged version changes are not acknowledged, we can re-compute manifest based on old rule for staged data, and apply them directly like above. This is like another smaller repartition for those staged data.
Region rule validation and diff calculation  Copy Chapter
In the current codebase, the rule checker is not complete. It can't check uniqueness and completeness of the rule. This RFC also propose a new way to validate the rule.
The proposed validation way is based on a check-point system, which first generates a group of check-points from the rule, and then check if all the point is covered and only covered by one rule.
All the partition rule expressionis limited to be the form of <column> <operator> <value>, and the operator is limited to be comparison operators. Those expressions are allowed to be nested with AND and OR operators. Based on this, we can first extract all the unique values on each column, adding and subtracting a little epsilon to cover its left and right boundary.
Since we accept integer, float and string as the value type, compute on them directly is not convenient. So we'll first normalize them to a common type and only need to preserve the relative partial ordering. This also avoids the problem of "what is next/previous value" of string and "what's a good precision" for float.
After normalization, we get a set of scatter points for each column. Then we can generate a set of check-points by combining all the scatter points like building a cartesian product. This might bring a large number of check-points, so we can do an prune optimization to remove some of them by merging some of the expression zones. Those expressions who have identical N-1 edge sub-expressions with one adjacent edge can be merged together. This prune check is with a time complexity of O(N  M  log(M)), where N is the number of active dimensions and M is the number of expression zones. Diff calculation is also done by finding different expression zones between the old and new rule set, and check if we can transform one to another by merging some of the expression zones.
The step to validate the check-points set against expressions can be treated as a tiny expression of PhysicalExpr. This evaluation will give a boolean matrix of K*M shape, where K is the number of check-points. We then check in each row of the matrix, if there is one and only one true value.
Compute and use new manifest  Copy Chapter
We can generate a new set of manifest file based on old manifest and two versions of rule. From abvoe rule processing part, we can tell how a new rule & region is from previous one. So a simple way to get the new manifest is also apply the step of change to manifest files. E.g., if region A is from region B and C, we simply combine all file IDs from B and C to generate the content of A.
If necessary, we can do this better by involving some metadata related to data, like min-max statistics of each file, and pre-evaluate over min-max to filter out unneeded files when generating new manifest.
The way to use new manifest needs one more extra step based on the current implementation. We'll need to record either in manifest or in file metadata, of what rule is used when generating (flush or compaction) a SST file. Then in every single read request, we need to append the current region rule as predicate to the read request, to ensure no data belong to other regions will be read. We can use the stored region rule to reduce the number of new predicates to apply, by removing the identical predicate between the current region rule and the stored region rule. So ideally in a table that has not been repartitioned recently, the overhead of checking region rule is minimal.
Pre-required tasks  Copy Chapter
In above steps, we assume some functionalities are implemented. Here list them with where they are used and how to implement them.
Cross-region read  Copy Section
The current data directory structure is {table_id}/{region_id}/[data/metadata]/{file_id}, every region can only access files under their own directory. After repartition, data file may be placed in other previous old regions. So we need to support cross-region read. This new access method allows region to access any file under the same table. Related tracking issue is <https://github.com/GreptimeTeam/greptimedb/issues/6409>.
Global GC worker  Copy Section
This is to simplify state management of data files. As one file may be referenced in multiple manifests, or no manifest at all. After this, every region and the repartition process only need to care about generateing and using new files, without tracking whether a file should be deleted or not. Leaving the deletion to the global GC worker. This worker basically works by counting reference from manifest file, and remove unused one. Related tracking issue is TBD.
Alternatives
In the "Data Processing" section, we can enlarge the "no ingestion" period to include almost all the steps. This can simplify the entire procedure by a lot, but will bring a longer time of ingestion pause which may not be acceptable.
---
Rfcs/2025 07 04 Compatibility Test Framework  Copy Section
---
Feature Name: Compatibility Test Framework
Tracking Issue: TBD
Date: 2025-07-04
Author: "Ruihang Xia <[email protected]>"
---
Summary
This RFC proposes a compatibility test framework for GreptimeDB to ensure backward/forward compatibility for different versions of GreptimeDB.
Motivation
In current practice, we don't have a systematic way to test and ensure the compatibility of different versions of GreptimeDB. Each time we release a new version, we need to manually test the compatibility with ad-hoc cases. This is not only time-consuming, but also prone to errors and unmaintainable. Highly rely on the release manager to ensure the compatibility of different versions of GreptimeDB.
We don't have a detailed guide on the release SoP of how to test and ensure the compatibility of the new version. And has broken the compatibility of the new version many times (v0.14.1 and v0.15.1 are two examples, which are both released right after the major release).
Details
This RFC proposes a compatibility test framework that is easy to maintain, extend and run. It can tell the compatibility between any given two versions of GreptimeDB, both backward and forward. It's based on the Sqlness library but used in a different way.
Generally speaking, the framework is composed of two parts:
1. Test cases: A set of test cases that are maintained dedicatedly for the compatibility test. Still in the .sql and .result format.
2. Test framework: A new sqlness runner that is used to run the test cases. With some new features that is not required by the integration sqlness test.
Test Cases  Copy Chapter
Structure  Copy Section
The case set is organized in three parts:
- 1.feature: Use a new feature
- 2.verify: Verify database behavior
- 3.cleanup: Paired with 1.feature, cleanup the test environment.
These three parts are organized in a tree structure, and should be run in sequence:

Example  Copy Section
For example, for a new feature like adding new index option (#6416), we (who implement the feature) create a new test case like this:

And

Since this new feature don't require some special way to verify the database behavior, we can reuse existing test cases in 2.verify/ to verify the database behavior. For example, we can reuse the verify-metadata test case to verify the metadata of the table.

In this example, we use some new sqlness features that will be introduced in the next section (since, IGNORE_RESULT, TEMPLATE).
Maintenance  Copy Section
Each time implement a new feature that should be covered by the compatibility test, we should create a new test case in 1.feature/ and 3.cleanup/ for them. And check if existing cases in 2.verify/ can be reused to verify the database behavior.
This simulates an enthusiastic user who uses all the new features at the first time. All the new Maintenance burden is on the feature implementer to write one more test case for the new feature, to "fixation" the behavior. And once there is a breaking change in the future, it can be detected by the compatibility test framework automatically.
Another topic is about deprecation. If a feature is deprecated, we should also mark it in the test case. Still use above example, assume we deprecate the index.granularity and index.false_positive_rate index options in v0.99.0, we can mark them as:

This tells the framework to ignore this feature in version v0.99.0 and later. Currently, we have so many experimental features that are scheduled to be broken in the future, this is a good way to mark them.
Test Framework  Copy Chapter
This section is about new sqlness features required by this framework.
Since and Till  Copy Section
Follows the ARG interceptor in sqlness, we can mark a feature is available between two given versions. Only the since is required:

IGNORE_RESULT  Copy Section
IGNORE_RESULT is a new interceptor, it tells the runner to ignore the result of the query, only check whether the query is executed successfully.
This is useful to reduce the Maintenance burden of the test cases, unlike the integration sqlness test, in most cases we don't care about the result of the query, only need to make sure the query is executed successfully.
TEMPLATE  Copy Section
TEMPLATE is another new interceptor, it can generate queries from a template based on a runtime data.
In above example, we need to run the SHOW CREATE TABLE query for all existing tables, so we can use the TEMPLATE interceptor to generate the query with a dynamic table list.
RUNNER  Copy Section
There are also some extra requirement for the runner itself:
- It should run the test cases in sequence, first 1.feature/, then 2.verify/, and finally 3.cleanup/.
- It should be able to fetch required version automatically to finish the test.
- It should handle the since and till properly.
On the 1.feature phase, the runner needs to identify all features need to be tested by version number. And then restart with a new version (the to version) to run 2.verify/ and 3.cleanup/ phase.
Test Report  Copy Chapter
Finally, we can run the compatibility test to verify the compatibility between any given two versions of GreptimeDB, for example:

We can also use a script to run the compatibility test for all the versions in a given range to give a quick report with all versions we need.
And we always bump the version in Cargo.toml to the next major release version, so the next major release version can be used as "latest" unpublished version for scenarios like local testing.
Alternatives
There was a previous attempt to implement a compatibility test framework that was disabled due to some reasons #3728.
---
Rfcs/2025 07 23 Global Gc Worker  Copy Section
---
Feature Name: "global-gc-worker"
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/6571
Date: 2025-07-23
Author: "discord9 <[email protected]>"
---
Global GC Worker
Summary  Copy Chapter
This RFC proposes the integration of a garbage collection (GC) mechanism within the Compaction process. This mechanism aims to manage and remove stale files that are no longer actively used by any system component, thereby reclaiming storage space.
Motivation  Copy Chapter
With the introduction of features such as table repartitioning, a substantial number of Parquet files can become obsolete. Furthermore, failures during manifest updates may result in orphaned files that are never referenced by the system. Therefore, a periodic garbage collection mechanism is essential to reclaim storage space by systematically removing these unused files.
Details  Copy Chapter
Overview  Copy Section
The garbage collection process will be integrated directly into the Compaction process. Upon the completion of a Compaction for a given region, the GC worker will be automatically triggered. Its primary function will be to identify and subsequently delete obsolete files that have persisted beyond their designated retention period. This integration ensures that garbage collection is performed in close conjunction with data lifecycle management, effectively leveraging the compaction process's inherent knowledge of file states.
This design prioritizes correctness and safety by explicitly linking GC execution to a well-defined operational boundary: the successful completion of a compaction cycle.
Terminology  Copy Section
- Unused File: Refers to a file present in the storage directory that has never been formally recorded in any manifest. A common scenario for this includes cases where a new SST file is successfully written to storage, but the subsequent update to the manifest fails, leaving the file unreferenced.
- Obsolete File: Denotes a file that was previously recorded in a manifest but has since been explicitly marked for removal. This typically occurs following operations such as data repartitioning or compaction.
GC Worker Process  Copy Section
The GC worker operates as an integral part of the Compaction process. Once a Compaction for a specific region is completed, the GC worker is automatically triggered. Executing this process on a datanode is preferred to eliminate the overhead associated with having to set object storage configurations in the metasrv.
The detailed process is as follows:
1.  Invocation: Upon the successful completion of a Compaction for a region, the GC worker is invoked.
2.  Manifest Reading: The worker reads the region's primary manifest to obtain a comprehensive list of all files marked as obsolete. Concurrently, it reads any temporary manifests generated by long-running queries to identify files that are currently in active use, thereby preventing their premature deletion.
3.  Lingering Time Check (Obsolete Files): For each identified obsolete file, the GC worker evaluates its "lingering time." Which is the time passed after it had been removed from manifest.
4.  Deletion Marking (Obsolete Files): Files that have exceeded their maximum configurable lingering time and are not referenced by any active temporary manifests are marked for deletion.
5.  Lingering Time (Unused Files): Unused files (those never recorded in any manifest) are also subject to a configurable maximum lingering time before they are eligible for deletion.
Following flowchart illustrates the GC worker's process:

#### Handling Obsolete Files
An obsolete file is permanently deleted only if two conditions are met:
1. The time elapsed since its removal from the manifest (its obsolescence timestamp) exceeds a configurable threshold.
2. It is not currently referenced by any active temporary manifests.

#### Handling Unused Files
With the integration of the GC worker into the Compaction process, the risk of accidentally deleting newly created SST files that have not yet been recorded in the manifest is significantly mitigated. Consequently, the concept of "Unused Files" as a distinct category primarily susceptible to accidental deletion is largely resolved. Any files that are genuinely "unused" (i.e., never referenced by any manifest, including temporary ones) can be safely deleted after a configurable maximum lingering time.
For debugging and auditing purposes, a comprehensive list of recently deleted files can be maintained.
Ensuring Read Consistency  Copy Section
To prevent the GC worker from inadvertently deleting files that are actively being utilized by long-running analytical queries, a robust protection mechanism is introduced. This mechanism relies on temporary manifests that are actively kept "alive" by the queries using them.
When a long-running query is detected (e.g., by a slow query recorder), it will write a temporary manifest to the region's manifest directory. This manifest lists all files required for the query. However, simply creating this file is not enough, as a query runner might crash, leaving the temporary manifest orphaned and preventing garbage collection indefinitely.
To address this, the following "heartbeat" mechanism is implemented:
1.  Periodic Updates: The process executing the long-running query is responsible for periodically updating the modification timestamp of its temporary manifest file (i.e., "touching" the file). This serves as a heartbeat, signaling that the query is still active.
2.  GC Worker Verification: When the GC worker runs, it scans for temporary manifests. For each one it finds, it checks the file's last modification time.
3.  Stale File Handling: If a temporary manifest's last modification time is older than a configurable threshold, the GC worker considers it stale (left over from a crashed or terminated query). The GC worker will then delete this stale temporary manifest. Files that were protected only by this stale manifest are no longer shielded from garbage collection.
This approach ensures that only files for genuinely active queries are protected. The lifecycle of the temporary manifest is managed dynamically: it is created when a long query starts, kept alive through periodic updates, and is either deleted by the query upon normal completion or automatically cleaned up by the GC worker if the query terminates unexpectedly.
This mechanism may be too complex to implement at once. We can consider a two-phased approach:
1.  Phase 1 (Simple Time-Based Deletion): Initially, implement a simpler GC strategy that deletes obsolete files based solely on a configurable lingering time. This provides a baseline for space reclamation without the complexity of temporary manifests.
2.  Phase 2 (Consistency-Aware GC): Based on the practical effectiveness and observed issues from Phase 1, we can then decide whether to implement the full temporary manifest and heartbeat mechanism to handle long-running queries. This iterative approach allows for a quicker initial implementation while gathering real-world data to justify the need for a more complex solution.
Drawbacks  Copy Chapter
- Dependency on Compaction Frequency: The integration of the GC worker with Compaction means that GC cycles are directly tied to the frequency of compactions. In environments with infrequent compaction operations, obsolete files may accumulate for extended periods before being reclaimed, potentially leading to increased storage consumption.
- Race Condition with Long-Running Queries: A potential race condition exists if a long-running query initiates but haven't write its temporary manifest in time, while a compaction process simultaneously begins and marks files used by that query as obsolete. This scenario could lead to the premature deletion of files still required by the active query. To mitigate this, the threshold time for writing a temporary manifest should be significantly shorter than the lingering time configured for obsolete files, ensuring that next GC worker runs do not delete files that are now referenced by a temporary manifest if the query is still running.
Also the read replica shouldn't be later in manifest version for more than the lingering time of obsolete files, otherwise it might ref to files that are already deleted by the GC worker.
- need to upload tmp manifest to object storage, which may introduce additional complexity and potential performance overhead. But since long-running queries are typically not frequent, the performance impact is expected to be minimal.
one potential race condition with region-migration is illustrated below:

which could cause gc worker to incorrectly mark the new file as orphan and delete it, if config the lingering time for orphan files(files not mentioned anywhere(in used or unused)) is not long enough.
A good enough solution could be to use lock to prevent gc worker to happen on the region if region migration is happening on the region, and vise versa.
The race condition between gc worker and repartition also needs to be considered carefully. For now, acquiring lock for both region-migration and repartition during gc worker process could be a simple solution.
Conclusion and Rationale  Copy Chapter
This section summarizes the key aspects and trade-offs of the proposed integrated GC worker, highlighting its advantages and potential challenges.
| Aspect | Current Proposal (Integrated GC) |
| :--- | :--- |
| Implementation Complexity | Medium. Requires careful integration with the compaction process and the slow query recorder for temporary manifest management. |
| Reliability | High. Integration with compaction and leveraging temporary manifests from long-running queries significantly mitigates the risk of incorrect deletion. Accurate management of lingering times for obsolete files and prevention of accidental deletion of newly created SSTs enhance data safety. |
| Performance Overhead | Low to Medium. The GC worker runs post-compaction, minimizing direct impact on write paths. Overhead from temporary manifest management by the slow query recorder is expected to be acceptable for long-running queries. |
| Impact on Other Components | Moderate. Requires modifications to the compaction process to trigger GC and the slow query recorder to manage temporary manifests. This introduces some coupling but enhances overall data safety. |
| Deletion Strategy | State- and Time-Based. Obsolete files are deleted based on a configurable lingering time, which is paused if the file is referenced by a temporary manifest. Unused files (never in a manifest) are also subject to a lingering time. |
Unresolved Questions and Future Work  Copy Chapter
This section outlines key areas requiring further discussion and defines potential avenues for future development.
*   Slow Query Recorder Implementation: Detailed specifications for modify slow query recorder's implementation and its precise interaction mechanisms with temporary manifests are needed.
*   Configurable Lingering Times: Establish and make configurable the specific lingering times for both obsolete and unused files to optimize storage reclamation and data availability.
Alternatives  Copy Chapter
1. Standalone GC Service  Copy Section
Instead of integrating the GC worker directly into the Compaction process, a standalone GC service could be implemented. This service would operate independently, periodically scanning the storage for obsolete and unused files based on manifest information and predefined retention policies.
Pros:
*   Decoupling: Separates GC logic from compaction, allowing independent scaling and deployment.
*   Flexibility: Can be configured to run at different frequencies and with different strategies than compaction.
Cons:
*   Increased Complexity: Requires a separate service to manage, monitor, and coordinate with other components.
*   Potential for Redundancy: May duplicate some file scanning logic already present in compaction.
*   Consistency Challenges: Ensuring read consistency would require more complex coordination mechanisms between the standalone GC service and active queries, potentially involving a distributed lock manager or a more sophisticated temporary manifest system.
This alternative could be implemented in the future if the integrated GC worker proves insufficient or if there is a need for more advanced GC strategies.
2. Manifest-Driven Deletion (No Lingering Time)  Copy Section
This alternative would involve immediate deletion of files once they are removed from the manifest, without a lingering time.
Pros:
*   Simplicity: Simplifies the GC logic by removing the need for lingering time management.
*   Immediate Space Reclamation: Storage space is reclaimed as soon as files are marked for deletion.
Cons:
*   Increased Risk of Data Loss: Higher risk of deleting files still in use by long-running queries or other processes if not perfectly synchronized.
*   Complex Read Consistency: Requires extremely robust and immediate mechanisms to ensure that no active queries are referencing files marked for deletion, potentially leading to performance bottlenecks or complex error handling.
*   Debugging Challenges: Difficult to debug issues related to premature file deletion due to the immediate nature of the operation.
---
Rfcs/2025 08 16 Async Index Build  Copy Section
---
Feature Name: Async Index Build
Tracking Issue: https://github.com/GreptimeTeam/greptimedb/issues/6756
Date: 2025-08-16
Author: "SNC123 <[email protected]>"
---
Summary

This RFC proposes an asynchronous index build mechanism in the database, with a configuration option to choose between synchronous and asynchronous modes, aiming to improve flexibility and adapt to different workload requirements.
Motivation

Currently, index creation is performed synchronously, which may lead to prolonged write suspension and impact business continuity. As data volume grows, the time required for index building increases significantly. An asynchronous solution is urgently needed to enhance user experience and system throughput.
Details
Overview  Copy Chapter
The following table highlights the difference between async and sync index approach:
| Approach | Trigger | Data Source | Additional Index Metadata Installation | Fine-grained FileMeta Index |
| :--- | :--- | :--- | :--- | :--- |
| Sync Index | On write_sst | Memory (on flush) / Disk (on compact) | Not required(already installed synchronously) | Not required |
| Async Index | 4 trigger types | Disk | Required | Required |
The index build mode (synchronous or asynchronous) can be selected via configuration file. 
Four Trigger Types  Copy Section
This RFC introduces four IndexBuildTypes to trigger index building:
- Manual Rebuild: Triggered by the user via ADMIN build_index("table_name"), for scenarios like recovering from failed builds or migrating data. SST files whose ColumnIndexMetadata (see below) is already consistent with the RegionMetadata will be skipped.
- Schema Change: Automatically triggered when the schema of an indexed column is altered.
- Flush: Automatically builds indexes for new SST files created by a flush.
- Compact: Automatically builds indexes for new SST files created by a compaction.
Additional Index Metadata Installation  Copy Section
Previously, index information in the in-memory FileMeta was updated synchronously. The async approach requires an explicit installation step.
A race condition can occur when compaction and index building run concurrently, leading to:
1. Building an index for a file that is about to be deleted by compaction.
2. Creating an unnecessary index file and an incorrect manifest record.
3. On restart, replaying the manifest could load metadata for a non-existent file.
To prevent this, the system checks if a file's FileMeta is in a compacting state before updating the manifest. If it is, the installation is aborted.
Fine-grained FileMeta Index  Index')" title="Copy section prompt for LLMs"> Copy Section
The original FileMeta only stored file-level index information. However, manual rebuilds require column-level details to identify files inconsistent with the current DDL. Therefore, the indexes field in FileMeta is updated as follows:

Process  Copy Chapter
The index building process is similar to a flush and is illustrated below: