{"owner":"apache","repo":"doris","hasSkills":true,"totalSkillsCount":17,"totalTokensCount":22261,"categories":["plugin-manifest","anthropic-skill","root-instruction"],"hasMcp":false,"mcpConfig":null,"found":["tools/emr_storage_regression/README.md","tools/coffeebench-tools/README.md",".claude/skills/code-review/SKILL.md","tools/cost_model_evaluate/README.MD","tools/lzo/README.md","tools/show_segment_status/README.md","tools/tpcds-tools/README.md","AGENTS.md","tools/pipeline-tracing/README.md","tools/ssb-tools/README.md","tools/row_to_column/README.md","tools/clickbench-tools/README.md","tools/tpch-tools/README.md","be/src/io/tools/readme.md","fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md","fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md","extension/DataX/doriswriter/src/main/resources/plugin.json"],"skills":{"tools/emr_storage_regression/README.md":"\n\n# Data Lake Regression Testing Tool For External Table\n\nUsed to test the doris external table on object storage for cloud vendors\n\n> Supported storage formats: HDFS, Alibaba Cloud OSS, Tencent Cloud COS, Huawei Cloud OBS\n\n> Supported data lake table formats: Iceberg\n\nThe following provides the example of the command line options:\n\n```\nsh tools/emr_storage_regression/emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\n```\nsh tools/emr_storage_regression/emr_tools.sh --case CASE --endpoint ENDPOINT --region REGION --service SERVICE --ak AK --sk SK --host HOST --user USER --port PORT\n```\n\nThe usage of each option is described below.\n\n## Connectivity Test\n\nWhen the `--case` option is set to `ping`, will check Doris's connectivity on EMR:\n\n- `--endpoint`, Object Storage Endpoint.\n\n- `--region`, Object Storage Region.\n\n- `--ak`, Object Storage Access Key.\n\n- `--sk`, Object Storage Secret Key.\n\n- `--host`, Doris Mysql Client IP.\n\n- `--user`, Doris Mysql Client Username.\n\n- `--port`, Doris Mysql Client Port.\n\n- `--service`, EMR cloud vendors: ali(Alibaba), hw(Huawei), tx(tencent).\n\n### Environment Variables\n\nNeed modify the environment variable in `default_emr_env.sh`, the script will execute `source default_emr_env.sh` to make the environment variable take effect.\n\nIf environment variables are configured, you can run the test script directly with the following command:\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\n### The Script Execution Steps For Connectivity Test\n\n1. Create Spark and Hive tables on EMR\n2. Use Spark and Hive command lines to insert sample data\n3. Doris creates the Catalog for connectivity test \n4. Execute SQL for connectivity test: `ping.sql`\n\n### Alibaba Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service` to `ali`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint oss-cn-beijing-internal.aliyuncs.com --region cn-beijing --service ali --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log\n```\n\nAlibaba Cloud EMR also supports testing connectivity for both Doris with DLF metadata and Doris on OSS-HDFS storage.\n\n- The DLF metadata connectivity test needs to be performed on the EMR cluster where the DLF serves as the metadata store, Default value of `DLF_ENDPOINT` is `datalake-vpc.cn-beijing.aliyuncs.com`, configured at ping_test/ping_poc.sh.\n\n- To test the OSS-HDFS storage connectivity, need to [enable the HDFS service on the OSS storage and configure](https://www.alibabacloud.com/help/en/e-mapreduce/latest/oss-hdfsnew), Default value of `JINDO_ENDPOINT` is `cn-beijing.oss-dls.aliyuncs.com`, configured at ping_test/ping_poc.sh.\n\n### Tencent Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service` to `tx`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint cos.ap-beijing.myqcloud.com --region ap-beijing --service tx --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log\n```\n\n### Huawei Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service`to `hw`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint obs.cn-north-4.myhuaweicloud.com --region cn-north-4 --service hw --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log \n```\n\n## Performance Testing on Standard Test Set\n\nWhen the `--case` option is set to `data_set`, will test the query performance of Doris external table:\n\n- `--test` test data set: ssb, ssb_flat, tpch, clickbench and all. Default `all`.\n\n- `--service`, EMR cloud vendors: ali(Alibaba), hw(Huawei), tx(tencent).\n\n- `--host`, Doris Mysql Client IP.\n\n- `--user`, Doris Mysql Client Username.\n\n- `--port`, Doris Mysql Client Port.\n\n### Environment Variables\n\nJust modify the above environment variable in `default_emr_env.sh`, the script will execute `source default_emr_env.sh` to make the environment variable take effect.\n\nIf environment variables are configured, you can run the test script directly with the following command:\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\n### Prepare Data\n\n1. To run the standard test set using the `emr_tools.sh` script, you need to rewrite the object storage bucket specified by the `BUCKET` variable, and then prepare data in advance and put them under the bucket. The script will generate table creation statements based on the bucket.\n\n2. Now the `emr_tools.sh` script supports iceberg, parquet and orc data for ssb, ssb_flat, tpch, clickbench.\n\n### Execution Steps\n\n1. After the connectivity test, the Doris Catalog corresponding to the standard test set is created\n2. Prepare the test set data based on the object storage bucket specified by the `BUCKET` variable\n3. Generate Spark table creation statements and create Spark object storage tables on EMR\n4. Create the spark table in the local HDFS directory: `hdfs:///benchmark-hdfs`\n5. You can choose to analyze Doris tables ahead of time and manually execute the statements in `analyze.sql` in the Doris Catalog\n6. Execute standard test set scripts: `run_standard_set.sh`\n\n### Standard data set: ssb, ssb_flat, tpch, clickbench\n\n- Full test. After executing the test command, Doris will run ssb, ssb_flat, tpch, clickbench tests in sequence, and the test results will include the cases on HDFS and on the object storage specified by `--service`.\n\n```\nsh emr_tools.sh --case data_set --service ali --host 127.0.0.1 --user root --port 9030 > log\n```\n\n- Specify a single test. `--test` option can be set to one of ssb, ssb_flat, tpch and clickbench.\n\n```\nsh emr_tools.sh --case data_set --test ssb --service ali --host 127.0.0.1 --user root --port 9030 > log\n```\n","tools/coffeebench-tools/README.md":"\n\n## Usage\n\nThese scripts are used to make coffee-bench test.\nfollow the steps below:\n\n### 1. create tables. modify `conf/doris-cluster.conf` to specify doris info, then run script below.\n\n ./bin/create-tables.sh\n\n### 2. load coffee-bench data. use -s for scale. \"500m/1b/5b\"\n\n ./bin/load-data.sh -s 500m\n\n **Note: The data loading process uses S3 data located in the US East (us-east-1) region.**\n If you cannot access this region due to network restrictions or other reasons, you can:\n 1. Use AWS CLI to copy the data to an S3 bucket in your preferred region\n 2. Download the data locally and use alternative methods to import it into Doris\n\n### 3. run queries.\n\n ./bin/run-queries.sh\n\n NOTICE: At present, we rewrite some queries in coffee bench to adapt to Doris' execution framework, but it does not affect the correctness of the results.\n",".claude/skills/code-review/SKILL.md":"---\nname: code-review\ndescription: Create consistent releases and changelogs\ncompatibility: opencode\n---\n\n## What I do\n\nComplete code review work to ensure code quality. Due to the length of content, you can read sections as needed based on the actual modifications being reviewed.\n\n## When to use me\n\nUse this when you need to review code, whether it's code you just completed or directly reviewing target code.\n\n---\n\n## Part 1: General Principles\n\n### 1.1 Core Invariants\n\nAlways focus on the following core invariants during review:\n1. **Data Correctness**: Data from any committed transaction must be visible to subsequent queries and not lost; data from uncommitted transactions must not be visible\n2. **Version Consistency**: Partition visible version is the sole standard for data visibility; BE must strictly read data not exceeding this version\n3. **Delete Bitmap Consistency** (MoW tables): delete bitmap must be strictly aligned with rowset versions; sentinel mark / TEMP_VERSION must be replaced with actual versions before use\n4. **Memory Safety**: All significant memory allocations must be tracked by MemTracker; orphan memory is not allowed\n5. **Error Means Failure**: Invariant violations should trigger exceptions or non-OK Status; silent continuation is never allowed\n\n### 1.2 Review Principles\n\n- **Defensive Programming Prohibited**: Do not use `if(valid)` style defensive checks to mask logic errors. If logically A=true should imply B=true, write `if(A) { DORIS_CHECK(B); ... }`, never `if(A && B)`\n- **Follow Context Conventions**: When adding code, strictly follow patterns in adjacent code—same error handling, same interface call order, same lock acquisition patterns, unless a clearly correct and more reasonable approach is identified.\n- **Prioritize Reuse**: Before adding new functionality, search for similar implementations that can be reused or extended; after adding, ensure good abstraction for shared code.\n- **Code First**: When this SKILL conflicts with actual code behavior, defer to your understanding of actual code behavior and explain\n- **Performance First**: All obviously redundant operations should be optimized away, all obvious performance optimizations must be applied, and obvious anti-patterns must be eliminated.\n\n### 1.3 Critical Checkpoints (Self-Review and Review Priority)\n\nThe following checkpoints must be **individually confirmed with conclusions** during self-review and review:\n\n- What is the goal of the current task? Does the current code accomplish this goal? Is there a test that proves it?\n- Is this modification as small, clear, and focused as possible?\n- Does the code segment involve concurrency? If yes:\n - Which threads introduce concurrency, what are the critical variables? What locks protect them?\n - Are operations within locks as lightweight as possible? Are all heavy operations outside locks while maintaining concurrency safety?\n - If multiple locks exist, is the locking order consistent? Is there deadlock risk?\n- Is there special or non-intuitive lifecycle management? If yes:\n - Understand the complete lifecycle and relationships of related variables. Are there circular references? When is each released? Can all lifecycles end normally?\n- Are configuration items added?\n - Should the configuration allow dynamic changes, and does it actually allow them? If yes:\n - Can affected processes detect the change promptly without restart?\n- Does it involve incompatible changes like function symbols or storage formats? If yes:\n - Is compatibility code added? Can it correctly handle requests during rolling upgrades?\n- Are there functionally parallel code paths to the modified one? If yes:\n - Should this modification be applied to other paths, and has it been?\n- Are there special conditional checks? If yes:\n - Does the condition have clear comments explaining why this check is necessary, simplest, and most viable?\n - Are there similar paths with similar issues?\n- What is the test coverage?\n - Are end-to-end functional tests comprehensive?\n - Are there comprehensive negative test cases from various angles?\n - For complex features, are key functions sufficiently modular with unit tests?\n- Does the feature need increased observability? If yes:\n - When bugs occur, are existing logs and VLOGs sufficient to investigate key issues?\n - Are INFO logs lightweight enough?\n - Are Metrics added for critical paths? Referencing similar features, are all necessary observability values covered?\n- Does it involve transaction and persistence-related modifications? If yes:\n - Are all EditLog reads and writes correctly covered? Is all information requiring persistent storage persisted? Does it follow our standard approach?\n - In transaction processing logic, can master failover at any time point be handled correctly?\n- Does it involve data writes and modifications? If yes:\n - Are transactionality and atomicity guaranteed? Are there issues with concurrency?\n - Would FE or BE crashes cause leaks, deadlocks, or incorrect data?\n- Are new variables added that need to be passed between FE and BE? If yes:\n - Is variable passing modified in all paths? For example, various scattered thrift sending points like constant folding, point queries.\n- Analyze deeply from all angles (including but not limited to time complexity, anti-patterns, CPU and memory friendliness, redundant calculations, etc.) based on the code context and available information. Are there any performance issues or optimization opportunities?\n- Based on your understanding, are there any other issues with the current modification?\n\nAfter checking all the above items with code. Use the remaining parts of this skill as needed. The following content does not need individual responses; just check during the review process.\n\n#### 1.3.1 Concurrency and Thread Safety (Highest Priority)\n\n- [ ] **New thread/bthread entry**: Is `SCOPED_ATTACH_TASK` used? Omission causes memory to be counted in orphan tracker, debug builds crash directly\n- [ ] **Lock protection scope**: Is shared data (version map, delete bitmap, transaction state) accessed within correct locks? Is lock type correct (shared vs exclusive)?\n- [ ] **Lock order**: Are there nested acquisitions of multiple locks? Must follow known lock order (e.g., `_txn_lock` -> `_txn_map_lock`), otherwise deadlock\n- [ ] **Atomic operation memory order**: Use `relaxed` for statistics, at least `acquire/release` for flag/signal\n- [ ] **COW column modification**: Is exclusive ownership ensured before `mutate()`? Is `assume_mutable_ref()` only used when exclusive ownership is confirmed?\n\n#### 1.3.2 Error Handling (Highest Priority)\n\n- [ ] **Status checking**: `Status` return values **must** be checked, `static_cast(...)` silent discard is prohibited\n- [ ] **Exception propagation chain**: Is there `RETURN_IF_CATCH_EXCEPTION` above `THROW_IF_ERROR` to catch?\n- [ ] **Error messages**: Are they clear, include context (table/partition/TabletID/transaction ID), provide solution direction?\n- [ ] **DORIS_CHECK usage**: Only for invariant assertions (e.g., `DORIS_CHECK(hash_table != nullptr)`), not for defensive programming\n\n#### 1.3.3 Memory Safety (High Priority)\n\n- [ ] **Large memory reservation**: Is `try_reserve()` called before allocation? Is there `DEFER_RELEASE_RESERVED()` to ensure release?\n- [ ] **Cache Handle**: Is `cache->release(handle)` called after using `Cache::Handle*`?\n- [ ] **shared_ptr circular references**: Do scenarios like `Dependency` use raw pointers or `weak_ptr` to break cycles?\n- [ ] **Object creation**: Do classes with `ENABLE_FACTORY_CREATOR` use `create_shared()` / `create_unique()`? Raw `new` is prohibited\n\n#### 1.3.4 Data Correctness (High Priority)\n\n- [ ] **Delete Bitmap version**: In Cloud mode, is `TEMP_VERSION_COMMON` manually replaced before use?\n- [ ] **Version consistency**: Does data reading strictly not exceed Partition visible version?\n- [ ] **MoW setting**: Is `SegmentWriterOptions::enable_unique_key_merge_on_write` correctly set to `true`?\n- [ ] **EditLog persistence**: Do metadata modifications call `editLog.logXxx()`? Is replay logic strictly equivalent to master?\n\n#### 1.3.5 Observability (Medium Priority)\n\n- [ ] **Metrics addition**: Are `ADD_TIMER/ADD_COUNTER` added to critical paths? Does naming follow `module_xxx` convention?\n- [ ] **Log levels**: Use `LOG(ERROR)` for error logs, `LOG(INFO)` for critical paths, `VLOG` or `DVLOG` for debug info\n- [ ] **Trace information**: Do distributed operations (RPC, transactions) include trace ID for tracking?\n\n### 1.4 Test Coverage Principles\n\n- All kernel features **must** have corresponding tests\n- End-to-end SQL functionality should prioritize adding regression tests under `regression-test/` (Groovy DSL)\n- Also add BE unit tests (`be/test/`, Google Test) and FE unit tests (`fe/fe-core/src/test/`, JUnit 5) where possible. For complex features, must be modular with unit tests.\n- Regression test result files (`.out`) **must not be handwritten**, must be auto-generated via `-genOut`\n- Regression tests must use `order_qt_` prefix or manual `ORDER BY` to ensure ordered results\n- Expected error cases use `test { sql \"...\"; exception \"...\" }` pattern\n- Tables should be **DROPped before use** (not after tests end), to preserve environment for debugging\n- Simple tests should hardcode table names directly, not use `def tableName`\n\n---\n\n## Part 2: BE Module Review Guide\n\n### 2.1 Error Handling (Highest Priority)\n\n#### 2.1.1 Status / Exception / Result\\ Relationship\n\n| Type | Usage | Propagation Macro |\n|------|-------|------------------|\n| `Status` | Default error return type, `[[nodiscard]]` | `RETURN_IF_ERROR` |\n| `Exception` | Vectorized execution engine hot path, expression evaluation | `RETURN_IF_CATCH_EXCEPTION` |\n| `Result` | Return value or error (replaces old output parameter pattern) | `DORIS_TRY` |\n\n**Review Checkpoints:**\n\n- [ ] Are `Status` return values checked? Search for `static_cast(...)` pattern—this is the **most dangerous anti-pattern**, silently swallowing errors without logging. Currently 171+ occurrences in codebase. Never allow this pattern in new code\n- [ ] Where `THROW_IF_ERROR` is used, is there `RETURN_IF_CATCH_EXCEPTION` or `RETURN_IF_ERROR_OR_CATCH_EXCEPTION` above in the call chain to catch? If not, exceptions propagate uncaught causing crashes. Current version has related catch in upper layers when Pipeline executes specific operators.\n- [ ] **`THROW_IF_ERROR` prohibited in Defer/RAII destructor lambdas**: Throwing exceptions during stack unwinding triggers `std::terminate`. Known issues: `pipeline_task.cpp:412-413, 638-639`. Use `WARN_IF_ERROR` or `static_cast` + logging in Defer\n- [ ] **`THROW_IF_ERROR` prohibited in Thrift RPC handlers without try/catch protection**: Thrift framework doesn't catch Doris exceptions, causing connection drops. Known issues: `backend_service.cpp:1323,1326`. RPC handlers should use `RETURN_IF_ERROR` pattern or outermost try/catch\n- [ ] Is `WARN_IF_ERROR` usage reasonable? Only allowed in destructors, cleanup paths, best-effort operations. Warning message must not be empty string\n- [ ] Is `DORIS_CHECK` used for **invariant assertions** (not defensive programming)? Correct usage example: `DORIS_CHECK(hash_table)` asserts hash table is initialized\n- [ ] Do new catch blocks correctly convert exceptions? Should follow standard pattern from `internal_service.cpp:348-357`: catch `doris::Exception` first, then `std::exception`, finally `...`\n\n#### 2.1.2 Cloud RPC Error Handling\n\nThe `retry_rpc` template in `cloud_meta_mgr.cpp` is the standard pattern for RPC error handling in Cloud mode:\n- [ ] Do new Cloud RPC calls use `retry_rpc`?\n- [ ] Does `INVALID_ARGUMENT` return directly without retry? (Correct behavior)\n- [ ] Does `KV_TXN_CONFLICT` use an independent conflict retry counter?\n- [ ] Do timeout and retry counts use config variables instead of hardcoding?\n\n#### 2.1.3 compile_check Mechanism\n\n`compile_check_begin.h` / `compile_check_end.h` elevate `-Wconversion` to **compile error** (excluding sign-conversion and float-conversion).\n\n- [ ] Do new `.h` files include paired `compile_check_begin.h` / `compile_check_end.h` in declaration regions?\n- [ ] **Coverage status**: Currently only 31.1% (1,049/3,372) of source files use compile_check. Additionally, 218 files contain `compile_check_begin.h` but lack corresponding `compile_check_end.h` (unpaired). New files should always include complete pairing\n- [ ] Are there implicit narrowing conversions? Pay special attention to `int64_t -> int32_t`, `size_t -> int`, `uint64_t -> int64_t`\n- [ ] If bypassing due to third-party code is necessary, are `compile_check_avoid_begin.h` / `compile_check_avoid_end.h` used?\n\n### 2.2 Memory Management\n\n#### 2.2.1 MemTracker Hierarchy\n\nDoris uses two-level memory tracking:\n- **`MemTrackerLimiter`**: Heavy-weight tracker with limits (QUERY / LOAD / COMPACTION / SCHEMA_CHANGE / METADATA / CACHE types)\n- **`MemTracker`**: Lightweight tracker without limits, for fine-grained sub-component tracking\n\nThread-level tracking is implemented via `ThreadMemTrackerMgr`, each thread has one active `MemTrackerLimiter` and a `MemTracker` stack.\n\n**Review Checkpoints:**\n\n- [ ] Do newly started threads/bthreads use `SCOPED_ATTACH_TASK` at entry? Without it, memory is counted in orphan tracker, debug builds trigger assertions\n- [ ] Does temporary tracker switching use `SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER` (supports nesting)?\n- [ ] **`ThreadPool::submit_func` callbacks**: Do lambdas submitted to thread pools use `SCOPED_ATTACH_TASK` internally? Currently 87+ `submit_func` calls in codebase, many rely on \"indirect attach\" (attach happens deep in call chain), which is fragile. Correct approach references gold-standard wrapper pattern from `calc_delete_bitmap_executor.h:67-83`: explicit attach at callback entry. Known problematic files: `s3_file_bufferpool.cpp:114`, `wal_manager.cpp:414`, `download_action.cpp:166`\n- [ ] Is `try_reserve()` called before large memory allocation? Is there `DEFER_RELEASE_RESERVED()` to ensure release after allocation?\n- [ ] Does memory hook path (`ThreadMemTrackerMgr::consume`) avoid operations that may trigger memory allocation (e.g., LOG, string formatting)? `_stop_consume` flag exists to prevent recursion\n\n#### 2.2.2 Object Lifecycle\n\n- [ ] Are classes using `ENABLE_FACTORY_CREATOR` created via `create_shared()` / `create_unique()`? Raw `new` is prohibited\n- [ ] Are there `shared_ptr` circular references? Standard ways to break cycles:\n - `Dependency::_shared_state` uses **raw pointer** (non-owning)\n - `Dependency::_blocked_task` uses `vector>`\n - `TrackerLimiterGroup::trackers` uses `list>`\n- [ ] Is Rowset's dual reference counting used correctly? `acquire()` / `release()` is manual reference counting (for performance), `shared_ptr` manages ownership. When `release()` decrements to 0 and in UNLOADING state, triggers `do_close()`\n- [ ] Are rowsets in `_unused_rowsets` judged deletable via `use_count() == 1`?\n- [ ] Are there suspicious `shared_ptr` circular reference issues? For clear lifecycles, prefer `unique_ptr` + raw pointer observer pattern.\n\n#### 2.2.3 COW Column Semantics\n\nVectorized columns (`IColumn`) use custom intrusive reference-counted Copy-on-Write mechanism:\n\n- [ ] Is exclusive ownership ensured when calling `mutate()`? When `use_count() > 1`, triggers deep copy, which is a performance hotspot\n- [ ] Is `assume_mutable_ref()` only used when exclusive ownership is confirmed? This method **does not check reference count**, using it in shared state causes data corruption\n- [ ] Is `set_columns()` called to restore after `Block::mutate_columns()`? After calling, original Block's column pointers are null\n- [ ] Can the pointer returned by `convert_to_full_column_if_const()` share with original column? For `ColumnConst` it materializes a new column, but for normal columns returns shared pointer\n\n#### 2.2.4 Cache Lifecycle\n\n- [ ] Is `cache->release(handle)` called after using `Cache::Handle*`? Omission causes memory leaks\n- [ ] Do cache values inherit from `LRUCacheValueBase`? This base class automatically releases tracking bytes on destruction\n- [ ] Are new cache types registered with `CacheManager`? Unregistered caches don't participate in global GC\n\n### 2.3 Concurrency and Locks\n\n#### 2.3.1 Tablet Lock Hierarchy\n\nThe `Tablet` class has 8+ independent locks, with confirmed usage rules:\n\n| Lock | Type | Rule |\n|----|------|------|\n| `_meta_lock` | `shared_mutex` | Modifying version map requires EXCLUSIVE; reading requires SHARED |\n| `_rowset_update_lock` | `mutex` | Serializes delete bitmap updates during MoW table publish_txn, **independent of _meta_lock** (avoids blocking read path) |\n| `_base_compaction_lock` | `mutex` | Serializes base compaction |\n| `_cumulative_compaction_lock` | `mutex` | Serializes cumulative compaction |\n| `_migration_lock` | `shared_timed_mutex` | Protects tablet migration |\n| `_cooldown_conf_lock` | `shared_mutex` | Protects cooldown configuration |\n\n**Review Checkpoints:**\n\n- [ ] Do operations on `_rs_version_map` or `_stale_rs_version_map` hold `_meta_lock` (shared or exclusive)?\n- [ ] Do `TabletMeta` modification operations hold the owning Tablet's EXCLUSIVE `_meta_lock`?\n- [ ] Is there acquisition of other tablet locks while holding `_meta_lock`? This may cause deadlock\n\n#### 2.3.2 TxnManager Lock Order\n\nConfirmed consistent lock order: `_txn_lock` -> `_txn_map_lock` (applies to commit, publish, delete, rollback).\n- [ ] Do new TxnManager operations follow this lock order?\n\n#### 2.3.3 Pipeline Dependency Concurrency\n\n`Dependency` uses double-checked locking + atomic `_ready` flag + mutex `_task_lock`:\n- `set_ready()`: First check `_ready` (fast path), then lock to set `_ready = true` and wake up\n- `is_blocked_by()`: Lock, check `_ready`, add to wait list if not ready\n\n- [ ] Do new `Dependency` subclasses correctly call `block()` / `set_ready()`?\n- [ ] Do `CountedFinishDependency`'s `add()` / `sub()` operate under `_mtx` protection?\n\n#### 2.3.4 Atomic Operation Memory Order\n\nMemory order usage patterns in codebase:\n- `memory_order_relaxed`: Statistics counters (correct)\n- `memory_order_acquire/release`: Lifecycle flags like `ExecEnv::_s_ready` (correct)\n- `memory_order_acq_rel`: CAS close operations (correct)\n\n- [ ] Do new atomic operations use appropriate memory order? Use relaxed for statistics counters, at least acquire/release for flag/signal\n- [ ] Does `std::atomic` as stop signal use at least acquire/release? Relaxed may theoretically delay propagation\n\n### 2.4 Type System and Serialization\n\n#### 2.4.1 Upgrade/Downgrade Compatibility\n\nUpgrade/downgrade compatibility no longer uses be_exec_version for control. Instead, each feature (e.g., functions, compression algorithms, etc.) adds its own option for confirmation. BE uses is_set&&is_true to branch or replace functionality at appropriate nodes.\n\n#### 2.4.2 Decimal Precision Overflow\n\n- [ ] Do Decimal operation results check precision upper limit? `decimal_result_type()` may promote result to DECIMAL256, causing unexpected memory layout changes\n- [ ] Are DecimalV2's `original_precision` / `original_scale` correctly set? Default value `UINT32_MAX` means \"unknown\", `get_format_scale()` returns different value from `scale` in this case\n\n#### 2.4.3 Thrift Type Mapping\n\n- [ ] Are `TScalarType` optional fields (`precision`, `scale`, `len`) set when needed? DECIMAL needs precision/scale, CHAR/VARCHAR needs len, DATETIMEV2 needs scale\n- [ ] Is depth-first flattened representation of `TTypeDesc.types` correctly traversed? Off-by-one errors are common with nested complex types (ARRAY>)\n\n#### 2.4.4 Block Merge Nullable Handling\n\n`MutableBlock::merge_impl()` assumes target is Nullable and source is non-Nullable when types mismatch, using C-style cast `(DataTypeNullable*)` instead of `dynamic_cast`.\n\n- [ ] Does new Block merge logic correctly handle Nullable promotion? DCHECK only protects in debug builds\n\n### 2.5 Pipeline Execution Engine\n\n#### 2.5.1 Operator Lifecycle\n\nPipeline Task state machine: `INITED -> RUNNABLE -> BLOCKED -> FINISHED -> FINALIZED`\n\n- [ ] Are `SharedState`'s source_deps and sink_deps correctly connected? `inject_shared_state()` is responsible for connection\n\n#### 2.5.2 Memory Reservation and Spill\n\n- [ ] Do memory-intensive operations (Hash Join build, aggregation) use `_try_to_reserve_memory()`?\n- [ ] Is there `_memory_sufficient_dependency` for backpressure when memory is insufficient?\n- [ ] Does `revoke_memory()` correctly implement spill logic?\n\n### 2.6 Storage Engine\n\n#### 2.6.1 Rowset Version Management\n\n- [ ] Are `add_rowset()` and `modify_rowsets()` executed under exclusive `_meta_lock`?\n- [ ] Are version ranges continuous? There should be no version gaps in `_rs_version_map`\n- [ ] Does compaction output rowset's version range correctly cover input rowsets?\n\n#### 2.6.2 Delete Bitmap (MoW Tables, Extremely High Priority)\n\n- [ ] In Cloud mode, is **TEMP_VERSION_COMMON manually replaced** for bitmaps obtained from `CloudTxnDeleteBitmapCache`? Code comments explicitly warn about this issue (`cloud_txn_delete_bitmap_cache.h:72-75`). Omission causes bitmap to be applied to wrong version\n- [ ] Is `_rowset_update_lock` held during delete bitmap calculation? Prevents concurrent calculation on same (txn_id, tablet_id)\n- [ ] Does compaction's delete bitmap calculation use **latest** compaction stats (`ms_base_compaction_cnt`, etc.)? Stale stats will miss recently compacted rowsets\n\n#### 2.6.3 Segment Writing\n\n- [ ] Is `SegmentWriterOptions::enable_unique_key_merge_on_write` correctly set for MoW tables? Default value is `false`, omission causes deduplication to be silently skipped\n- [ ] Is performance impact of row-stored columns (`_serialize_block_to_row_column`) on wide tables within expectations?\n\n### 2.7 IO Subsystem\n\n- [ ] Do `FileSystem` operations use `FILESYSTEM_M` macro to handle bthread async IO?\n- [ ] Do remote storage (S3/HDFS) reads use `create_cached_file_reader()` for file caching?\n\n### 2.8 Inverted Index Subsystem\n\n#### 2.8.1 Lifecycle and Destruction Order\n\n- [ ] **Field declaration order is destruction order**: In `inverted_index_writer.h:92-95`, `_dir` must be declared **before** `_index_writer`, because `_index_writer`'s `write.lock` is created by `_dir.lockFactory`, C++ destructs in reverse declaration order. New fields must follow this constraint\n- [ ] **Cache handle destruction extends LRU retention**: `inverted_index_cache.h:142-152`, `InvertedIndexCacheHandle` destructor sets `last_visit_time` to future timestamp (`UnixMillis() + config::index_cache_entry_stay_time_after_lookup_s * 1000`), making actually queried searchers survive longer in LRU. Be aware of this extension mechanism when modifying cache logic\n\n#### 2.8.2 CLucene Exception Handling\n\nCLucene doesn't use C++ exceptions, `inverted_index_common.h:65-111` implements `ErrorContext` + FINALLY macro system:\n- `FINALLY`: Catches exceptions and returns `Status` error\n- `FINALLY_EXCEPTION`: Catches and rethrows\n- All three macros use `static_assert` to enforce existence of `error_context` local variable in scope\n\n- [ ] Does new CLucene-related code use `ErrorContext` + `FINALLY` pattern instead of raw try/catch?\n\n#### 2.8.3 Three-Valued Logic Bitmap\n\n`InvertedIndexResultBitmap` (`inverted_index_reader.h:80-214`) implements SQL three-valued logic (TRUE/FALSE/NULL), storing separately via `_data_bitmap` + `_null_bitmap`:\n- [ ] **`op_not` is const but actually modifies data**: Modifies pointed object via `shared_ptr` to bypass const qualifier. Callers should not assume const reference returned results are immutable\n- [ ] `operator|=` implements complete SQL three-valued OR semantics (lines 148-157), same for `operator&=` and `operator-=`. New bitmap operations must follow three-valued logic\n\n### 2.9 Schema Change Subsystem\n\n#### 2.9.1 Three-Strategy Selection and Lock Order\n\n`_get_sc_procedure()` (`schema_change.h:303-315`) selects among three strategies based on change type:\n- `LinkedSchemaChange`: Column name/type alias changes only, zero-copy\n- `VSchemaChangeDirectly`: Direct conversion without sorting\n- `VLocalSchemaChangeWithSorting`: Full rewrite with sorting\n\n**Lock acquisition order** (`schema_change.cpp:972-976`): `base_tablet push_lock → new_tablet push_lock → base_tablet header_lock → new_tablet header_lock`\n\n- [ ] Do schema change lock acquisitions follow this four-level order? Out-of-order causes deadlock\n\n#### 2.9.2 MOW Delete Bitmap Four-Step Flow (Extremely High Priority)\n\n`schema_change.cpp:1595-1657` describes critical correctness flow for MOW table schema change:\n1. Newly imported rowsets during dual-write **skip** delete bitmap calculation\n2. After conversion completes, calculate delete bitmap for rowsets during dual-write period (lines 1618-1634)\n3. **Block new publish**, calculate delete bitmap for incremental rowsets (lines 1636-1652)\n4. Switch tablet state to `TABLET_RUNNING` (lines 1653-1656)\n\n- [ ] When modifying schema change delete bitmap path, are the four-step order and blocking logic complete? Step 3's blocking is key to ensuring consistency\n\n#### 2.9.3 Column Deletion Order Trap\n\n`return_columns` is calculated **before** `merge_dropped_columns()` (`schema_change.cpp:959-964` vs line 1062), this is intentional: `return_columns` only covers non-dropped columns, dropped columns are handled internally by `SegmentIterator` for delete predicate evaluation.\n\n- [ ] When modifying schema change column processing logic, is the timing of `return_columns` and dropped columns calculation understood?\n- [ ] `_check_row_nums` can be bypassed by `config::ignore_schema_change_check` (default false) (`schema_change.h:155-171`), this config should remain off in production\n\n### 2.10 Workload Group Memory Tracking\n\n`WorkloadGroup` (`workload_group.h`) uses dual-lock pattern: `_mutex` protects name/version/memory limit/resource_ctxs/shutdown state, `_task_sched_lock` protects scheduler and cgroup pointers.\n\n- [ ] **Memory tracking inconsistency**: `check_mem_used()` (lines 115-123) calculates `_total_mem_used + _wg_refresh_interval_memory_growth` for watermark check, but `exceed_limit()` (lines 132-135) **only compares `_total_mem_used > _memory_limit`**, ignoring `_wg_refresh_interval_memory_growth`. This means during refresh interval, workload group may exceed hard limit without being detected by `exceed_limit()`\n- [ ] Does new memory check logic use the correct method? If precise limit checking is needed, should reference `check_mem_used()` approach\n\n---\n\n## Part 3: FE Module Review Guide\n\n### 3.1 Metadata Persistence (EditLog)\n\n#### 3.1.1 Write Path\n\nAll metadata modifications must go through EditLog persistence. Flow:\n1. Modify in-memory state\n2. Call `editLog.logXxx()` to write log\n3. Log is replicated to Follower via BDBJE\n4. Caller blocks until persistence confirmation\n\n**Review Checkpoints:**\n\n- [ ] Do new DDL/DML operations call corresponding `editLog.logXxx()` methods? Omission causes Follower state inconsistency\n- [ ] Do new OperationTypes have corresponding replay handling in `EditLog.loadJournal()` switch?\n- [ ] Is replay logic **strictly equivalent** to master execution logic? Any difference causes Follower state drift\n- [ ] Do new serializable objects correctly implement Gson serialization? Are `@SerializedName` annotations complete?\n- [ ] Can Image checkpoint correctly include new metadata?\n- [ ] Can master failover at any moment guarantee no duplicate or lost data? Short-term task failures are tolerable, but memory/transaction leaks and data inconsistencies are not allowed.\n\n#### 3.1.2 EditLog Batch and Outside-Lock Commit Mode\n\n`DatabaseTransactionMgr` implements two EditLog commit modes, controlled by `Config.enable_txn_log_outside_lock`:\n\n- **Inside-lock mode** (default): Directly calls `editLog.logInsertTransactionState()` within transaction lock\n- **Outside-lock mode**: Only modifies in-memory state and calls `submitEdit()` to submit to queue within lock, waits for persistence completion via `awaitTransactionState()` outside lock (`DatabaseTransactionMgr.java:383-395`)\n\nFuzzy testing (`DorisFE.java:612-615`) randomly switches between these two modes, so both paths must be correct.\n\n**Review Checkpoints:**\n\n- [ ] Do modifications involving transaction EditLog consider both inside-lock and outside-lock commit paths?\n- [ ] In outside-lock mode, are `awaitTransactionState()` timeout and exception handling correct? Timeout doesn't mean failure—transaction may still be committing\n- [ ] Do new transaction state modifications complete all in-memory state changes before `submitEdit()` submission? After submission, other threads may immediately see new state\n\n#### 3.1.3 EditLog Fatal Error Handling\n\nWhen EditLog flush fails, FE calls `System.exit(-1)` to terminate process—this is **intentional**, preventing split-brain.\n- [ ] Does new code catch and swallow exceptions in EditLog-related paths? This may cause metadata loss\n\n### 3.2 Lock Hierarchy and Concurrency\n\n#### 3.2.1 Confirmed Lock Orders\n\n| Scenario | Lock Order |\n|------|------|\n| ConsistencyChecker | `jobs_lock` -> `synchronized(job)` -> `db_lock` |\n| AlterJobV2 | `synchronized(job)` -> `db_lock` |\n| TabletScheduler | `synchronized(this)` **internal** operations, then **release**, then acquire `db_lock` (two-step method avoids deadlock) |\n| DatabaseTransactionMgr | Its RWLock is **leaf lock**, should not acquire other locks internally |\n| MTMVTask | Acquire table locks after sorting by table ID (avoids deadlock) |\n\n**Review Checkpoints:**\n\n- [ ] Do new lock acquisitions follow the above lock orders?\n- [ ] Is `tryLock(timeout)` used instead of infinite-wait `lock()`? FE's standard pattern is `tryLock` + timeout logging\n- [ ] **Lock timeout standard values**: Env lock = 5s, DB write lock = 100s, Table read lock = 1min. Operations involving multiple tables must lock in ascending table ID order (implemented via `PriorityQueue`) to prevent deadlock\n- [ ] Does `Database.tryWriteLock()` log current owner thread on timeout?\n- [ ] Are there paths acquiring database lock within `synchronized` block? This is a known deadlock risk, should use two-step method\n\n#### 3.2.2 ConcurrentModificationException\n\n- [ ] Is traversal of shared collections under lock protection? `CatalogRecycleBin.write()` added `synchronized` because dump image API concurrent access caused CME\n- [ ] Can there be concurrent modifications during traversal of `Map` / `List`? Prefer `ConcurrentHashMap` or snapshot before traversal\n\n#### 3.2.3 Known Deadlock Cases (New Code Must Avoid Similar Patterns)\n\nThe following are confirmed deadlock risk points in codebase, be extra vigilant when reviewing similar patterns:\n\n| Location | Pattern | Description |\n|------|------|------|\n| `MasterCatalogExecutor.java:70-78` | Initiating RPC while holding catalog lock | RPC timeout blocks catalog operations for long time |\n| `ExternalDatabase.java:137-158` | Nested acquisition of db lock + makeSureInitialized() | Comments explicitly mark deadlock avoidance strategy |\n| `HiveMetaStoreCache.java:126-130` | Acquiring cache lock in thread pool callback | Comment marks \"this may cause a deadlock\" |\n| `CacheFactory.java:85-92` | Circular dependency in cache initialization | Avoided via lazy initialization |\n\n- [ ] Does new code initiate RPC or IO operations while holding locks? This is the most common source of deadlock and performance issues\n- [ ] Does new code acquire locks in callbacks (e.g., thread pool, cache loader) that callers may hold?\n\n### 3.3 Nereids Optimizer\n\n#### 3.3.1 Rule Implementation Specifications\n\n**Review Checkpoints:**\n\n- [ ] Is new Rule's `RuleType` added to `RuleType.java` enum?\n- [ ] Does Rule correctly handle **all Join types**? Pay special attention to ASOF variants (`ASOF_LEFT_INNER_JOIN`, etc.). `COULD_PUSH_THROUGH_LEFT/RIGHT` lists in `PushDownFilterThroughJoin` explicitly include ASOF types\n- [ ] Are constant predicates (`slots.isEmpty()`) correctly handled? Inner Join can push to both sides, Outer Join cannot\n- [ ] Is there **infinite loop risk**? Can created nodes be matched again by another rule in same batch? Code comments have multiple \"separate topDown passes to avoid dead loops\" warnings\n- [ ] Does `OneRewriteRuleFactory` pattern match correct wrapper nodes? For example, `EliminateJoinByFK` only matches `project(join)` not bare `join`\n- [ ] Are predicates checked for `containsUniqueFunction()`? Predicates containing unique functions cannot be pushed down\n\n#### 3.3.2 Rewriter Stage Order\n\nRewriter organizes rules into multi-stage pipeline, key dependencies:\n- `PUSH_DOWN_FILTERS` is called 5-6 times throughout pipeline, because it interacts with many other transformations\n- `InferPredicates` is called twice (with `PUSH_DOWN_FILTERS` in between), because eliminating outer join exposes new inference opportunities\n- `costBased(...)` rules are controlled by `runCboRules` flag\n\n- [ ] Is new rewrite rule placed in correct stage? For example, rules needing predicate pushdown should be after `PUSH_DOWN_FILTERS`\n- [ ] Is new exploration rule registered in `RuleSet`?\n\n#### 3.3.3 Property Derivation\n\n`RequestPropertyDeriver` implements top-down property requests:\n\n- [ ] Does new Physical Operator have visit method in `RequestPropertyDeriver`?\n- [ ] Are multiple alternative property requirements provided (e.g., HashJoin provides both shuffle and broadcast choices)? Single requirement narrows search space\n- [ ] Is `PhysicalProject` alias penetration logic correct? Only `Alias(SlotRef)` and bare `SlotRef` can penetrate\n\n### 3.4 Transaction Management\n\n#### 3.4.1 Transaction Lifecycle\n\nState machine: `PREPARE -> COMMITTED -> VISIBLE` (or `ABORTED`)\n\n- [ ] After transaction commit, can `PublishVersionDaemon` correctly send `PublishVersionTask` to all BEs?\n- [ ] Does publish wait for quorum confirmation?\n- [ ] Does transaction correctly abort after timeout?\n\n**Known race condition—`TransactionState.tableIdList`**: At `TransactionState.java:632`, TODO comment marks that `addTableId()` / `removeTableId()` lack synchronization protection. `tableIdList` is plain `ArrayList`, has race condition (CME or data loss) when multiple threads concurrently add/remove table IDs.\n\n- [ ] Are operations on `TransactionState.tableIdList` under external lock protection? Or should it be replaced with thread-safe collection?\n\n#### 3.4.2 Cloud Mode Differences\n\nIn Cloud mode, transactions are centrally managed by Meta-Service, not FE:\n- [ ] Do Cloud mode transaction operations use `CloudGlobalTransactionMgr` instead of local `GlobalTransactionMgr`?\n- [ ] **Unsafe downcasting prohibited**: Current code has 4 places forcibly casting `GlobalTransactionMgrIface` to `CloudGlobalTransactionMgr` (e.g., `InternalCatalog.java:994`). These casts cause ClassCastException in non-Cloud mode. New code should add required methods by extending `GlobalTransactionMgrIface` interface, not rely on downcasting\n- [ ] Does version retrieval use `getVisibleVersion()` (RPC + cache in Cloud mode)?\n- [ ] Is `getVisibleVersion()`'s `RpcException` correctly handled? Multiple callers catch and return 0, may cause MV refresh misjudgment\n\n### 3.5 Exception System\n\n```\nUserException (checked)\n ├── AnalysisException -- SQL analysis error\n ├── DdlException -- DDL execution failure\n ├── MetaNotFoundException -- Metadata lookup failure\n ├── LoadException -- Load failure\n └── ...\n\nNereidsException (unchecked, RuntimeException)\n ├── nereids/AnalysisException -- Nereids-specific, different from common/\n └── nereids/ParseException\n```\n\n- [ ] **Note Nereids has its own `AnalysisException`** (`nereids/exceptions/AnalysisException.java`, extends RuntimeException), which is a **different class** from `common/AnalysisException` (checked)\n- [ ] Does `ErrorReport.reportAnalysisException()` use correct `ErrorCode`? Custom Doris error codes start from 5000\n- [ ] Does new RPC error handling use `Status` pattern (`TStatusCode` / `PStatus`)?\n\n### 3.6 OlapTable Version Visibility\n\n- [ ] `getVisibleVersion()` behaves differently in Cloud vs non-Cloud mode:\n - Non-Cloud: Returns local `tableAttributes.getVisibleVersion()`\n - Cloud: RPC to meta-service, has cache TTL\n- [ ] Can Cloud mode's version cache TTL (`cloudTableVersionCacheTtlMs`) cause reading stale versions?\n- [ ] `VERSION_NOT_FOUND` is forcibly converted to version=1 (`PARTITION_INIT_VERSION`)—cannot distinguish \"never loaded\" from \"loaded to version 1\"\n\n### 3.7 External Catalog Concurrency Traps\n\n#### 3.7.1 Initialization Race Condition\n\n`ExternalCatalog.makeSureInitialized()` (`ExternalCatalog.java:354-379`) is synchronized, but has the following traps:\n\n- [ ] **`isInitializing` guard is dead code in `ExternalCatalog`**: Although code checks `isInitializing` (line 358), it's never set to `true` (only set to `false` in finally). Compare with `ExternalDatabase.makeSureInitialized()` (`ExternalDatabase.java:137-158`) which **correctly** sets `isInitializing = true`. This means ExternalCatalog has no reentrancy protection\n- [ ] **`lowerCaseToDatabaseName.clear()` race window**: `getFilteredDatabaseNames()` (line 504) first `clear()` then refills `ConcurrentMap`, this method is **not synchronized**. Between clear and refill completion, concurrent `get()` calls return null\n\n#### 3.7.2 resetToUninitialized() NPE Risk\n\n`resetToUninitialized()` (`ExternalCatalog.java:581-590`) sets `objectCreated` and `initialized` to false within synchronized block, then calls `onClose()` (lines 773-784) to set `executionAuthenticator` and `transactionManager` to null.\n\n- [ ] Concurrent queries that have passed `makeSureInitialized()` may be using these fields being set to null. synchronized only protects callers of other synchronized methods, not non-synchronized read paths already in execution\n\n### 3.8 Backup/Restore State Machine\n\n#### 3.8.1 BackupJob State Machine and EditLog Timing\n\nEditLog write timing in `BackupJob` state machine is **uneven**, this is the most important non-obvious convention:\n\n- [ ] **PENDING→SNAPSHOTING doesn't write EditLog** (`BackupJob.java:620-624`): After FE restart, state reverts to PENDING, all snapshot tasks are regenerated and resent. Don't add EditLog when modifying this transition—restart recovery relies on this behavior\n- [ ] **BarrierLog mechanism**: `prepareSnapshotTaskForOlapTableWithoutLock()` creates `BarrierLog` for each table and writes via `editLog.logBarrier()` (lines 648-657), returned `commitSeq` is stored in properties map, used to ensure snapshot consistency\n- [ ] **UPLOAD_INFO stage errors only retry, not cancel** (`BackupJob.java:509-514`): This is the final stage, code comments explicitly state continuous retry until timeout even when encountering unrecoverable errors. Don't call `cancelInternal()` in this stage when modifying error handling\n\n#### 3.8.2 RestoreJob state vs showState\n\n`RestoreJob` has two independent state fields (`RestoreJob.java:157,162`):\n- `state`: Serialized persistent actual state, master sets to FINISHED **before** writing EditLog\n- `showState`: Non-serialized, only for display, master sets to FINISHED **after** writing EditLog (line 2191)\n\n- [ ] This prevents follower from seeing FINISHED state before EditLog replication completes. After deserialization, `gsonPostProcess()` syncs `showState = state`. Must update both fields when modifying state transitions\n\n---\n\n## Part 4: Cloud Module Review Guide\n\n### 4.1 FDB Transaction Mode\n\nCloud module uses FoundationDB (FDB) as the sole metadata storage. All metadata operations must be completed within FDB transactions.\n\n#### 4.1.1 TxnKv Interface Specifications\n\n- [ ] Is `TxnErrorCode` return value from each `Transaction::get()` / `commit()` checked? Marked as `[[nodiscard]]`\n- [ ] Is `TXN_CONFLICT` correctly handled? `commit_txn_immediately` returns error without retry, `commit_txn_eventually` has lazy committer path\n- [ ] Can transaction exceed FDB's 10MB limit? Large transactions need to use `TxnLazyCommitter` for batched commits\n- [ ] Is `TXN_MAYBE_COMMITTED` correctly handled? This indicates commit may have succeeded or failed, requires idempotent retry\n\n#### 4.1.2 Key Encoding Specifications\n\nThree key spaces:\n- `0x01`: User space (mutable metadata)\n- `0x02`: System space (service registration, encryption keys)\n- `0x03`: Version space (multi-version metadata with versionstamp, for snapshots)\n\n- [ ] Do new keys use correct key space prefix?\n- [ ] Does key encoding use `encode_bytes()` (order-preserving byte encoding) and `encode_int64()` (big-endian + sign marker)?\n- [ ] Does compound key field order match query pattern? Range scan relies on key prefix\n\n#### 4.1.3 Commit Path\n\nTwo commit paths:\n- **`commit_txn_immediately`**: Single FDB transaction completes all operations (read txn info, move tmp_rowset, bump version, update stats)\n- **`commit_txn_eventually`**: Uses lazy committer for batched commits when transaction is too large\n\n- [ ] Are the three config switches for lazy commit (`is_2pc`, `enable_txn_lazy_commit`, `config::enable_cloud_txn_lazy_commit`) correctly combined?\n- [ ] Is fallback path from `commit_txn_immediately` to `commit_txn_eventually` (triggered by `TXN_BYTES_TOO_LARGE`) correct?\n\n**Known risk—`commit_txn` infinite loop**: The `commit_txn` entry in `meta_service_txn.cpp:3257-3341` uses `do { ... } while(true)` loop to dispatch `commit_txn_immediately` and `commit_txn_eventually`. When pending lazy-committed transactions are found, it retries, but this loop **has no independent retry count limit**. If lazy committer continuously fails to complete (e.g., FDB continuous conflicts), theoretically infinite loop.\n\n- [ ] Do modifications involving `commit_txn` dispatch loop consider maximum retry count or total timeout?\n\n#### 4.1.4 Versionstamp Usage\n\nTransaction ID is derived from FDB versionstamp (`get_txn_id_from_fdb_ts`).\n\n- [ ] Does code depending on FDB versionstamp correctly use `atomic_set_ver_key()` / `atomic_set_ver_value()`?\n- [ ] Is `get_versionstamp()` only called after `commit()` succeeds?\n\n### 4.2 Meta-Service RPC Specifications\n\n#### 4.2.1 RPC Preprocessing\n\nEach RPC uses `RPC_PREPROCESS` macro for preprocessing:\n- StopWatch timing\n- DORIS_CLOUD_DEFER sets response status and records metrics on return\n- `RPC_RATE_LIMIT` performs per-instance QPS throttling\n\n- [ ] Do new RPCs use `RPC_PREPROCESS` and `RPC_RATE_LIMIT`?\n- [ ] Is response `MetaServiceCode` correctly set?\n\n#### 4.2.2 MetaServiceProxy Automatic Retry\n\n`MetaServiceProxy` decorator automatically retries (exponential backoff) on following error codes:\n- `KV_TXN_STORE_*_RETRYABLE`, `KV_TXN_TOO_OLD`, `KV_TXN_CONFLICT`\n\n- [ ] Do new RPCs need to be idempotent? Retry means same request may execute multiple times\n- [ ] Do non-idempotent operations correctly handle `TXN_MAYBE_COMMITTED`?\n\n### 4.3 Recycler Safety Specifications\n\nRecycler is responsible for GC of expired/deleted data:\n\n- [ ] Before deleting rowset data, are associated transactions aborted first (`abort_txn_for_related_rowset`)? Prevents commit-after-delete data loss\n- [ ] Are recycle operations best-effort + next-round retry mode? Should not stop entire recycle flow due to single failure\n- [ ] Is packed file reference count recycling correct? (`recycle_packed_files`)\n\n### 4.4 Delete Bitmap (Cloud MoW, Extremely High Priority)\n\nCloud mode delete bitmap management fundamentally differs from shared-nothing mode:\n\n- [ ] Bitmaps in `CloudTxnDeleteBitmapCache` contain sentinel marks and `TEMP_VERSION_COMMON`, **must manually replace version** before use. This is a critical correctness constraint explicitly marked in code comments\n- [ ] Empty rowset marks returned by `is_empty_rowset()` are not actively cleared (rely on expiration cleanup), does this affect correctness?\n- [ ] **`CloudTxnDeleteBitmapCache` LRU eviction side effect**: When cache entry is LRU evicted, `publish_status` is reset to `INIT`, causing next access to **completely recalculate** delete bitmap (not incremental update). This is correct but expensive. Additionally, marks written by `mark_empty_rowset()` are never actively cleared—they only disappear when entire cache entry is evicted. On high-frequency MoW tables with insufficient cache capacity, causes repeated recalculation\n- [ ] Do modifications involving `CloudTxnDeleteBitmapCache` consider impact of state reset after eviction?\n- [ ] Are `CalcDeleteBitmapTask`'s compaction stats consistent with latest state in meta-service?\n\n---\n\n## Part 5: Cross-Module Concerns\n\n### 5.1 FE-BE Protocol Compatibility\n\n- [ ] Do new `TPlanNodeType`s have corresponding branches in BE's processing switch?\n- [ ] Can deprecated node types (e.g., `MERGE_JOIN_NODE`, `KUDU_SCAN_NODE`) still be gracefully handled?\n- [ ] If new variables are added, are they correctly set in all paths where FE sends execution plans to BE?\n\n### 5.2 Cloud Mode vs Shared-Nothing Mode\n\nCode extensively uses `Config.isNotCloudMode()` / `Config.isCloudMode()` branches.\n\n- [ ] Do new features consider both modes?\n- [ ] In Cloud mode, tablets are not bound to specific BE—any compute node can load any tablet's metadata\n- [ ] In Cloud mode, rowsets go through two-phase protocol (`prepare_rowset` -> `commit_rowset`), with recycle marker protection in between\n- [ ] In Cloud mode, partition version is centrally managed by Meta-Service, not FE\n\n### 5.3 Merge-on-Write Unique Key Tables\n\nMoW tables are one of the most complex data paths, require special attention during review:\n\n- [ ] Write path: Does `SegmentWriter::probe_key_for_mow()` correctly probe existing rowsets?\n- [ ] Publish path: Is delete bitmap calculation under `_rowset_update_lock` protection?\n- [ ] Query path: Is delete bitmap version-aligned before reading?\n- [ ] Compaction path: Does `calc_delete_bitmap_for_compaction()` correctly handle `allow_delete_in_cumu_compaction`?\n- [ ] Cloud mode: Does delete bitmap cache have expiration mechanism? Is `_unused_delete_bitmap` cleaned up promptly?\n\n### 5.4 Performance Hotspot Concerns\n\n- [ ] Does `SendBatchThreadPool` (64 threads, 102K queue) have saturation risk in heavy shuffle scenarios?\n- [ ] Does `SegmentPrefetchThreadPool` (max 2000 threads) create too many OS threads in intensive scan scenarios?\n- [ ] In Cloud mode, does label deduplication check perform linear scan on labels with many aborted transactions, causing performance cliff?\n- [ ] Is serialization overhead of `_serialize_block_to_row_column()` for wide tables within expectations?\n- [ ] COW columns trigger deep copy on `mutate()` when `use_count > 1`—are there unnecessary copies in high-concurrency scenarios?\n\n---\n\n## Part 6: Testing Review Guide\n\n### 6.1 Regression Test Specifications\n\n- [ ] Is `order_qt_` prefix or manual `ORDER BY` used to ensure ordered results?\n- [ ] Do expected error cases use `test { sql \"...\"; exception \"...\" }` pattern?\n- [ ] Are tables **DROPped before use** (`DROP TABLE IF EXISTS xxx`), not after test ends?\n- [ ] Do simple tests directly hardcode table names instead of using `def tableName`?\n- [ ] Do tests cover these seven scenarios: empty table, all null, mixed null, pure non-null, constant, constant null, nullable wrapper (`nullable()`)?\n- [ ] Are result files (`.out`) auto-generated via `-genOut`? **Handwriting prohibited**\n\n### 6.2 BE Unit Test Specifications\n\n- [ ] Are Google Test's `TEST` / `TEST_F` macros used?\n- [ ] Are tests independent? Not dependent on other tests' execution order\n- [ ] Are `ASSERT_*` (failure terminates test) and `EXPECT_*` (failure continues execution) used correctly?\n- [ ] Do memory-intensive tests use `MemTracker` to check for leaks?\n\n### 6.3 FE Unit Test Specifications\n\n- [ ] Is JUnit 5's `@Test` annotation used?\n- [ ] Do test method names clearly describe test scenarios?\n- [ ] Is `Assertions.assertXxx()` used instead of JUnit 4's static imports?\n- [ ] Do tests involving concurrency use `CountDownLatch` / `CyclicBarrier` for synchronization?\n\n### 6.4 FE-BE Integration Test Specifications\n---\n\n## Part 7: Known High-Risk Code Patterns Quick Reference\n\n| Pattern | Risk Level | Location | Description |\n|------|---------|------|------|\n| `static_cast(status_returning_call)` | **CRITICAL** | BE global (195+ places) | Silently swallows errors, no logging |\n| Delete bitmap TEMP_VERSION not replaced | **CRITICAL** | `cloud_txn_delete_bitmap_cache.h:72-75` | Data visibility error |\n| `be_exec_version` mismatch | **CRITICAL** | FE-BE communication all paths | Serialization format incompatibility causes data corruption |\n| `(DataTypeNullable*)` C-style cast | **HIGH** | `block.h merge_impl` | No protection in release builds |\n| `sizeof(size_t*)` vs `sizeof(size_t)` | **HIGH** | `data_type.cpp:195` | Happens to work on 64-bit, crashes on 32-bit |\n| Lazy commit three-switch combination | **HIGH** | `meta_service_txn.cpp` | Missing config causes large transaction failure |\n| DecimalV2 original_precision default UINT32_MAX | **HIGH** | `data_type_decimal.h` | Affects string representation correctness |\n| `MOW` default false | **MEDIUM** | `segment_writer.h:70` | Omission causes deduplication to be silently skipped |\n| Cloud version cache TTL | **MEDIUM** | `OlapTable.getVisibleVersion()` | May read stale version |\n| Label scan linear complexity | **MEDIUM** | `meta_service_txn.cpp` | Many aborted txns cause performance cliff |\n| `THROW_IF_ERROR` no outer catch | **MEDIUM** | vec/ execution engine | Uncaught exception propagation causes crash |\n| `THROW_IF_ERROR` in Defer lambda | **CRITICAL** | `pipeline_task.cpp:412-413, 638-639` | Throwing exception during stack unwinding causes `std::terminate` |\n| `THROW_IF_ERROR` in Thrift RPC handler | **HIGH** | `backend_service.cpp:1323,1326` | No try/catch protection, connection drops |\n| `DCHECK` disappears in RELEASE builds | **HIGH** | BE global (2,872 places) | No invariant protection in production, should migrate","tools/cost_model_evaluate/README.MD":"\n\nThis code is used to evaluate the cost model in doris.\nYou can config the query in config of main.py \n\nBefore running, you should install the libraries in requirements.txt","tools/lzo/README.md":"\n\n# LZO Writer\n\nThis is simple LZO writer, you can use it to generate lzo compressed file.\n\nSee `main()` for details\n\n## Compilation\n\n```\nsh build.sh\n```\n\n## Generate file\n\n```\n./lzo_writer file_name.lzo\n```\n\n","tools/show_segment_status/README.md":"\n\nThis tool is used to get the progress of all current table transitions \nduring the online `segment_2` function.\n\nCurrently, you can specify 3 dimensions (in the conf file) to view the \nresults, you can specify one of them individually, or you can customize \nthe combination (that is, specify multiple at the same time).\n\n# Note\nWe use MySQLdb python lib to fetch meta from FE, so you must install it.\n\nYou can get MySQLdb lib from https://pypi.python.org/pypi/MySQL-python, \nthen you can install it as follows:\n```\n$ tar zxvf MySQL-python-*.tar.gz\n$ cd MySQL-python-*\n$ python setup.py build\n$ python setup.py install\n``` \n\n# Steps\n1. Fill in the conf according to your cluster configuration, and specify \n the table or be you want to watch.\n2. Execute `python show_segment_status.py`\n\n# Example\n1. If you want to watch the process of a table named `xxxx`, you can specify \n `table_name = xxxx` in conf file\n\n2. If you want to watch the process on be whose be_id is `xxxx`, you can specify \n `be_id = xxxx` in conf file\n\n# Output Example Format\n\nThe first number is about segment v2, the latter one is the total count.\n\n```\n==========SUMMARY()===========\nrowset_count: 289845 / 289845\nrowset_disk_size: 84627551189 / 84627551189\nrowset_row_count: 1150899153 / 1150899153\n===========================================================\n==========SUMMARY(table=xxxx)===========\nrowset_count: 289845 / 289845\nrowset_disk_size: 84627551189 / 84627551189\nrowset_row_count: 1150899153 / 1150899153\n===========================================================\n==========SUMMARY(be=10003 )===========\nrowset_count: 79650 / 79650\nrowset_disk_size: 24473921575 / 24473921575\nrowset_row_count: 331449328 / 331449328\n===========================================================\n```\n","tools/tpcds-tools/README.md":"\n\n# Run TPCDS with one command\n\nInstall the Trino TPCDS generator on every FE and BE using the\n[shared installation instructions](../benchmark/README.md#install-the-generators-once),\nthen configure `conf/doris-cluster.conf` and run:\n\n```bash\n./bin/run-tpcds.sh -s 1 -d tpcds_sf1\n```\n\nThis streams generated rows directly into Doris, creates the existing benchmark\ntables, collects statistics, and records three executions per query. It supports\nSF1, SF100, SF1000, and SF10000. Preparation requires a new database.\nUse `--queries-only` to measure existing data again, with the same scale factor.\n\nSee [shared workflow options, result files, and timing semantics](../benchmark/README.md).\nTPCDS uses the original scale-specific table DDL and multi-table queries.\nInstall the plugin before running the command.\n\n# Original file-based workflow\n\n## Usage\n\nThese scripts are used to make tpc-ds test.\nfollow the steps below:\n\n### 1. build tpc-ds dsdgen dsqgen tool.\n\n ./bin/build-tpcds-tools.sh\n\n If the build failed in dbgen tools' compilation, update your GCC version or change all \"TPC-DS_Tools_v3.2.0new.zip\" in build-tpcds-dbgen.sh to \"TPC-DS_Tools_v3.2.0.zip\"\n\n### 2. generate tpc-ds data. use -h for more infomations.\n\n ./bin/gen-tpcds-data.sh -s 1\n\n### 3. create tpc-ds tables. modify `conf/doris-cluster.conf` to specify doris info, then run script below.\n\n ./bin/create-tpcds-tables.sh -s 1\n\n### 4. load tpc-ds data. use -h for help.\n\n ./bin/load-tpcds-data.sh\n\n### 5. run tpc-ds queries.\n\n ./bin/run-tpcds-queries.sh -s 1\n","AGENTS.md":"# AGENTS.md — Apache Doris\n\nThis is the codebase for Apache Doris, an MPP OLAP database. It primarily consists of the Backend module BE (`be/`, execution and storage engine), the Frontend module FE (`fe/`, optimizer and transaction core), and the Cloud module (`cloud/`, storage-compute separation). Your basic development workflow is: modify code, build using standard procedures, add and run tests, and submit relevant changes.\n\n## Security Threat Model\n\nFor security scans, vulnerability triage, security reviews, and changes involving authentication, authorization, network boundaries, external catalogs, cloud tenancy, or other security-sensitive behavior, read `SECURITY.md` first, then follow it to `threat-model.md`. Use `threat-model.md` to determine in-scope components, trust boundaries, attacker roles, explicit non-goals, and triage classification. Findings that are out of model or by design under `threat-model.md` should be reported as such, not treated as Doris vulnerabilities. Such reviews will not be conducted unless explicitly requested by the user, and no conclusions regarding this matter need to be provided.\n\n## Remote Artifacts and Dynamic Code Loading\n\nDo not add any new code path that fetches an artifact from a user-supplied URL and loads it into a Doris process. This covers `driver_url`-style catalog/resource properties, plugin and UDF locations, and any other property whose value reaches a `URLClassLoader`, `Class.forName`, `System.load`/`dlopen`, or an equivalent BE-side loader. A user who can write such a property gets code execution inside the process that loads it, so the property's effective privilege is that process, not the DDL privilege it was gated on.\n\nThe existing `driver_url` paths (jdbc / iceberg-jdbc / paimon-jdbc catalogs) remain for backward compatibility. They are grandfathered, not a precedent — do not copy the pattern into a new connector or feature.\n\nWhen a feature genuinely needs an operator-supplied artifact, read it from a local, operator-controlled directory (the `jdbc_drivers_dir` / plugin-directory pattern), not from a URL. The ADBC catalog's `AdbcDriverPathResolver` is the model: remote schemes rejected outright, a scheme-less value restricted to a bare file name so no path separator can escape the configured directory, and the check placed where the artifact is loaded rather than only at CREATE.\n\nEvery consumer of the **jdbc-flavored** `driver_url` (the jdbc / iceberg-jdbc / paimon-jdbc catalogs, which do accept a URL) must route the raw value through `JdbcDriverUrlSecurity.check` (`fe-foundation`, `foundation.security`) from its property holder's statement-time validation — `JdbcCatalogProperties.checkCreateTimeOnlyRules` and the iceberg/paimon JDBC metastore holders' `validate()` — which the engine reaches on both CREATE and ALTER CATALOG and never on replay or a catalog rebuild. That class is the single source of truth for the rule; do not fork or re-derive it per connector, and never call it from a holder's `of()`. Such a property must also get a row in the key table of `PluginDrivenExternalCatalog#driverUrlKeysOf` (one table drives both the gate trigger set and the checked values), which is how the operator's `jdbc_driver_secure_path` / `jdbc_driver_url_white_list` policy reaches the ALTER CATALOG path — ALTER never runs `Connector.preCreateValidation`, where CREATE applies that policy, so a key missing from that table lets an ALTER repoint the jar past the operator's configuration.\n\nSeparately, any new outbound HTTP request the FE or BE issues to a host named by a non-SUPER user is an SSRF surface. Call it out explicitly in the PR description together with the privilege required to reach it.\n\n## Thrift and Protobuf Field Ids\n\nEvery change under `gensrc/thrift` and `gensrc/proto` must keep every upgrade compatible (branch-3.1 → 4.0 → 4.1 → 4.2 → master): a field id or enum value means the same thing on every branch, an id is never reused, and a new field is optional. The rule and how to review against it are in `gensrc/AGENTS.md`; read it before touching those directories.\n\n## When running in a WORKTREE directory\n\nTo ensure smooth test execution without interference between worktrees, the first thing to do upon entering a worktree directory is to check if `.worktree_initialized` exists. If not, execute `hooks/setup_worktree.sh`, setting `$ROOT_WORKSPACE_PATH` to the base directory (typically `${DORIS_REPO}`) beforehand. After successful execution, verify that `.worktree_initialized` has been touched and that `thirdparty/installed` dependencies exist correctly.\n\nSubmodules are not part of the default worktree setup. If the task does not compile Doris or run tests that require submodules, leave all submodules uninitialized. Build and test entry points initialize the submodule paths they require. Only initialize a submodule manually when the task actually needs it and an entry point does not handle it; initialize the exact required path instead of running an unscoped recursive update for every submodule.\n\nWhen working in worktree mode, all operations must be confined to the current worktree directory. Do not enter `${DORIS_REPO}` or use any resources there. Compilation and execution must be done within the current worktree directory. The compiled Doris cluster must use random ports not used by other worktrees (modify BE and FE conf before compilation, using a uniform offset of `${DORIS_PORT_OFFSET_RANGE}` from default ports without conflicting with other worktrees' ports). Run from the `output` directory within the worktree. To run regression tests, modify `regression-test/conf/regression-conf.groovy` and set the port numbers in `jdbcUrl` and other configuration items to your new ports so the corresponding worktree cluster can be used for regression testing.\n\n## Coding Standards\n\nAssert correctness only—never use defensive programming with `if` or similar constructs. Any `if` check for errors must have a clearly known inevitable failure path (not speculation). If no such scenario is found, strictly avoid using `if(valid)` checks. However, you may use the `DORIS_CHECK` macro for precondition assertions (if inside performance-sensitive areas like loops, it can only be `DCHECK`). For example, if logically A=true should always imply B=true, then strictly avoid `if (A && B)` and instead use `if (A) { DORIS_CHECK(B); ... }`. In short, the principle is: upon discovering errors or unexpected situations, report errors or crash—never allow the process to continue.\n\nFor `PaddedPODArray` and its peripheral packaging types, such as some certain Column, negative alignment allows the use of -1 as a valid index. No additional special handling is needed when the index may be -1.\n\nWhen adding code, strictly follow existing similar code in similar contexts, including interface usage, error handling, and locking patterns. When adding any code, first try to reference existing functionality. Second, examine the relevant context paragraphs to fully understand the logic.\n\nAfter adding code, conduct self-review and refactoring attempts to ensure good abstraction and reuse as much as possible.\n\n### Code Style Enforcement\n\nAll code must pass style checks before committing. Use the corresponding skill for detailed step-by-step procedures.\n\n**BE (C++) Formatting**: Run `build-support/clang-format.sh` to auto-fix formatting. This script enforces clang-format v16; do not use other versions. Run `build-support/check-format.sh` to check without modifying files. See the `be-code-style` skill for details.\n\n**BE (C++) Static Analysis**: After building BE (which generates `compile_commands.json`), run `build-support/run-clang-tidy.sh` to check modified C++ files against the `.clang-tidy` config. The script parses `git diff` to filter warnings to changed lines where possible, reducing noise from pre-existing code (diagnostics from included headers may still appear). For Cloud C++ files, pass `--build-dir` pointing to the Cloud compilation database (e.g., `cloud/build_ASAN`). Try to fix all reported warnings; if a warning cannot be reasonably fixed, add a `// NOLINT` comment with justification and report it. See the `clang-tidy-check` skill for details.\n\n**BE (C++) Header Hygiene**: BE configure enforces compile-time hygiene gates (header layering rules, include-closure/reach budgets, pch whitelist, extern-template pairing, unity-skip coverage) via `build-support/check-build-hygiene.sh` (~1s, pure text). Run it directly after touching BE headers, includes, template instantiation lists, or unity skip lists — do not wait for CI. Violations are build errors whose messages carry the mechanism and fix; deliberate budget/whitelist changes are one-line table edits in the same commit with justification. Rules and review checkpoints: `be/AGENTS.md`; mechanics: `be/README.md`.\n\n**FE (Java) Style**: Checkstyle is integrated into the Maven build (`maven-checkstyle-plugin`). Running `build.sh --fe` automatically validates style via `mvn validate`. If checkstyle fails, fix the reported issues according to `fe/check/checkstyle/checkstyle.xml`. See the `fe-code-style` skill for details.\n\n## Code Review\n\nWhen conducting code review (including self-review and review tasks), complete the key checkpoints from the `code-review` skill and provide conclusions for each key checkpoint when applicable. Other content does not require item-by-item responses; check them during the review process.\n\n## Build and Run Standards\n\nAlways use only the `build.sh` script with its correct parameters to build Doris BE and FE. For example, the simplest BE+FE build command is `./build.sh --be --fe`.\nBuild type can be set via `BUILD_TYPE` in `custom_env.sh`, but only set it to `RELEASE` when explicitly required for performance testing; otherwise, keep it as `ASAN`.\nYou may modify BE and FE ports and network settings in `conf/` before compilation to ensure correctness and avoid conflicts.\nBuild artifacts are in the current directory's `output/`. If starting the service, ensure all process artifacts have their conf set with appropriate non-conflicting ports and the correct `priority_networks` that match the current network environment, for example `priority_networks = 10.16.10.3/24`. Use `--daemon` when starting. Cluster startup is slow; wait at least 30s for success. If still not ready after waiting, continue waiting. If not ready after a long time, check BE and FE logs to investigate.\nFor first-time cluster startup, you may need to manually add the backend.\n\n## Testing Standards\n\nAll kernel features must have corresponding tests. Prioritize adding regression tests under `regression-test/`, while also having BE unit tests (`be/test/`) and FE unit tests (`fe/fe-core/src/test/`) where possible. Interface usage in test cases must first reference similar cases.\n\nYou must use the preset scripts in the codebase with their correct parameters to run tests (`run-regression-test.sh`, `run-be-ut.sh`, `run-fe-ut.sh`). Regression test result files must not be handwritten; they must be auto-generated via test scripts. When running regression tests, if using `-s` to specify a case, also try to use `-d` to specify the parent directory for faster execution. For example, for cases under `nereids_p0`, you can use `-d nereids_p0 -s xxx`, where `xxx` is the name from `suite(\"xxx\")` in the groovy file.\n\nKey utility functions in BE code, as well as the core logic of complete features, must have corresponding unit tests. If it is inconvenient to add unit tests, revisit the module design and function decomposition to ensure high cohesion and low coupling.\n\nAdded regression tests must comply with the following standards:\n\n1. Use `order_qt` prefix or manually add `order by` to ensure ordered results\n2. For cases expected to error, use the `test{sql,exception}` pattern\n3. After completing tests, do not drop tables; instead drop tables before using them in tests, to preserve the environment for debugging\n4. For ordinary single test tables, do not use `def tableName` form; instead hardcode your table name in all SQL\n5. Except for variables you explicitly need to adjust for testing current functionality, other variables do not need extra setup before testing. For example, nereids optimizer and pipeline engine settings can use default states\n6. For determined expected results, do not using methods like `assert` in test groovy files, but instead generate the `.out` file using `qt_sql` and similar methods.\n\n## Commit Standards\n\nFiles in git commit should only be related to the current modification task. Environment modifications for running (e.g., `conf/`, `AGENTS.md`, `hooks/`, etc.) must not be `git add`ed. When delivering the final task, you must ensure all actual code modifications have been committed.\n\nCommit messages must follow the format below, which mirrors the PR template (`.github/PULL_REQUEST_TEMPLATE.md`):\n\n```\n[]() \n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary: \n\n### Release note\n\n\n\n### Check List (For Author)\n\n- Test: \n - Regression test / Unit Test / Manual test / No need to test (with reason)\n- Behavior changed: No / Yes (with explanation)\n- Does this need documentation: No / Yes (with doc PR link)\n```\n\nKey rules for commit messages:\n\n1. The title must follow the `[type](module)` format validated by the PR title checker (`.github/workflows/title-checker.yml`). Common types include: `fix`, `feature`, `improvement`, `refactor`, `chore`, `test`, `doc`. Common modules include: `fe`, `be`, `cloud`, `regression`, `build`\n2. The short summary must be concise and written in imperative mood (e.g., `[fix](fe) Fix null pointer in scan node` not `[fix](fe) Fixed null pointer`)\n3. The `Issue Number` field must reference the corresponding GitHub Issue with `close #xxx` syntax when applicable\n4. The `Release note` section must be filled in for any user-visible behavior or feature change; write \"None\" for internal refactoring or test-only changes\n5. The `Problem Summary` section should cover the following content when available: problem reproduction method, root cause in code, end-to-end results/phenomena before and after repair, and the fix. If it's a refactoring, explain the reason. If it's a performance improvement, specify the case and the exact improvement amount. DO NOT mention any specific JIRA numbers. The background of the problem should be fully understandable through this section alone.\n6. The test section must honestly reflect the testing performed; do not claim tests that were not actually run\n\nFiles in a git commit should only be related to the current modification task. Environment modifications for running (for example `conf/`, `AGENTS.md`, `hooks/`) must not be `git add`ed. When delivering the final task, ensure all actual code modifications have been committed.\n","tools/pipeline-tracing/README.md":"\n\n# Pipeline Tracing and Display Tool\n\nIn the Pipeline execution engine, we split the execution plan tree of each Instance into multiple small Pipeline Tasks and execute them under our custom Pipeline scheduler. Therefore, in an environment with a large number of Pipeline Tasks executing, how these Tasks are scheduled across threads and CPU cores is an important factor for execution performance. We have developed a specialised tool to observe the scheduling process on a particular query or time period, which we call \"Pipeline Tracing\".\n\nThis tool converts record files to proper JSON format for visualization.\n\n## How to Use\n\n```shell\npython3 origin-to-show.py -s -d .json\n```\nto transfer record file `` to `.json`. Then it could be visualized.\n\n```shell\npython3 origin-to-show.py --help\n```\nfor help details.","tools/ssb-tools/README.md":"\n\n# Run SSB with one command\n\nThe Trino SSB connector generates rows while Doris reads them. It uses the same\n`ssb-dbgen` distribution as the existing tools, with stdout connected to the scan\nthrough a bounded pipe. There are no intermediate `.tbl` files, Stream Load\nrequests, or separate Trino servers.\n\n## Install the generator plugin once\n\nBuild on Linux with the same CPU architecture as the Doris nodes. The build needs\nJDK 17, Maven, GCC, Make, curl, patch, and sha256sum:\n\n```bash\n./trino-ssb/build.sh\n```\n\nCopy the resulting **entire directory** `trino-ssb/target/ssb` into the configured\nTrino plugin directory on **every FE and BE**. For current Doris installations:\n\n```bash\ncp -r trino-ssb/target/ssb /path/to/fe/plugins/trino_plugins/\ncp -r trino-ssb/target/ssb /path/to/be/plugins/trino_plugins/\n```\n\nStart or restart these nodes after installing the plugin. It consists of a JAR,\nthe native `dbgen` executable, and its distribution file. The executable must\nremain executable. No paths or executable commands are accepted as catalog properties.\n\n## Create, import, and benchmark\n\nSet the Doris connection in `conf/doris-cluster.conf`, then run:\n\n```bash\n./bin/run-ssb.sh -s 1 -d ssb_sf1\n```\n\nThis creates a new database and its `ssb_gen_ssb_sf1` catalog, creates the five\nSSB tables, and imports them using `INSERT INTO ... SELECT ...` from the catalog's\n`sf1` schema. It then builds `lineorder_flat`, collects statistics synchronously,\nand runs all 13 SSB queries and all 13 SSB Flat queries three times each.\nThe supported scale factors are 1, 100, and 1000. Use `--mode ssb` or `--mode flat`\nto select one benchmark; the default is `both`.\n\nEach run writes a new `results/-/` directory with:\n\n- `result.csv`: first-run, two repeated-run, and best repeated-run times in milliseconds.\n- `ssb/` and `ssb-flat/`: each query's output and errors.\n- `prepare.log` and `environment.txt`: import diagnostics and Doris settings.\n\nTiming includes the mysql client connection and result transfer, as in the old\nscripts. The `cold` column means the first execution; the script does not flush\ndata caches. SQL result caching and BE query caching are disabled for every measured\nsession. A query or import error stops the run with a nonzero exit status.\n\nPreparation requires a **new database**, so rerunning a failed or completed load\ncannot append duplicate data. Tables and the catalog are preserved for inspection.\nTo rerun only the queries against completed data:\n\n```bash\n./bin/run-ssb.sh -d ssb_sf1 --queries-only\n```\n\nUse `-c /path/to/cluster.conf` for a separate connection configuration, and\n`--result-dir /path/to/new-directory` for a custom output directory. Existing\nresult directories are never overwritten.\n\n## Generator consistency and tests\n\nThe connector keeps the old tools' default of ten `lineorder` generator partitions.\nEach dimension uses one generator. Changing the original dbgen partition count\nchanges some random streams, so partition count is intentionally fixed here.\nThe date generator runs in UTC on every BE. The native build enables dbgen's\nexisting 64-bit code path for large scale factors and avoids an out-of-bounds\nusage-counter update for its SSB-only random streams. Neither change replaces\nthe data-generation algorithm.\n\nThe existing benchmark SQL is preserved. In particular, SSB Q1.3 filters the\ngenerator's `d_weeknuminyear`, whereas SSB Flat Q1.3 uses `weekofyear()`; these week\ndefinitions differ, so their aggregate results are not expected to match.\n\nThe plugin build runs its Java tests, including full SF1 checksums of all five tables,\nhandle serialization, projected columns, large keys, producer failures, and early\ncursor closure. Run script orchestration tests with:\n\n```bash\npython3 -m unittest discover -s ../benchmark/tests -v\n```\n\nThe shared runner also supports [TPCH and TPCDS](../benchmark/README.md).\n\n# Original file-based workflow\n\n These scripts are used to make ssb and ssb flat test.\n The ssb flat data comes from ssb tables by way of 'INSERT INTO ... SELECT ...'.\n\n## ssb test, follow the steps below:\n### 1. build ssb dbgen tool.\n ./bin/build-ssb-dbgen.sh\n### 2. generate ssb data. use -h for more infomations.\n ./bin/gen-ssb-data.sh -s 1\n### 3. create ssb tables. modify `conf/doris-cluster.conf` to specify Doris cluster info, then run script below.\n ./bin/create-ssb-tables.sh -s 1\n### 4. load ssb data. use -h for help.\n ./bin/load-ssb-data.sh\n### 5. run ssb queries.\n ./bin/run-ssb-queries.sh\n\n## ssb flat test, follow the steps below:\n### 1. prepare ssb data, which means ssb test steps 1 to 4 have been done.\n### 2. run ssb flat queries.\n ./bin/run-ssb-flat-queries.sh\n","tools/row_to_column/README.md":"\n\nSteps of row-to-column conversion:\n\n1. Modify the corresponding cluster configuration in `conf`.\n\n2. Run `python convert_row_to_column.py` to generate all row storage to column storage commands.\n\n3. If the script runs without any results, it means that there is no row storage table in the cluster, so nothing need to be done.\n\n4. If there is a table in the script that fails to obtain information.\n\n Please use `show create table` in the cluster to see the `storage_type` of the table. If it is row storage, please manually convert it.\n\n5. After the schema change is completed, the old schema needs to be deleted after 10 minutes.\n\n Please wait for a while to make sure that the data of the old schema has been deleted, and then upgrade.\n Otherwise, the BE process will exit with an error when it is automatically detected when it starts.\n\n6. Run `python convert_row_to_column.py` again to verify that there are no rows in the table before upgrading.\n","tools/clickbench-tools/README.md":"\n\n# Usage\n\n**These scripts are used to do [ClickBench](https://benchmark.clickhouse.com/) test, more info [there](https://github.com/ClickHouse/ClickBench).**\n\n## follow the steps below:\n\n### 1. create table\n ./create-clickbench-table.sh\n\n### 2. load data\n ./load-clickbench-data.sh\n\n### 3. run queries\n ./run-clickbench-queries.sh\n","tools/tpch-tools/README.md":"# tpch-tools Documentation\n\nPath: `tools/tpch-tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/tools/tpch-tools/README.md)","be/src/io/tools/readme.md":"# tools Documentation\n\nPath: `be/src/io/tools/readme.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/be/src/io/tools/readme.md)","fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md":"# fe-authentication-plugin-ldap Documentation\n\nPath: `fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md)","fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md":"# fe-authentication-plugin-password Documentation\n\nPath: `fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md)","extension/DataX/doriswriter/src/main/resources/plugin.json":"# resources Plugin Manifest\n\nPath: `extension/DataX/doriswriter/src/main/resources/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/extension/DataX/doriswriter/src/main/resources/plugin.json)"},"files":{"tools/emr_storage_regression/README.md":"\n\n# Data Lake Regression Testing Tool For External Table\n\nUsed to test the doris external table on object storage for cloud vendors\n\n> Supported storage formats: HDFS, Alibaba Cloud OSS, Tencent Cloud COS, Huawei Cloud OBS\n\n> Supported data lake table formats: Iceberg\n\nThe following provides the example of the command line options:\n\n```\nsh tools/emr_storage_regression/emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\n```\nsh tools/emr_storage_regression/emr_tools.sh --case CASE --endpoint ENDPOINT --region REGION --service SERVICE --ak AK --sk SK --host HOST --user USER --port PORT\n```\n\nThe usage of each option is described below.\n\n## Connectivity Test\n\nWhen the `--case` option is set to `ping`, will check Doris's connectivity on EMR:\n\n- `--endpoint`, Object Storage Endpoint.\n\n- `--region`, Object Storage Region.\n\n- `--ak`, Object Storage Access Key.\n\n- `--sk`, Object Storage Secret Key.\n\n- `--host`, Doris Mysql Client IP.\n\n- `--user`, Doris Mysql Client Username.\n\n- `--port`, Doris Mysql Client Port.\n\n- `--service`, EMR cloud vendors: ali(Alibaba), hw(Huawei), tx(tencent).\n\n### Environment Variables\n\nNeed modify the environment variable in `default_emr_env.sh`, the script will execute `source default_emr_env.sh` to make the environment variable take effect.\n\nIf environment variables are configured, you can run the test script directly with the following command:\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\n### The Script Execution Steps For Connectivity Test\n\n1. Create Spark and Hive tables on EMR\n2. Use Spark and Hive command lines to insert sample data\n3. Doris creates the Catalog for connectivity test \n4. Execute SQL for connectivity test: `ping.sql`\n\n### Alibaba Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service` to `ali`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint oss-cn-beijing-internal.aliyuncs.com --region cn-beijing --service ali --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log\n```\n\nAlibaba Cloud EMR also supports testing connectivity for both Doris with DLF metadata and Doris on OSS-HDFS storage.\n\n- The DLF metadata connectivity test needs to be performed on the EMR cluster where the DLF serves as the metadata store, Default value of `DLF_ENDPOINT` is `datalake-vpc.cn-beijing.aliyuncs.com`, configured at ping_test/ping_poc.sh.\n\n- To test the OSS-HDFS storage connectivity, need to [enable the HDFS service on the OSS storage and configure](https://www.alibabacloud.com/help/en/e-mapreduce/latest/oss-hdfsnew), Default value of `JINDO_ENDPOINT` is `cn-beijing.oss-dls.aliyuncs.com`, configured at ping_test/ping_poc.sh.\n\n### Tencent Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service` to `tx`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint cos.ap-beijing.myqcloud.com --region ap-beijing --service tx --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log\n```\n\n### Huawei Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service`to `hw`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint obs.cn-north-4.myhuaweicloud.com --region cn-north-4 --service hw --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log \n```\n\n## Performance Testing on Standard Test Set\n\nWhen the `--case` option is set to `data_set`, will test the query performance of Doris external table:\n\n- `--test` test data set: ssb, ssb_flat, tpch, clickbench and all. Default `all`.\n\n- `--service`, EMR cloud vendors: ali(Alibaba), hw(Huawei), tx(tencent).\n\n- `--host`, Doris Mysql Client IP.\n\n- `--user`, Doris Mysql Client Username.\n\n- `--port`, Doris Mysql Client Port.\n\n### Environment Variables\n\nJust modify the above environment variable in `default_emr_env.sh`, the script will execute `source default_emr_env.sh` to make the environment variable take effect.\n\nIf environment variables are configured, you can run the test script directly with the following command:\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\n### Prepare Data\n\n1. To run the standard test set using the `emr_tools.sh` script, you need to rewrite the object storage bucket specified by the `BUCKET` variable, and then prepare data in advance and put them under the bucket. The script will generate table creation statements based on the bucket.\n\n2. Now the `emr_tools.sh` script supports iceberg, parquet and orc data for ssb, ssb_flat, tpch, clickbench.\n\n### Execution Steps\n\n1. After the connectivity test, the Doris Catalog corresponding to the standard test set is created\n2. Prepare the test set data based on the object storage bucket specified by the `BUCKET` variable\n3. Generate Spark table creation statements and create Spark object storage tables on EMR\n4. Create the spark table in the local HDFS directory: `hdfs:///benchmark-hdfs`\n5. You can choose to analyze Doris tables ahead of time and manually execute the statements in `analyze.sql` in the Doris Catalog\n6. Execute standard test set scripts: `run_standard_set.sh`\n\n### Standard data set: ssb, ssb_flat, tpch, clickbench\n\n- Full test. After executing the test command, Doris will run ssb, ssb_flat, tpch, clickbench tests in sequence, and the test results will include the cases on HDFS and on the object storage specified by `--service`.\n\n```\nsh emr_tools.sh --case data_set --service ali --host 127.0.0.1 --user root --port 9030 > log\n```\n\n- Specify a single test. `--test` option can be set to one of ssb, ssb_flat, tpch and clickbench.\n\n```\nsh emr_tools.sh --case data_set --test ssb --service ali --host 127.0.0.1 --user root --port 9030 > log\n```\n","tools/coffeebench-tools/README.md":"\n\n## Usage\n\nThese scripts are used to make coffee-bench test.\nfollow the steps below:\n\n### 1. create tables. modify `conf/doris-cluster.conf` to specify doris info, then run script below.\n\n ./bin/create-tables.sh\n\n### 2. load coffee-bench data. use -s for scale. \"500m/1b/5b\"\n\n ./bin/load-data.sh -s 500m\n\n **Note: The data loading process uses S3 data located in the US East (us-east-1) region.**\n If you cannot access this region due to network restrictions or other reasons, you can:\n 1. Use AWS CLI to copy the data to an S3 bucket in your preferred region\n 2. Download the data locally and use alternative methods to import it into Doris\n\n### 3. run queries.\n\n ./bin/run-queries.sh\n\n NOTICE: At present, we rewrite some queries in coffee bench to adapt to Doris' execution framework, but it does not affect the correctness of the results.\n",".claude/skills/code-review/SKILL.md":"---\nname: code-review\ndescription: Create consistent releases and changelogs\ncompatibility: opencode\n---\n\n## What I do\n\nComplete code review work to ensure code quality. Due to the length of content, you can read sections as needed based on the actual modifications being reviewed.\n\n## When to use me\n\nUse this when you need to review code, whether it's code you just completed or directly reviewing target code.\n\n---\n\n## Part 1: General Principles\n\n### 1.1 Core Invariants\n\nAlways focus on the following core invariants during review:\n1. **Data Correctness**: Data from any committed transaction must be visible to subsequent queries and not lost; data from uncommitted transactions must not be visible\n2. **Version Consistency**: Partition visible version is the sole standard for data visibility; BE must strictly read data not exceeding this version\n3. **Delete Bitmap Consistency** (MoW tables): delete bitmap must be strictly aligned with rowset versions; sentinel mark / TEMP_VERSION must be replaced with actual versions before use\n4. **Memory Safety**: All significant memory allocations must be tracked by MemTracker; orphan memory is not allowed\n5. **Error Means Failure**: Invariant violations should trigger exceptions or non-OK Status; silent continuation is never allowed\n\n### 1.2 Review Principles\n\n- **Defensive Programming Prohibited**: Do not use `if(valid)` style defensive checks to mask logic errors. If logically A=true should imply B=true, write `if(A) { DORIS_CHECK(B); ... }`, never `if(A && B)`\n- **Follow Context Conventions**: When adding code, strictly follow patterns in adjacent code—same error handling, same interface call order, same lock acquisition patterns, unless a clearly correct and more reasonable approach is identified.\n- **Prioritize Reuse**: Before adding new functionality, search for similar implementations that can be reused or extended; after adding, ensure good abstraction for shared code.\n- **Code First**: When this SKILL conflicts with actual code behavior, defer to your understanding of actual code behavior and explain\n- **Performance First**: All obviously redundant operations should be optimized away, all obvious performance optimizations must be applied, and obvious anti-patterns must be eliminated.\n\n### 1.3 Critical Checkpoints (Self-Review and Review Priority)\n\nThe following checkpoints must be **individually confirmed with conclusions** during self-review and review:\n\n- What is the goal of the current task? Does the current code accomplish this goal? Is there a test that proves it?\n- Is this modification as small, clear, and focused as possible?\n- Does the code segment involve concurrency? If yes:\n - Which threads introduce concurrency, what are the critical variables? What locks protect them?\n - Are operations within locks as lightweight as possible? Are all heavy operations outside locks while maintaining concurrency safety?\n - If multiple locks exist, is the locking order consistent? Is there deadlock risk?\n- Is there special or non-intuitive lifecycle management? If yes:\n - Understand the complete lifecycle and relationships of related variables. Are there circular references? When is each released? Can all lifecycles end normally?\n- Are configuration items added?\n - Should the configuration allow dynamic changes, and does it actually allow them? If yes:\n - Can affected processes detect the change promptly without restart?\n- Does it involve incompatible changes like function symbols or storage formats? If yes:\n - Is compatibility code added? Can it correctly handle requests during rolling upgrades?\n- Are there functionally parallel code paths to the modified one? If yes:\n - Should this modification be applied to other paths, and has it been?\n- Are there special conditional checks? If yes:\n - Does the condition have clear comments explaining why this check is necessary, simplest, and most viable?\n - Are there similar paths with similar issues?\n- What is the test coverage?\n - Are end-to-end functional tests comprehensive?\n - Are there comprehensive negative test cases from various angles?\n - For complex features, are key functions sufficiently modular with unit tests?\n- Does the feature need increased observability? If yes:\n - When bugs occur, are existing logs and VLOGs sufficient to investigate key issues?\n - Are INFO logs lightweight enough?\n - Are Metrics added for critical paths? Referencing similar features, are all necessary observability values covered?\n- Does it involve transaction and persistence-related modifications? If yes:\n - Are all EditLog reads and writes correctly covered? Is all information requiring persistent storage persisted? Does it follow our standard approach?\n - In transaction processing logic, can master failover at any time point be handled correctly?\n- Does it involve data writes and modifications? If yes:\n - Are transactionality and atomicity guaranteed? Are there issues with concurrency?\n - Would FE or BE crashes cause leaks, deadlocks, or incorrect data?\n- Are new variables added that need to be passed between FE and BE? If yes:\n - Is variable passing modified in all paths? For example, various scattered thrift sending points like constant folding, point queries.\n- Analyze deeply from all angles (including but not limited to time complexity, anti-patterns, CPU and memory friendliness, redundant calculations, etc.) based on the code context and available information. Are there any performance issues or optimization opportunities?\n- Based on your understanding, are there any other issues with the current modification?\n\nAfter checking all the above items with code. Use the remaining parts of this skill as needed. The following content does not need individual responses; just check during the review process.\n\n#### 1.3.1 Concurrency and Thread Safety (Highest Priority)\n\n- [ ] **New thread/bthread entry**: Is `SCOPED_ATTACH_TASK` used? Omission causes memory to be counted in orphan tracker, debug builds crash directly\n- [ ] **Lock protection scope**: Is shared data (version map, delete bitmap, transaction state) accessed within correct locks? Is lock type correct (shared vs exclusive)?\n- [ ] **Lock order**: Are there nested acquisitions of multiple locks? Must follow known lock order (e.g., `_txn_lock` -> `_txn_map_lock`), otherwise deadlock\n- [ ] **Atomic operation memory order**: Use `relaxed` for statistics, at least `acquire/release` for flag/signal\n- [ ] **COW column modification**: Is exclusive ownership ensured before `mutate()`? Is `assume_mutable_ref()` only used when exclusive ownership is confirmed?\n\n#### 1.3.2 Error Handling (Highest Priority)\n\n- [ ] **Status checking**: `Status` return values **must** be checked, `static_cast(...)` silent discard is prohibited\n- [ ] **Exception propagation chain**: Is there `RETURN_IF_CATCH_EXCEPTION` above `THROW_IF_ERROR` to catch?\n- [ ] **Error messages**: Are they clear, include context (table/partition/TabletID/transaction ID), provide solution direction?\n- [ ] **DORIS_CHECK usage**: Only for invariant assertions (e.g., `DORIS_CHECK(hash_table != nullptr)`), not for defensive programming\n\n#### 1.3.3 Memory Safety (High Priority)\n\n- [ ] **Large memory reservation**: Is `try_reserve()` called before allocation? Is there `DEFER_RELEASE_RESERVED()` to ensure release?\n- [ ] **Cache Handle**: Is `cache->release(handle)` called after using `Cache::Handle*`?\n- [ ] **shared_ptr circular references**: Do scenarios like `Dependency` use raw pointers or `weak_ptr` to break cycles?\n- [ ] **Object creation**: Do classes with `ENABLE_FACTORY_CREATOR` use `create_shared()` / `create_unique()`? Raw `new` is prohibited\n\n#### 1.3.4 Data Correctness (High Priority)\n\n- [ ] **Delete Bitmap version**: In Cloud mode, is `TEMP_VERSION_COMMON` manually replaced before use?\n- [ ] **Version consistency**: Does data reading strictly not exceed Partition visible version?\n- [ ] **MoW setting**: Is `SegmentWriterOptions::enable_unique_key_merge_on_write` correctly set to `true`?\n- [ ] **EditLog persistence**: Do metadata modifications call `editLog.logXxx()`? Is replay logic strictly equivalent to master?\n\n#### 1.3.5 Observability (Medium Priority)\n\n- [ ] **Metrics addition**: Are `ADD_TIMER/ADD_COUNTER` added to critical paths? Does naming follow `module_xxx` convention?\n- [ ] **Log levels**: Use `LOG(ERROR)` for error logs, `LOG(INFO)` for critical paths, `VLOG` or `DVLOG` for debug info\n- [ ] **Trace information**: Do distributed operations (RPC, transactions) include trace ID for tracking?\n\n### 1.4 Test Coverage Principles\n\n- All kernel features **must** have corresponding tests\n- End-to-end SQL functionality should prioritize adding regression tests under `regression-test/` (Groovy DSL)\n- Also add BE unit tests (`be/test/`, Google Test) and FE unit tests (`fe/fe-core/src/test/`, JUnit 5) where possible. For complex features, must be modular with unit tests.\n- Regression test result files (`.out`) **must not be handwritten**, must be auto-generated via `-genOut`\n- Regression tests must use `order_qt_` prefix or manual `ORDER BY` to ensure ordered results\n- Expected error cases use `test { sql \"...\"; exception \"...\" }` pattern\n- Tables should be **DROPped before use** (not after tests end), to preserve environment for debugging\n- Simple tests should hardcode table names directly, not use `def tableName`\n\n---\n\n## Part 2: BE Module Review Guide\n\n### 2.1 Error Handling (Highest Priority)\n\n#### 2.1.1 Status / Exception / Result\\ Relationship\n\n| Type | Usage | Propagation Macro |\n|------|-------|------------------|\n| `Status` | Default error return type, `[[nodiscard]]` | `RETURN_IF_ERROR` |\n| `Exception` | Vectorized execution engine hot path, expression evaluation | `RETURN_IF_CATCH_EXCEPTION` |\n| `Result` | Return value or error (replaces old output parameter pattern) | `DORIS_TRY` |\n\n**Review Checkpoints:**\n\n- [ ] Are `Status` return values checked? Search for `static_cast(...)` pattern—this is the **most dangerous anti-pattern**, silently swallowing errors without logging. Currently 171+ occurrences in codebase. Never allow this pattern in new code\n- [ ] Where `THROW_IF_ERROR` is used, is there `RETURN_IF_CATCH_EXCEPTION` or `RETURN_IF_ERROR_OR_CATCH_EXCEPTION` above in the call chain to catch? If not, exceptions propagate uncaught causing crashes. Current version has related catch in upper layers when Pipeline executes specific operators.\n- [ ] **`THROW_IF_ERROR` prohibited in Defer/RAII destructor lambdas**: Throwing exceptions during stack unwinding triggers `std::terminate`. Known issues: `pipeline_task.cpp:412-413, 638-639`. Use `WARN_IF_ERROR` or `static_cast` + logging in Defer\n- [ ] **`THROW_IF_ERROR` prohibited in Thrift RPC handlers without try/catch protection**: Thrift framework doesn't catch Doris exceptions, causing connection drops. Known issues: `backend_service.cpp:1323,1326`. RPC handlers should use `RETURN_IF_ERROR` pattern or outermost try/catch\n- [ ] Is `WARN_IF_ERROR` usage reasonable? Only allowed in destructors, cleanup paths, best-effort operations. Warning message must not be empty string\n- [ ] Is `DORIS_CHECK` used for **invariant assertions** (not defensive programming)? Correct usage example: `DORIS_CHECK(hash_table)` asserts hash table is initialized\n- [ ] Do new catch blocks correctly convert exceptions? Should follow standard pattern from `internal_service.cpp:348-357`: catch `doris::Exception` first, then `std::exception`, finally `...`\n\n#### 2.1.2 Cloud RPC Error Handling\n\nThe `retry_rpc` template in `cloud_meta_mgr.cpp` is the standard pattern for RPC error handling in Cloud mode:\n- [ ] Do new Cloud RPC calls use `retry_rpc`?\n- [ ] Does `INVALID_ARGUMENT` return directly without retry? (Correct behavior)\n- [ ] Does `KV_TXN_CONFLICT` use an independent conflict retry counter?\n- [ ] Do timeout and retry counts use config variables instead of hardcoding?\n\n#### 2.1.3 compile_check Mechanism\n\n`compile_check_begin.h` / `compile_check_end.h` elevate `-Wconversion` to **compile error** (excluding sign-conversion and float-conversion).\n\n- [ ] Do new `.h` files include paired `compile_check_begin.h` / `compile_check_end.h` in declaration regions?\n- [ ] **Coverage status**: Currently only 31.1% (1,049/3,372) of source files use compile_check. Additionally, 218 files contain `compile_check_begin.h` but lack corresponding `compile_check_end.h` (unpaired). New files should always include complete pairing\n- [ ] Are there implicit narrowing conversions? Pay special attention to `int64_t -> int32_t`, `size_t -> int`, `uint64_t -> int64_t`\n- [ ] If bypassing due to third-party code is necessary, are `compile_check_avoid_begin.h` / `compile_check_avoid_end.h` used?\n\n### 2.2 Memory Management\n\n#### 2.2.1 MemTracker Hierarchy\n\nDoris uses two-level memory tracking:\n- **`MemTrackerLimiter`**: Heavy-weight tracker with limits (QUERY / LOAD / COMPACTION / SCHEMA_CHANGE / METADATA / CACHE types)\n- **`MemTracker`**: Lightweight tracker without limits, for fine-grained sub-component tracking\n\nThread-level tracking is implemented via `ThreadMemTrackerMgr`, each thread has one active `MemTrackerLimiter` and a `MemTracker` stack.\n\n**Review Checkpoints:**\n\n- [ ] Do newly started threads/bthreads use `SCOPED_ATTACH_TASK` at entry? Without it, memory is counted in orphan tracker, debug builds trigger assertions\n- [ ] Does temporary tracker switching use `SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER` (supports nesting)?\n- [ ] **`ThreadPool::submit_func` callbacks**: Do lambdas submitted to thread pools use `SCOPED_ATTACH_TASK` internally? Currently 87+ `submit_func` calls in codebase, many rely on \"indirect attach\" (attach happens deep in call chain), which is fragile. Correct approach references gold-standard wrapper pattern from `calc_delete_bitmap_executor.h:67-83`: explicit attach at callback entry. Known problematic files: `s3_file_bufferpool.cpp:114`, `wal_manager.cpp:414`, `download_action.cpp:166`\n- [ ] Is `try_reserve()` called before large memory allocation? Is there `DEFER_RELEASE_RESERVED()` to ensure release after allocation?\n- [ ] Does memory hook path (`ThreadMemTrackerMgr::consume`) avoid operations that may trigger memory allocation (e.g., LOG, string formatting)? `_stop_consume` flag exists to prevent recursion\n\n#### 2.2.2 Object Lifecycle\n\n- [ ] Are classes using `ENABLE_FACTORY_CREATOR` created via `create_shared()` / `create_unique()`? Raw `new` is prohibited\n- [ ] Are there `shared_ptr` circular references? Standard ways to break cycles:\n - `Dependency::_shared_state` uses **raw pointer** (non-owning)\n - `Dependency::_blocked_task` uses `vector>`\n - `TrackerLimiterGroup::trackers` uses `list>`\n- [ ] Is Rowset's dual reference counting used correctly? `acquire()` / `release()` is manual reference counting (for performance), `shared_ptr` manages ownership. When `release()` decrements to 0 and in UNLOADING state, triggers `do_close()`\n- [ ] Are rowsets in `_unused_rowsets` judged deletable via `use_count() == 1`?\n- [ ] Are there suspicious `shared_ptr` circular reference issues? For clear lifecycles, prefer `unique_ptr` + raw pointer observer pattern.\n\n#### 2.2.3 COW Column Semantics\n\nVectorized columns (`IColumn`) use custom intrusive reference-counted Copy-on-Write mechanism:\n\n- [ ] Is exclusive ownership ensured when calling `mutate()`? When `use_count() > 1`, triggers deep copy, which is a performance hotspot\n- [ ] Is `assume_mutable_ref()` only used when exclusive ownership is confirmed? This method **does not check reference count**, using it in shared state causes data corruption\n- [ ] Is `set_columns()` called to restore after `Block::mutate_columns()`? After calling, original Block's column pointers are null\n- [ ] Can the pointer returned by `convert_to_full_column_if_const()` share with original column? For `ColumnConst` it materializes a new column, but for normal columns returns shared pointer\n\n#### 2.2.4 Cache Lifecycle\n\n- [ ] Is `cache->release(handle)` called after using `Cache::Handle*`? Omission causes memory leaks\n- [ ] Do cache values inherit from `LRUCacheValueBase`? This base class automatically releases tracking bytes on destruction\n- [ ] Are new cache types registered with `CacheManager`? Unregistered caches don't participate in global GC\n\n### 2.3 Concurrency and Locks\n\n#### 2.3.1 Tablet Lock Hierarchy\n\nThe `Tablet` class has 8+ independent locks, with confirmed usage rules:\n\n| Lock | Type | Rule |\n|----|------|------|\n| `_meta_lock` | `shared_mutex` | Modifying version map requires EXCLUSIVE; reading requires SHARED |\n| `_rowset_update_lock` | `mutex` | Serializes delete bitmap updates during MoW table publish_txn, **independent of _meta_lock** (avoids blocking read path) |\n| `_base_compaction_lock` | `mutex` | Serializes base compaction |\n| `_cumulative_compaction_lock` | `mutex` | Serializes cumulative compaction |\n| `_migration_lock` | `shared_timed_mutex` | Protects tablet migration |\n| `_cooldown_conf_lock` | `shared_mutex` | Protects cooldown configuration |\n\n**Review Checkpoints:**\n\n- [ ] Do operations on `_rs_version_map` or `_stale_rs_version_map` hold `_meta_lock` (shared or exclusive)?\n- [ ] Do `TabletMeta` modification operations hold the owning Tablet's EXCLUSIVE `_meta_lock`?\n- [ ] Is there acquisition of other tablet locks while holding `_meta_lock`? This may cause deadlock\n\n#### 2.3.2 TxnManager Lock Order\n\nConfirmed consistent lock order: `_txn_lock` -> `_txn_map_lock` (applies to commit, publish, delete, rollback).\n- [ ] Do new TxnManager operations follow this lock order?\n\n#### 2.3.3 Pipeline Dependency Concurrency\n\n`Dependency` uses double-checked locking + atomic `_ready` flag + mutex `_task_lock`:\n- `set_ready()`: First check `_ready` (fast path), then lock to set `_ready = true` and wake up\n- `is_blocked_by()`: Lock, check `_ready`, add to wait list if not ready\n\n- [ ] Do new `Dependency` subclasses correctly call `block()` / `set_ready()`?\n- [ ] Do `CountedFinishDependency`'s `add()` / `sub()` operate under `_mtx` protection?\n\n#### 2.3.4 Atomic Operation Memory Order\n\nMemory order usage patterns in codebase:\n- `memory_order_relaxed`: Statistics counters (correct)\n- `memory_order_acquire/release`: Lifecycle flags like `ExecEnv::_s_ready` (correct)\n- `memory_order_acq_rel`: CAS close operations (correct)\n\n- [ ] Do new atomic operations use appropriate memory order? Use relaxed for statistics counters, at least acquire/release for flag/signal\n- [ ] Does `std::atomic` as stop signal use at least acquire/release? Relaxed may theoretically delay propagation\n\n### 2.4 Type System and Serialization\n\n#### 2.4.1 Upgrade/Downgrade Compatibility\n\nUpgrade/downgrade compatibility no longer uses be_exec_version for control. Instead, each feature (e.g., functions, compression algorithms, etc.) adds its own option for confirmation. BE uses is_set&&is_true to branch or replace functionality at appropriate nodes.\n\n#### 2.4.2 Decimal Precision Overflow\n\n- [ ] Do Decimal operation results check precision upper limit? `decimal_result_type()` may promote result to DECIMAL256, causing unexpected memory layout changes\n- [ ] Are DecimalV2's `original_precision` / `original_scale` correctly set? Default value `UINT32_MAX` means \"unknown\", `get_format_scale()` returns different value from `scale` in this case\n\n#### 2.4.3 Thrift Type Mapping\n\n- [ ] Are `TScalarType` optional fields (`precision`, `scale`, `len`) set when needed? DECIMAL needs precision/scale, CHAR/VARCHAR needs len, DATETIMEV2 needs scale\n- [ ] Is depth-first flattened representation of `TTypeDesc.types` correctly traversed? Off-by-one errors are common with nested complex types (ARRAY>)\n\n#### 2.4.4 Block Merge Nullable Handling\n\n`MutableBlock::merge_impl()` assumes target is Nullable and source is non-Nullable when types mismatch, using C-style cast `(DataTypeNullable*)` instead of `dynamic_cast`.\n\n- [ ] Does new Block merge logic correctly handle Nullable promotion? DCHECK only protects in debug builds\n\n### 2.5 Pipeline Execution Engine\n\n#### 2.5.1 Operator Lifecycle\n\nPipeline Task state machine: `INITED -> RUNNABLE -> BLOCKED -> FINISHED -> FINALIZED`\n\n- [ ] Are `SharedState`'s source_deps and sink_deps correctly connected? `inject_shared_state()` is responsible for connection\n\n#### 2.5.2 Memory Reservation and Spill\n\n- [ ] Do memory-intensive operations (Hash Join build, aggregation) use `_try_to_reserve_memory()`?\n- [ ] Is there `_memory_sufficient_dependency` for backpressure when memory is insufficient?\n- [ ] Does `revoke_memory()` correctly implement spill logic?\n\n### 2.6 Storage Engine\n\n#### 2.6.1 Rowset Version Management\n\n- [ ] Are `add_rowset()` and `modify_rowsets()` executed under exclusive `_meta_lock`?\n- [ ] Are version ranges continuous? There should be no version gaps in `_rs_version_map`\n- [ ] Does compaction output rowset's version range correctly cover input rowsets?\n\n#### 2.6.2 Delete Bitmap (MoW Tables, Extremely High Priority)\n\n- [ ] In Cloud mode, is **TEMP_VERSION_COMMON manually replaced** for bitmaps obtained from `CloudTxnDeleteBitmapCache`? Code comments explicitly warn about this issue (`cloud_txn_delete_bitmap_cache.h:72-75`). Omission causes bitmap to be applied to wrong version\n- [ ] Is `_rowset_update_lock` held during delete bitmap calculation? Prevents concurrent calculation on same (txn_id, tablet_id)\n- [ ] Does compaction's delete bitmap calculation use **latest** compaction stats (`ms_base_compaction_cnt`, etc.)? Stale stats will miss recently compacted rowsets\n\n#### 2.6.3 Segment Writing\n\n- [ ] Is `SegmentWriterOptions::enable_unique_key_merge_on_write` correctly set for MoW tables? Default value is `false`, omission causes deduplication to be silently skipped\n- [ ] Is performance impact of row-stored columns (`_serialize_block_to_row_column`) on wide tables within expectations?\n\n### 2.7 IO Subsystem\n\n- [ ] Do `FileSystem` operations use `FILESYSTEM_M` macro to handle bthread async IO?\n- [ ] Do remote storage (S3/HDFS) reads use `create_cached_file_reader()` for file caching?\n\n### 2.8 Inverted Index Subsystem\n\n#### 2.8.1 Lifecycle and Destruction Order\n\n- [ ] **Field declaration order is destruction order**: In `inverted_index_writer.h:92-95`, `_dir` must be declared **before** `_index_writer`, because `_index_writer`'s `write.lock` is created by `_dir.lockFactory`, C++ destructs in reverse declaration order. New fields must follow this constraint\n- [ ] **Cache handle destruction extends LRU retention**: `inverted_index_cache.h:142-152`, `InvertedIndexCacheHandle` destructor sets `last_visit_time` to future timestamp (`UnixMillis() + config::index_cache_entry_stay_time_after_lookup_s * 1000`), making actually queried searchers survive longer in LRU. Be aware of this extension mechanism when modifying cache logic\n\n#### 2.8.2 CLucene Exception Handling\n\nCLucene doesn't use C++ exceptions, `inverted_index_common.h:65-111` implements `ErrorContext` + FINALLY macro system:\n- `FINALLY`: Catches exceptions and returns `Status` error\n- `FINALLY_EXCEPTION`: Catches and rethrows\n- All three macros use `static_assert` to enforce existence of `error_context` local variable in scope\n\n- [ ] Does new CLucene-related code use `ErrorContext` + `FINALLY` pattern instead of raw try/catch?\n\n#### 2.8.3 Three-Valued Logic Bitmap\n\n`InvertedIndexResultBitmap` (`inverted_index_reader.h:80-214`) implements SQL three-valued logic (TRUE/FALSE/NULL), storing separately via `_data_bitmap` + `_null_bitmap`:\n- [ ] **`op_not` is const but actually modifies data**: Modifies pointed object via `shared_ptr` to bypass const qualifier. Callers should not assume const reference returned results are immutable\n- [ ] `operator|=` implements complete SQL three-valued OR semantics (lines 148-157), same for `operator&=` and `operator-=`. New bitmap operations must follow three-valued logic\n\n### 2.9 Schema Change Subsystem\n\n#### 2.9.1 Three-Strategy Selection and Lock Order\n\n`_get_sc_procedure()` (`schema_change.h:303-315`) selects among three strategies based on change type:\n- `LinkedSchemaChange`: Column name/type alias changes only, zero-copy\n- `VSchemaChangeDirectly`: Direct conversion without sorting\n- `VLocalSchemaChangeWithSorting`: Full rewrite with sorting\n\n**Lock acquisition order** (`schema_change.cpp:972-976`): `base_tablet push_lock → new_tablet push_lock → base_tablet header_lock → new_tablet header_lock`\n\n- [ ] Do schema change lock acquisitions follow this four-level order? Out-of-order causes deadlock\n\n#### 2.9.2 MOW Delete Bitmap Four-Step Flow (Extremely High Priority)\n\n`schema_change.cpp:1595-1657` describes critical correctness flow for MOW table schema change:\n1. Newly imported rowsets during dual-write **skip** delete bitmap calculation\n2. After conversion completes, calculate delete bitmap for rowsets during dual-write period (lines 1618-1634)\n3. **Block new publish**, calculate delete bitmap for incremental rowsets (lines 1636-1652)\n4. Switch tablet state to `TABLET_RUNNING` (lines 1653-1656)\n\n- [ ] When modifying schema change delete bitmap path, are the four-step order and blocking logic complete? Step 3's blocking is key to ensuring consistency\n\n#### 2.9.3 Column Deletion Order Trap\n\n`return_columns` is calculated **before** `merge_dropped_columns()` (`schema_change.cpp:959-964` vs line 1062), this is intentional: `return_columns` only covers non-dropped columns, dropped columns are handled internally by `SegmentIterator` for delete predicate evaluation.\n\n- [ ] When modifying schema change column processing logic, is the timing of `return_columns` and dropped columns calculation understood?\n- [ ] `_check_row_nums` can be bypassed by `config::ignore_schema_change_check` (default false) (`schema_change.h:155-171`), this config should remain off in production\n\n### 2.10 Workload Group Memory Tracking\n\n`WorkloadGroup` (`workload_group.h`) uses dual-lock pattern: `_mutex` protects name/version/memory limit/resource_ctxs/shutdown state, `_task_sched_lock` protects scheduler and cgroup pointers.\n\n- [ ] **Memory tracking inconsistency**: `check_mem_used()` (lines 115-123) calculates `_total_mem_used + _wg_refresh_interval_memory_growth` for watermark check, but `exceed_limit()` (lines 132-135) **only compares `_total_mem_used > _memory_limit`**, ignoring `_wg_refresh_interval_memory_growth`. This means during refresh interval, workload group may exceed hard limit without being detected by `exceed_limit()`\n- [ ] Does new memory check logic use the correct method? If precise limit checking is needed, should reference `check_mem_used()` approach\n\n---\n\n## Part 3: FE Module Review Guide\n\n### 3.1 Metadata Persistence (EditLog)\n\n#### 3.1.1 Write Path\n\nAll metadata modifications must go through EditLog persistence. Flow:\n1. Modify in-memory state\n2. Call `editLog.logXxx()` to write log\n3. Log is replicated to Follower via BDBJE\n4. Caller blocks until persistence confirmation\n\n**Review Checkpoints:**\n\n- [ ] Do new DDL/DML operations call corresponding `editLog.logXxx()` methods? Omission causes Follower state inconsistency\n- [ ] Do new OperationTypes have corresponding replay handling in `EditLog.loadJournal()` switch?\n- [ ] Is replay logic **strictly equivalent** to master execution logic? Any difference causes Follower state drift\n- [ ] Do new serializable objects correctly implement Gson serialization? Are `@SerializedName` annotations complete?\n- [ ] Can Image checkpoint correctly include new metadata?\n- [ ] Can master failover at any moment guarantee no duplicate or lost data? Short-term task failures are tolerable, but memory/transaction leaks and data inconsistencies are not allowed.\n\n#### 3.1.2 EditLog Batch and Outside-Lock Commit Mode\n\n`DatabaseTransactionMgr` implements two EditLog commit modes, controlled by `Config.enable_txn_log_outside_lock`:\n\n- **Inside-lock mode** (default): Directly calls `editLog.logInsertTransactionState()` within transaction lock\n- **Outside-lock mode**: Only modifies in-memory state and calls `submitEdit()` to submit to queue within lock, waits for persistence completion via `awaitTransactionState()` outside lock (`DatabaseTransactionMgr.java:383-395`)\n\nFuzzy testing (`DorisFE.java:612-615`) randomly switches between these two modes, so both paths must be correct.\n\n**Review Checkpoints:**\n\n- [ ] Do modifications involving transaction EditLog consider both inside-lock and outside-lock commit paths?\n- [ ] In outside-lock mode, are `awaitTransactionState()` timeout and exception handling correct? Timeout doesn't mean failure—transaction may still be committing\n- [ ] Do new transaction state modifications complete all in-memory state changes before `submitEdit()` submission? After submission, other threads may immediately see new state\n\n#### 3.1.3 EditLog Fatal Error Handling\n\nWhen EditLog flush fails, FE calls `System.exit(-1)` to terminate process—this is **intentional**, preventing split-brain.\n- [ ] Does new code catch and swallow exceptions in EditLog-related paths? This may cause metadata loss\n\n### 3.2 Lock Hierarchy and Concurrency\n\n#### 3.2.1 Confirmed Lock Orders\n\n| Scenario | Lock Order |\n|------|------|\n| ConsistencyChecker | `jobs_lock` -> `synchronized(job)` -> `db_lock` |\n| AlterJobV2 | `synchronized(job)` -> `db_lock` |\n| TabletScheduler | `synchronized(this)` **internal** operations, then **release**, then acquire `db_lock` (two-step method avoids deadlock) |\n| DatabaseTransactionMgr | Its RWLock is **leaf lock**, should not acquire other locks internally |\n| MTMVTask | Acquire table locks after sorting by table ID (avoids deadlock) |\n\n**Review Checkpoints:**\n\n- [ ] Do new lock acquisitions follow the above lock orders?\n- [ ] Is `tryLock(timeout)` used instead of infinite-wait `lock()`? FE's standard pattern is `tryLock` + timeout logging\n- [ ] **Lock timeout standard values**: Env lock = 5s, DB write lock = 100s, Table read lock = 1min. Operations involving multiple tables must lock in ascending table ID order (implemented via `PriorityQueue`) to prevent deadlock\n- [ ] Does `Database.tryWriteLock()` log current owner thread on timeout?\n- [ ] Are there paths acquiring database lock within `synchronized` block? This is a known deadlock risk, should use two-step method\n\n#### 3.2.2 ConcurrentModificationException\n\n- [ ] Is traversal of shared collections under lock protection? `CatalogRecycleBin.write()` added `synchronized` because dump image API concurrent access caused CME\n- [ ] Can there be concurrent modifications during traversal of `Map` / `List`? Prefer `ConcurrentHashMap` or snapshot before traversal\n\n#### 3.2.3 Known Deadlock Cases (New Code Must Avoid Similar Patterns)\n\nThe following are confirmed deadlock risk points in codebase, be extra vigilant when reviewing similar patterns:\n\n| Location | Pattern | Description |\n|------|------|------|\n| `MasterCatalogExecutor.java:70-78` | Initiating RPC while holding catalog lock | RPC timeout blocks catalog operations for long time |\n| `ExternalDatabase.java:137-158` | Nested acquisition of db lock + makeSureInitialized() | Comments explicitly mark deadlock avoidance strategy |\n| `HiveMetaStoreCache.java:126-130` | Acquiring cache lock in thread pool callback | Comment marks \"this may cause a deadlock\" |\n| `CacheFactory.java:85-92` | Circular dependency in cache initialization | Avoided via lazy initialization |\n\n- [ ] Does new code initiate RPC or IO operations while holding locks? This is the most common source of deadlock and performance issues\n- [ ] Does new code acquire locks in callbacks (e.g., thread pool, cache loader) that callers may hold?\n\n### 3.3 Nereids Optimizer\n\n#### 3.3.1 Rule Implementation Specifications\n\n**Review Checkpoints:**\n\n- [ ] Is new Rule's `RuleType` added to `RuleType.java` enum?\n- [ ] Does Rule correctly handle **all Join types**? Pay special attention to ASOF variants (`ASOF_LEFT_INNER_JOIN`, etc.). `COULD_PUSH_THROUGH_LEFT/RIGHT` lists in `PushDownFilterThroughJoin` explicitly include ASOF types\n- [ ] Are constant predicates (`slots.isEmpty()`) correctly handled? Inner Join can push to both sides, Outer Join cannot\n- [ ] Is there **infinite loop risk**? Can created nodes be matched again by another rule in same batch? Code comments have multiple \"separate topDown passes to avoid dead loops\" warnings\n- [ ] Does `OneRewriteRuleFactory` pattern match correct wrapper nodes? For example, `EliminateJoinByFK` only matches `project(join)` not bare `join`\n- [ ] Are predicates checked for `containsUniqueFunction()`? Predicates containing unique functions cannot be pushed down\n\n#### 3.3.2 Rewriter Stage Order\n\nRewriter organizes rules into multi-stage pipeline, key dependencies:\n- `PUSH_DOWN_FILTERS` is called 5-6 times throughout pipeline, because it interacts with many other transformations\n- `InferPredicates` is called twice (with `PUSH_DOWN_FILTERS` in between), because eliminating outer join exposes new inference opportunities\n- `costBased(...)` rules are controlled by `runCboRules` flag\n\n- [ ] Is new rewrite rule placed in correct stage? For example, rules needing predicate pushdown should be after `PUSH_DOWN_FILTERS`\n- [ ] Is new exploration rule registered in `RuleSet`?\n\n#### 3.3.3 Property Derivation\n\n`RequestPropertyDeriver` implements top-down property requests:\n\n- [ ] Does new Physical Operator have visit method in `RequestPropertyDeriver`?\n- [ ] Are multiple alternative property requirements provided (e.g., HashJoin provides both shuffle and broadcast choices)? Single requirement narrows search space\n- [ ] Is `PhysicalProject` alias penetration logic correct? Only `Alias(SlotRef)` and bare `SlotRef` can penetrate\n\n### 3.4 Transaction Management\n\n#### 3.4.1 Transaction Lifecycle\n\nState machine: `PREPARE -> COMMITTED -> VISIBLE` (or `ABORTED`)\n\n- [ ] After transaction commit, can `PublishVersionDaemon` correctly send `PublishVersionTask` to all BEs?\n- [ ] Does publish wait for quorum confirmation?\n- [ ] Does transaction correctly abort after timeout?\n\n**Known race condition—`TransactionState.tableIdList`**: At `TransactionState.java:632`, TODO comment marks that `addTableId()` / `removeTableId()` lack synchronization protection. `tableIdList` is plain `ArrayList`, has race condition (CME or data loss) when multiple threads concurrently add/remove table IDs.\n\n- [ ] Are operations on `TransactionState.tableIdList` under external lock protection? Or should it be replaced with thread-safe collection?\n\n#### 3.4.2 Cloud Mode Differences\n\nIn Cloud mode, transactions are centrally managed by Meta-Service, not FE:\n- [ ] Do Cloud mode transaction operations use `CloudGlobalTransactionMgr` instead of local `GlobalTransactionMgr`?\n- [ ] **Unsafe downcasting prohibited**: Current code has 4 places forcibly casting `GlobalTransactionMgrIface` to `CloudGlobalTransactionMgr` (e.g., `InternalCatalog.java:994`). These casts cause ClassCastException in non-Cloud mode. New code should add required methods by extending `GlobalTransactionMgrIface` interface, not rely on downcasting\n- [ ] Does version retrieval use `getVisibleVersion()` (RPC + cache in Cloud mode)?\n- [ ] Is `getVisibleVersion()`'s `RpcException` correctly handled? Multiple callers catch and return 0, may cause MV refresh misjudgment\n\n### 3.5 Exception System\n\n```\nUserException (checked)\n ├── AnalysisException -- SQL analysis error\n ├── DdlException -- DDL execution failure\n ├── MetaNotFoundException -- Metadata lookup failure\n ├── LoadException -- Load failure\n └── ...\n\nNereidsException (unchecked, RuntimeException)\n ├── nereids/AnalysisException -- Nereids-specific, different from common/\n └── nereids/ParseException\n```\n\n- [ ] **Note Nereids has its own `AnalysisException`** (`nereids/exceptions/AnalysisException.java`, extends RuntimeException), which is a **different class** from `common/AnalysisException` (checked)\n- [ ] Does `ErrorReport.reportAnalysisException()` use correct `ErrorCode`? Custom Doris error codes start from 5000\n- [ ] Does new RPC error handling use `Status` pattern (`TStatusCode` / `PStatus`)?\n\n### 3.6 OlapTable Version Visibility\n\n- [ ] `getVisibleVersion()` behaves differently in Cloud vs non-Cloud mode:\n - Non-Cloud: Returns local `tableAttributes.getVisibleVersion()`\n - Cloud: RPC to meta-service, has cache TTL\n- [ ] Can Cloud mode's version cache TTL (`cloudTableVersionCacheTtlMs`) cause reading stale versions?\n- [ ] `VERSION_NOT_FOUND` is forcibly converted to version=1 (`PARTITION_INIT_VERSION`)—cannot distinguish \"never loaded\" from \"loaded to version 1\"\n\n### 3.7 External Catalog Concurrency Traps\n\n#### 3.7.1 Initialization Race Condition\n\n`ExternalCatalog.makeSureInitialized()` (`ExternalCatalog.java:354-379`) is synchronized, but has the following traps:\n\n- [ ] **`isInitializing` guard is dead code in `ExternalCatalog`**: Although code checks `isInitializing` (line 358), it's never set to `true` (only set to `false` in finally). Compare with `ExternalDatabase.makeSureInitialized()` (`ExternalDatabase.java:137-158`) which **correctly** sets `isInitializing = true`. This means ExternalCatalog has no reentrancy protection\n- [ ] **`lowerCaseToDatabaseName.clear()` race window**: `getFilteredDatabaseNames()` (line 504) first `clear()` then refills `ConcurrentMap`, this method is **not synchronized**. Between clear and refill completion, concurrent `get()` calls return null\n\n#### 3.7.2 resetToUninitialized() NPE Risk\n\n`resetToUninitialized()` (`ExternalCatalog.java:581-590`) sets `objectCreated` and `initialized` to false within synchronized block, then calls `onClose()` (lines 773-784) to set `executionAuthenticator` and `transactionManager` to null.\n\n- [ ] Concurrent queries that have passed `makeSureInitialized()` may be using these fields being set to null. synchronized only protects callers of other synchronized methods, not non-synchronized read paths already in execution\n\n### 3.8 Backup/Restore State Machine\n\n#### 3.8.1 BackupJob State Machine and EditLog Timing\n\nEditLog write timing in `BackupJob` state machine is **uneven**, this is the most important non-obvious convention:\n\n- [ ] **PENDING→SNAPSHOTING doesn't write EditLog** (`BackupJob.java:620-624`): After FE restart, state reverts to PENDING, all snapshot tasks are regenerated and resent. Don't add EditLog when modifying this transition—restart recovery relies on this behavior\n- [ ] **BarrierLog mechanism**: `prepareSnapshotTaskForOlapTableWithoutLock()` creates `BarrierLog` for each table and writes via `editLog.logBarrier()` (lines 648-657), returned `commitSeq` is stored in properties map, used to ensure snapshot consistency\n- [ ] **UPLOAD_INFO stage errors only retry, not cancel** (`BackupJob.java:509-514`): This is the final stage, code comments explicitly state continuous retry until timeout even when encountering unrecoverable errors. Don't call `cancelInternal()` in this stage when modifying error handling\n\n#### 3.8.2 RestoreJob state vs showState\n\n`RestoreJob` has two independent state fields (`RestoreJob.java:157,162`):\n- `state`: Serialized persistent actual state, master sets to FINISHED **before** writing EditLog\n- `showState`: Non-serialized, only for display, master sets to FINISHED **after** writing EditLog (line 2191)\n\n- [ ] This prevents follower from seeing FINISHED state before EditLog replication completes. After deserialization, `gsonPostProcess()` syncs `showState = state`. Must update both fields when modifying state transitions\n\n---\n\n## Part 4: Cloud Module Review Guide\n\n### 4.1 FDB Transaction Mode\n\nCloud module uses FoundationDB (FDB) as the sole metadata storage. All metadata operations must be completed within FDB transactions.\n\n#### 4.1.1 TxnKv Interface Specifications\n\n- [ ] Is `TxnErrorCode` return value from each `Transaction::get()` / `commit()` checked? Marked as `[[nodiscard]]`\n- [ ] Is `TXN_CONFLICT` correctly handled? `commit_txn_immediately` returns error without retry, `commit_txn_eventually` has lazy committer path\n- [ ] Can transaction exceed FDB's 10MB limit? Large transactions need to use `TxnLazyCommitter` for batched commits\n- [ ] Is `TXN_MAYBE_COMMITTED` correctly handled? This indicates commit may have succeeded or failed, requires idempotent retry\n\n#### 4.1.2 Key Encoding Specifications\n\nThree key spaces:\n- `0x01`: User space (mutable metadata)\n- `0x02`: System space (service registration, encryption keys)\n- `0x03`: Version space (multi-version metadata with versionstamp, for snapshots)\n\n- [ ] Do new keys use correct key space prefix?\n- [ ] Does key encoding use `encode_bytes()` (order-preserving byte encoding) and `encode_int64()` (big-endian + sign marker)?\n- [ ] Does compound key field order match query pattern? Range scan relies on key prefix\n\n#### 4.1.3 Commit Path\n\nTwo commit paths:\n- **`commit_txn_immediately`**: Single FDB transaction completes all operations (read txn info, move tmp_rowset, bump version, update stats)\n- **`commit_txn_eventually`**: Uses lazy committer for batched commits when transaction is too large\n\n- [ ] Are the three config switches for lazy commit (`is_2pc`, `enable_txn_lazy_commit`, `config::enable_cloud_txn_lazy_commit`) correctly combined?\n- [ ] Is fallback path from `commit_txn_immediately` to `commit_txn_eventually` (triggered by `TXN_BYTES_TOO_LARGE`) correct?\n\n**Known risk—`commit_txn` infinite loop**: The `commit_txn` entry in `meta_service_txn.cpp:3257-3341` uses `do { ... } while(true)` loop to dispatch `commit_txn_immediately` and `commit_txn_eventually`. When pending lazy-committed transactions are found, it retries, but this loop **has no independent retry count limit**. If lazy committer continuously fails to complete (e.g., FDB continuous conflicts), theoretically infinite loop.\n\n- [ ] Do modifications involving `commit_txn` dispatch loop consider maximum retry count or total timeout?\n\n#### 4.1.4 Versionstamp Usage\n\nTransaction ID is derived from FDB versionstamp (`get_txn_id_from_fdb_ts`).\n\n- [ ] Does code depending on FDB versionstamp correctly use `atomic_set_ver_key()` / `atomic_set_ver_value()`?\n- [ ] Is `get_versionstamp()` only called after `commit()` succeeds?\n\n### 4.2 Meta-Service RPC Specifications\n\n#### 4.2.1 RPC Preprocessing\n\nEach RPC uses `RPC_PREPROCESS` macro for preprocessing:\n- StopWatch timing\n- DORIS_CLOUD_DEFER sets response status and records metrics on return\n- `RPC_RATE_LIMIT` performs per-instance QPS throttling\n\n- [ ] Do new RPCs use `RPC_PREPROCESS` and `RPC_RATE_LIMIT`?\n- [ ] Is response `MetaServiceCode` correctly set?\n\n#### 4.2.2 MetaServiceProxy Automatic Retry\n\n`MetaServiceProxy` decorator automatically retries (exponential backoff) on following error codes:\n- `KV_TXN_STORE_*_RETRYABLE`, `KV_TXN_TOO_OLD`, `KV_TXN_CONFLICT`\n\n- [ ] Do new RPCs need to be idempotent? Retry means same request may execute multiple times\n- [ ] Do non-idempotent operations correctly handle `TXN_MAYBE_COMMITTED`?\n\n### 4.3 Recycler Safety Specifications\n\nRecycler is responsible for GC of expired/deleted data:\n\n- [ ] Before deleting rowset data, are associated transactions aborted first (`abort_txn_for_related_rowset`)? Prevents commit-after-delete data loss\n- [ ] Are recycle operations best-effort + next-round retry mode? Should not stop entire recycle flow due to single failure\n- [ ] Is packed file reference count recycling correct? (`recycle_packed_files`)\n\n### 4.4 Delete Bitmap (Cloud MoW, Extremely High Priority)\n\nCloud mode delete bitmap management fundamentally differs from shared-nothing mode:\n\n- [ ] Bitmaps in `CloudTxnDeleteBitmapCache` contain sentinel marks and `TEMP_VERSION_COMMON`, **must manually replace version** before use. This is a critical correctness constraint explicitly marked in code comments\n- [ ] Empty rowset marks returned by `is_empty_rowset()` are not actively cleared (rely on expiration cleanup), does this affect correctness?\n- [ ] **`CloudTxnDeleteBitmapCache` LRU eviction side effect**: When cache entry is LRU evicted, `publish_status` is reset to `INIT`, causing next access to **completely recalculate** delete bitmap (not incremental update). This is correct but expensive. Additionally, marks written by `mark_empty_rowset()` are never actively cleared—they only disappear when entire cache entry is evicted. On high-frequency MoW tables with insufficient cache capacity, causes repeated recalculation\n- [ ] Do modifications involving `CloudTxnDeleteBitmapCache` consider impact of state reset after eviction?\n- [ ] Are `CalcDeleteBitmapTask`'s compaction stats consistent with latest state in meta-service?\n\n---\n\n## Part 5: Cross-Module Concerns\n\n### 5.1 FE-BE Protocol Compatibility\n\n- [ ] Do new `TPlanNodeType`s have corresponding branches in BE's processing switch?\n- [ ] Can deprecated node types (e.g., `MERGE_JOIN_NODE`, `KUDU_SCAN_NODE`) still be gracefully handled?\n- [ ] If new variables are added, are they correctly set in all paths where FE sends execution plans to BE?\n\n### 5.2 Cloud Mode vs Shared-Nothing Mode\n\nCode extensively uses `Config.isNotCloudMode()` / `Config.isCloudMode()` branches.\n\n- [ ] Do new features consider both modes?\n- [ ] In Cloud mode, tablets are not bound to specific BE—any compute node can load any tablet's metadata\n- [ ] In Cloud mode, rowsets go through two-phase protocol (`prepare_rowset` -> `commit_rowset`), with recycle marker protection in between\n- [ ] In Cloud mode, partition version is centrally managed by Meta-Service, not FE\n\n### 5.3 Merge-on-Write Unique Key Tables\n\nMoW tables are one of the most complex data paths, require special attention during review:\n\n- [ ] Write path: Does `SegmentWriter::probe_key_for_mow()` correctly probe existing rowsets?\n- [ ] Publish path: Is delete bitmap calculation under `_rowset_update_lock` protection?\n- [ ] Query path: Is delete bitmap version-aligned before reading?\n- [ ] Compaction path: Does `calc_delete_bitmap_for_compaction()` correctly handle `allow_delete_in_cumu_compaction`?\n- [ ] Cloud mode: Does delete bitmap cache have expiration mechanism? Is `_unused_delete_bitmap` cleaned up promptly?\n\n### 5.4 Performance Hotspot Concerns\n\n- [ ] Does `SendBatchThreadPool` (64 threads, 102K queue) have saturation risk in heavy shuffle scenarios?\n- [ ] Does `SegmentPrefetchThreadPool` (max 2000 threads) create too many OS threads in intensive scan scenarios?\n- [ ] In Cloud mode, does label deduplication check perform linear scan on labels with many aborted transactions, causing performance cliff?\n- [ ] Is serialization overhead of `_serialize_block_to_row_column()` for wide tables within expectations?\n- [ ] COW columns trigger deep copy on `mutate()` when `use_count > 1`—are there unnecessary copies in high-concurrency scenarios?\n\n---\n\n## Part 6: Testing Review Guide\n\n### 6.1 Regression Test Specifications\n\n- [ ] Is `order_qt_` prefix or manual `ORDER BY` used to ensure ordered results?\n- [ ] Do expected error cases use `test { sql \"...\"; exception \"...\" }` pattern?\n- [ ] Are tables **DROPped before use** (`DROP TABLE IF EXISTS xxx`), not after test ends?\n- [ ] Do simple tests directly hardcode table names instead of using `def tableName`?\n- [ ] Do tests cover these seven scenarios: empty table, all null, mixed null, pure non-null, constant, constant null, nullable wrapper (`nullable()`)?\n- [ ] Are result files (`.out`) auto-generated via `-genOut`? **Handwriting prohibited**\n\n### 6.2 BE Unit Test Specifications\n\n- [ ] Are Google Test's `TEST` / `TEST_F` macros used?\n- [ ] Are tests independent? Not dependent on other tests' execution order\n- [ ] Are `ASSERT_*` (failure terminates test) and `EXPECT_*` (failure continues execution) used correctly?\n- [ ] Do memory-intensive tests use `MemTracker` to check for leaks?\n\n### 6.3 FE Unit Test Specifications\n\n- [ ] Is JUnit 5's `@Test` annotation used?\n- [ ] Do test method names clearly describe test scenarios?\n- [ ] Is `Assertions.assertXxx()` used instead of JUnit 4's static imports?\n- [ ] Do tests involving concurrency use `CountDownLatch` / `CyclicBarrier` for synchronization?\n\n### 6.4 FE-BE Integration Test Specifications\n---\n\n## Part 7: Known High-Risk Code Patterns Quick Reference\n\n| Pattern | Risk Level | Location | Description |\n|------|---------|------|------|\n| `static_cast(status_returning_call)` | **CRITICAL** | BE global (195+ places) | Silently swallows errors, no logging |\n| Delete bitmap TEMP_VERSION not replaced | **CRITICAL** | `cloud_txn_delete_bitmap_cache.h:72-75` | Data visibility error |\n| `be_exec_version` mismatch | **CRITICAL** | FE-BE communication all paths | Serialization format incompatibility causes data corruption |\n| `(DataTypeNullable*)` C-style cast | **HIGH** | `block.h merge_impl` | No protection in release builds |\n| `sizeof(size_t*)` vs `sizeof(size_t)` | **HIGH** | `data_type.cpp:195` | Happens to work on 64-bit, crashes on 32-bit |\n| Lazy commit three-switch combination | **HIGH** | `meta_service_txn.cpp` | Missing config causes large transaction failure |\n| DecimalV2 original_precision default UINT32_MAX | **HIGH** | `data_type_decimal.h` | Affects string representation correctness |\n| `MOW` default false | **MEDIUM** | `segment_writer.h:70` | Omission causes deduplication to be silently skipped |\n| Cloud version cache TTL | **MEDIUM** | `OlapTable.getVisibleVersion()` | May read stale version |\n| Label scan linear complexity | **MEDIUM** | `meta_service_txn.cpp` | Many aborted txns cause performance cliff |\n| `THROW_IF_ERROR` no outer catch | **MEDIUM** | vec/ execution engine | Uncaught exception propagation causes crash |\n| `THROW_IF_ERROR` in Defer lambda | **CRITICAL** | `pipeline_task.cpp:412-413, 638-639` | Throwing exception during stack unwinding causes `std::terminate` |\n| `THROW_IF_ERROR` in Thrift RPC handler | **HIGH** | `backend_service.cpp:1323,1326` | No try/catch protection, connection drops |\n| `DCHECK` disappears in RELEASE builds | **HIGH** | BE global (2,872 places) | No invariant protection in production, should migrate","tools/cost_model_evaluate/README.MD":"\n\nThis code is used to evaluate the cost model in doris.\nYou can config the query in config of main.py \n\nBefore running, you should install the libraries in requirements.txt","tools/lzo/README.md":"\n\n# LZO Writer\n\nThis is simple LZO writer, you can use it to generate lzo compressed file.\n\nSee `main()` for details\n\n## Compilation\n\n```\nsh build.sh\n```\n\n## Generate file\n\n```\n./lzo_writer file_name.lzo\n```\n\n","tools/show_segment_status/README.md":"\n\nThis tool is used to get the progress of all current table transitions \nduring the online `segment_2` function.\n\nCurrently, you can specify 3 dimensions (in the conf file) to view the \nresults, you can specify one of them individually, or you can customize \nthe combination (that is, specify multiple at the same time).\n\n# Note\nWe use MySQLdb python lib to fetch meta from FE, so you must install it.\n\nYou can get MySQLdb lib from https://pypi.python.org/pypi/MySQL-python, \nthen you can install it as follows:\n```\n$ tar zxvf MySQL-python-*.tar.gz\n$ cd MySQL-python-*\n$ python setup.py build\n$ python setup.py install\n``` \n\n# Steps\n1. Fill in the conf according to your cluster configuration, and specify \n the table or be you want to watch.\n2. Execute `python show_segment_status.py`\n\n# Example\n1. If you want to watch the process of a table named `xxxx`, you can specify \n `table_name = xxxx` in conf file\n\n2. If you want to watch the process on be whose be_id is `xxxx`, you can specify \n `be_id = xxxx` in conf file\n\n# Output Example Format\n\nThe first number is about segment v2, the latter one is the total count.\n\n```\n==========SUMMARY()===========\nrowset_count: 289845 / 289845\nrowset_disk_size: 84627551189 / 84627551189\nrowset_row_count: 1150899153 / 1150899153\n===========================================================\n==========SUMMARY(table=xxxx)===========\nrowset_count: 289845 / 289845\nrowset_disk_size: 84627551189 / 84627551189\nrowset_row_count: 1150899153 / 1150899153\n===========================================================\n==========SUMMARY(be=10003 )===========\nrowset_count: 79650 / 79650\nrowset_disk_size: 24473921575 / 24473921575\nrowset_row_count: 331449328 / 331449328\n===========================================================\n```\n","tools/tpcds-tools/README.md":"\n\n# Run TPCDS with one command\n\nInstall the Trino TPCDS generator on every FE and BE using the\n[shared installation instructions](../benchmark/README.md#install-the-generators-once),\nthen configure `conf/doris-cluster.conf` and run:\n\n```bash\n./bin/run-tpcds.sh -s 1 -d tpcds_sf1\n```\n\nThis streams generated rows directly into Doris, creates the existing benchmark\ntables, collects statistics, and records three executions per query. It supports\nSF1, SF100, SF1000, and SF10000. Preparation requires a new database.\nUse `--queries-only` to measure existing data again, with the same scale factor.\n\nSee [shared workflow options, result files, and timing semantics](../benchmark/README.md).\nTPCDS uses the original scale-specific table DDL and multi-table queries.\nInstall the plugin before running the command.\n\n# Original file-based workflow\n\n## Usage\n\nThese scripts are used to make tpc-ds test.\nfollow the steps below:\n\n### 1. build tpc-ds dsdgen dsqgen tool.\n\n ./bin/build-tpcds-tools.sh\n\n If the build failed in dbgen tools' compilation, update your GCC version or change all \"TPC-DS_Tools_v3.2.0new.zip\" in build-tpcds-dbgen.sh to \"TPC-DS_Tools_v3.2.0.zip\"\n\n### 2. generate tpc-ds data. use -h for more infomations.\n\n ./bin/gen-tpcds-data.sh -s 1\n\n### 3. create tpc-ds tables. modify `conf/doris-cluster.conf` to specify doris info, then run script below.\n\n ./bin/create-tpcds-tables.sh -s 1\n\n### 4. load tpc-ds data. use -h for help.\n\n ./bin/load-tpcds-data.sh\n\n### 5. run tpc-ds queries.\n\n ./bin/run-tpcds-queries.sh -s 1\n","AGENTS.md":"# AGENTS.md — Apache Doris\n\nThis is the codebase for Apache Doris, an MPP OLAP database. It primarily consists of the Backend module BE (`be/`, execution and storage engine), the Frontend module FE (`fe/`, optimizer and transaction core), and the Cloud module (`cloud/`, storage-compute separation). Your basic development workflow is: modify code, build using standard procedures, add and run tests, and submit relevant changes.\n\n## Security Threat Model\n\nFor security scans, vulnerability triage, security reviews, and changes involving authentication, authorization, network boundaries, external catalogs, cloud tenancy, or other security-sensitive behavior, read `SECURITY.md` first, then follow it to `threat-model.md`. Use `threat-model.md` to determine in-scope components, trust boundaries, attacker roles, explicit non-goals, and triage classification. Findings that are out of model or by design under `threat-model.md` should be reported as such, not treated as Doris vulnerabilities. Such reviews will not be conducted unless explicitly requested by the user, and no conclusions regarding this matter need to be provided.\n\n## Remote Artifacts and Dynamic Code Loading\n\nDo not add any new code path that fetches an artifact from a user-supplied URL and loads it into a Doris process. This covers `driver_url`-style catalog/resource properties, plugin and UDF locations, and any other property whose value reaches a `URLClassLoader`, `Class.forName`, `System.load`/`dlopen`, or an equivalent BE-side loader. A user who can write such a property gets code execution inside the process that loads it, so the property's effective privilege is that process, not the DDL privilege it was gated on.\n\nThe existing `driver_url` paths (jdbc / iceberg-jdbc / paimon-jdbc catalogs) remain for backward compatibility. They are grandfathered, not a precedent — do not copy the pattern into a new connector or feature.\n\nWhen a feature genuinely needs an operator-supplied artifact, read it from a local, operator-controlled directory (the `jdbc_drivers_dir` / plugin-directory pattern), not from a URL. The ADBC catalog's `AdbcDriverPathResolver` is the model: remote schemes rejected outright, a scheme-less value restricted to a bare file name so no path separator can escape the configured directory, and the check placed where the artifact is loaded rather than only at CREATE.\n\nEvery consumer of the **jdbc-flavored** `driver_url` (the jdbc / iceberg-jdbc / paimon-jdbc catalogs, which do accept a URL) must route the raw value through `JdbcDriverUrlSecurity.check` (`fe-foundation`, `foundation.security`) from its property holder's statement-time validation — `JdbcCatalogProperties.checkCreateTimeOnlyRules` and the iceberg/paimon JDBC metastore holders' `validate()` — which the engine reaches on both CREATE and ALTER CATALOG and never on replay or a catalog rebuild. That class is the single source of truth for the rule; do not fork or re-derive it per connector, and never call it from a holder's `of()`. Such a property must also get a row in the key table of `PluginDrivenExternalCatalog#driverUrlKeysOf` (one table drives both the gate trigger set and the checked values), which is how the operator's `jdbc_driver_secure_path` / `jdbc_driver_url_white_list` policy reaches the ALTER CATALOG path — ALTER never runs `Connector.preCreateValidation`, where CREATE applies that policy, so a key missing from that table lets an ALTER repoint the jar past the operator's configuration.\n\nSeparately, any new outbound HTTP request the FE or BE issues to a host named by a non-SUPER user is an SSRF surface. Call it out explicitly in the PR description together with the privilege required to reach it.\n\n## Thrift and Protobuf Field Ids\n\nEvery change under `gensrc/thrift` and `gensrc/proto` must keep every upgrade compatible (branch-3.1 → 4.0 → 4.1 → 4.2 → master): a field id or enum value means the same thing on every branch, an id is never reused, and a new field is optional. The rule and how to review against it are in `gensrc/AGENTS.md`; read it before touching those directories.\n\n## When running in a WORKTREE directory\n\nTo ensure smooth test execution without interference between worktrees, the first thing to do upon entering a worktree directory is to check if `.worktree_initialized` exists. If not, execute `hooks/setup_worktree.sh`, setting `$ROOT_WORKSPACE_PATH` to the base directory (typically `${DORIS_REPO}`) beforehand. After successful execution, verify that `.worktree_initialized` has been touched and that `thirdparty/installed` dependencies exist correctly.\n\nSubmodules are not part of the default worktree setup. If the task does not compile Doris or run tests that require submodules, leave all submodules uninitialized. Build and test entry points initialize the submodule paths they require. Only initialize a submodule manually when the task actually needs it and an entry point does not handle it; initialize the exact required path instead of running an unscoped recursive update for every submodule.\n\nWhen working in worktree mode, all operations must be confined to the current worktree directory. Do not enter `${DORIS_REPO}` or use any resources there. Compilation and execution must be done within the current worktree directory. The compiled Doris cluster must use random ports not used by other worktrees (modify BE and FE conf before compilation, using a uniform offset of `${DORIS_PORT_OFFSET_RANGE}` from default ports without conflicting with other worktrees' ports). Run from the `output` directory within the worktree. To run regression tests, modify `regression-test/conf/regression-conf.groovy` and set the port numbers in `jdbcUrl` and other configuration items to your new ports so the corresponding worktree cluster can be used for regression testing.\n\n## Coding Standards\n\nAssert correctness only—never use defensive programming with `if` or similar constructs. Any `if` check for errors must have a clearly known inevitable failure path (not speculation). If no such scenario is found, strictly avoid using `if(valid)` checks. However, you may use the `DORIS_CHECK` macro for precondition assertions (if inside performance-sensitive areas like loops, it can only be `DCHECK`). For example, if logically A=true should always imply B=true, then strictly avoid `if (A && B)` and instead use `if (A) { DORIS_CHECK(B); ... }`. In short, the principle is: upon discovering errors or unexpected situations, report errors or crash—never allow the process to continue.\n\nFor `PaddedPODArray` and its peripheral packaging types, such as some certain Column, negative alignment allows the use of -1 as a valid index. No additional special handling is needed when the index may be -1.\n\nWhen adding code, strictly follow existing similar code in similar contexts, including interface usage, error handling, and locking patterns. When adding any code, first try to reference existing functionality. Second, examine the relevant context paragraphs to fully understand the logic.\n\nAfter adding code, conduct self-review and refactoring attempts to ensure good abstraction and reuse as much as possible.\n\n### Code Style Enforcement\n\nAll code must pass style checks before committing. Use the corresponding skill for detailed step-by-step procedures.\n\n**BE (C++) Formatting**: Run `build-support/clang-format.sh` to auto-fix formatting. This script enforces clang-format v16; do not use other versions. Run `build-support/check-format.sh` to check without modifying files. See the `be-code-style` skill for details.\n\n**BE (C++) Static Analysis**: After building BE (which generates `compile_commands.json`), run `build-support/run-clang-tidy.sh` to check modified C++ files against the `.clang-tidy` config. The script parses `git diff` to filter warnings to changed lines where possible, reducing noise from pre-existing code (diagnostics from included headers may still appear). For Cloud C++ files, pass `--build-dir` pointing to the Cloud compilation database (e.g., `cloud/build_ASAN`). Try to fix all reported warnings; if a warning cannot be reasonably fixed, add a `// NOLINT` comment with justification and report it. See the `clang-tidy-check` skill for details.\n\n**BE (C++) Header Hygiene**: BE configure enforces compile-time hygiene gates (header layering rules, include-closure/reach budgets, pch whitelist, extern-template pairing, unity-skip coverage) via `build-support/check-build-hygiene.sh` (~1s, pure text). Run it directly after touching BE headers, includes, template instantiation lists, or unity skip lists — do not wait for CI. Violations are build errors whose messages carry the mechanism and fix; deliberate budget/whitelist changes are one-line table edits in the same commit with justification. Rules and review checkpoints: `be/AGENTS.md`; mechanics: `be/README.md`.\n\n**FE (Java) Style**: Checkstyle is integrated into the Maven build (`maven-checkstyle-plugin`). Running `build.sh --fe` automatically validates style via `mvn validate`. If checkstyle fails, fix the reported issues according to `fe/check/checkstyle/checkstyle.xml`. See the `fe-code-style` skill for details.\n\n## Code Review\n\nWhen conducting code review (including self-review and review tasks), complete the key checkpoints from the `code-review` skill and provide conclusions for each key checkpoint when applicable. Other content does not require item-by-item responses; check them during the review process.\n\n## Build and Run Standards\n\nAlways use only the `build.sh` script with its correct parameters to build Doris BE and FE. For example, the simplest BE+FE build command is `./build.sh --be --fe`.\nBuild type can be set via `BUILD_TYPE` in `custom_env.sh`, but only set it to `RELEASE` when explicitly required for performance testing; otherwise, keep it as `ASAN`.\nYou may modify BE and FE ports and network settings in `conf/` before compilation to ensure correctness and avoid conflicts.\nBuild artifacts are in the current directory's `output/`. If starting the service, ensure all process artifacts have their conf set with appropriate non-conflicting ports and the correct `priority_networks` that match the current network environment, for example `priority_networks = 10.16.10.3/24`. Use `--daemon` when starting. Cluster startup is slow; wait at least 30s for success. If still not ready after waiting, continue waiting. If not ready after a long time, check BE and FE logs to investigate.\nFor first-time cluster startup, you may need to manually add the backend.\n\n## Testing Standards\n\nAll kernel features must have corresponding tests. Prioritize adding regression tests under `regression-test/`, while also having BE unit tests (`be/test/`) and FE unit tests (`fe/fe-core/src/test/`) where possible. Interface usage in test cases must first reference similar cases.\n\nYou must use the preset scripts in the codebase with their correct parameters to run tests (`run-regression-test.sh`, `run-be-ut.sh`, `run-fe-ut.sh`). Regression test result files must not be handwritten; they must be auto-generated via test scripts. When running regression tests, if using `-s` to specify a case, also try to use `-d` to specify the parent directory for faster execution. For example, for cases under `nereids_p0`, you can use `-d nereids_p0 -s xxx`, where `xxx` is the name from `suite(\"xxx\")` in the groovy file.\n\nKey utility functions in BE code, as well as the core logic of complete features, must have corresponding unit tests. If it is inconvenient to add unit tests, revisit the module design and function decomposition to ensure high cohesion and low coupling.\n\nAdded regression tests must comply with the following standards:\n\n1. Use `order_qt` prefix or manually add `order by` to ensure ordered results\n2. For cases expected to error, use the `test{sql,exception}` pattern\n3. After completing tests, do not drop tables; instead drop tables before using them in tests, to preserve the environment for debugging\n4. For ordinary single test tables, do not use `def tableName` form; instead hardcode your table name in all SQL\n5. Except for variables you explicitly need to adjust for testing current functionality, other variables do not need extra setup before testing. For example, nereids optimizer and pipeline engine settings can use default states\n6. For determined expected results, do not using methods like `assert` in test groovy files, but instead generate the `.out` file using `qt_sql` and similar methods.\n\n## Commit Standards\n\nFiles in git commit should only be related to the current modification task. Environment modifications for running (e.g., `conf/`, `AGENTS.md`, `hooks/`, etc.) must not be `git add`ed. When delivering the final task, you must ensure all actual code modifications have been committed.\n\nCommit messages must follow the format below, which mirrors the PR template (`.github/PULL_REQUEST_TEMPLATE.md`):\n\n```\n[]() \n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary: \n\n### Release note\n\n\n\n### Check List (For Author)\n\n- Test: \n - Regression test / Unit Test / Manual test / No need to test (with reason)\n- Behavior changed: No / Yes (with explanation)\n- Does this need documentation: No / Yes (with doc PR link)\n```\n\nKey rules for commit messages:\n\n1. The title must follow the `[type](module)` format validated by the PR title checker (`.github/workflows/title-checker.yml`). Common types include: `fix`, `feature`, `improvement`, `refactor`, `chore`, `test`, `doc`. Common modules include: `fe`, `be`, `cloud`, `regression`, `build`\n2. The short summary must be concise and written in imperative mood (e.g., `[fix](fe) Fix null pointer in scan node` not `[fix](fe) Fixed null pointer`)\n3. The `Issue Number` field must reference the corresponding GitHub Issue with `close #xxx` syntax when applicable\n4. The `Release note` section must be filled in for any user-visible behavior or feature change; write \"None\" for internal refactoring or test-only changes\n5. The `Problem Summary` section should cover the following content when available: problem reproduction method, root cause in code, end-to-end results/phenomena before and after repair, and the fix. If it's a refactoring, explain the reason. If it's a performance improvement, specify the case and the exact improvement amount. DO NOT mention any specific JIRA numbers. The background of the problem should be fully understandable through this section alone.\n6. The test section must honestly reflect the testing performed; do not claim tests that were not actually run\n\nFiles in a git commit should only be related to the current modification task. Environment modifications for running (for example `conf/`, `AGENTS.md`, `hooks/`) must not be `git add`ed. When delivering the final task, ensure all actual code modifications have been committed.\n","tools/pipeline-tracing/README.md":"\n\n# Pipeline Tracing and Display Tool\n\nIn the Pipeline execution engine, we split the execution plan tree of each Instance into multiple small Pipeline Tasks and execute them under our custom Pipeline scheduler. Therefore, in an environment with a large number of Pipeline Tasks executing, how these Tasks are scheduled across threads and CPU cores is an important factor for execution performance. We have developed a specialised tool to observe the scheduling process on a particular query or time period, which we call \"Pipeline Tracing\".\n\nThis tool converts record files to proper JSON format for visualization.\n\n## How to Use\n\n```shell\npython3 origin-to-show.py -s -d .json\n```\nto transfer record file `` to `.json`. Then it could be visualized.\n\n```shell\npython3 origin-to-show.py --help\n```\nfor help details.","tools/ssb-tools/README.md":"\n\n# Run SSB with one command\n\nThe Trino SSB connector generates rows while Doris reads them. It uses the same\n`ssb-dbgen` distribution as the existing tools, with stdout connected to the scan\nthrough a bounded pipe. There are no intermediate `.tbl` files, Stream Load\nrequests, or separate Trino servers.\n\n## Install the generator plugin once\n\nBuild on Linux with the same CPU architecture as the Doris nodes. The build needs\nJDK 17, Maven, GCC, Make, curl, patch, and sha256sum:\n\n```bash\n./trino-ssb/build.sh\n```\n\nCopy the resulting **entire directory** `trino-ssb/target/ssb` into the configured\nTrino plugin directory on **every FE and BE**. For current Doris installations:\n\n```bash\ncp -r trino-ssb/target/ssb /path/to/fe/plugins/trino_plugins/\ncp -r trino-ssb/target/ssb /path/to/be/plugins/trino_plugins/\n```\n\nStart or restart these nodes after installing the plugin. It consists of a JAR,\nthe native `dbgen` executable, and its distribution file. The executable must\nremain executable. No paths or executable commands are accepted as catalog properties.\n\n## Create, import, and benchmark\n\nSet the Doris connection in `conf/doris-cluster.conf`, then run:\n\n```bash\n./bin/run-ssb.sh -s 1 -d ssb_sf1\n```\n\nThis creates a new database and its `ssb_gen_ssb_sf1` catalog, creates the five\nSSB tables, and imports them using `INSERT INTO ... SELECT ...` from the catalog's\n`sf1` schema. It then builds `lineorder_flat`, collects statistics synchronously,\nand runs all 13 SSB queries and all 13 SSB Flat queries three times each.\nThe supported scale factors are 1, 100, and 1000. Use `--mode ssb` or `--mode flat`\nto select one benchmark; the default is `both`.\n\nEach run writes a new `results/-/` directory with:\n\n- `result.csv`: first-run, two repeated-run, and best repeated-run times in milliseconds.\n- `ssb/` and `ssb-flat/`: each query's output and errors.\n- `prepare.log` and `environment.txt`: import diagnostics and Doris settings.\n\nTiming includes the mysql client connection and result transfer, as in the old\nscripts. The `cold` column means the first execution; the script does not flush\ndata caches. SQL result caching and BE query caching are disabled for every measured\nsession. A query or import error stops the run with a nonzero exit status.\n\nPreparation requires a **new database**, so rerunning a failed or completed load\ncannot append duplicate data. Tables and the catalog are preserved for inspection.\nTo rerun only the queries against completed data:\n\n```bash\n./bin/run-ssb.sh -d ssb_sf1 --queries-only\n```\n\nUse `-c /path/to/cluster.conf` for a separate connection configuration, and\n`--result-dir /path/to/new-directory` for a custom output directory. Existing\nresult directories are never overwritten.\n\n## Generator consistency and tests\n\nThe connector keeps the old tools' default of ten `lineorder` generator partitions.\nEach dimension uses one generator. Changing the original dbgen partition count\nchanges some random streams, so partition count is intentionally fixed here.\nThe date generator runs in UTC on every BE. The native build enables dbgen's\nexisting 64-bit code path for large scale factors and avoids an out-of-bounds\nusage-counter update for its SSB-only random streams. Neither change replaces\nthe data-generation algorithm.\n\nThe existing benchmark SQL is preserved. In particular, SSB Q1.3 filters the\ngenerator's `d_weeknuminyear`, whereas SSB Flat Q1.3 uses `weekofyear()`; these week\ndefinitions differ, so their aggregate results are not expected to match.\n\nThe plugin build runs its Java tests, including full SF1 checksums of all five tables,\nhandle serialization, projected columns, large keys, producer failures, and early\ncursor closure. Run script orchestration tests with:\n\n```bash\npython3 -m unittest discover -s ../benchmark/tests -v\n```\n\nThe shared runner also supports [TPCH and TPCDS](../benchmark/README.md).\n\n# Original file-based workflow\n\n These scripts are used to make ssb and ssb flat test.\n The ssb flat data comes from ssb tables by way of 'INSERT INTO ... SELECT ...'.\n\n## ssb test, follow the steps below:\n### 1. build ssb dbgen tool.\n ./bin/build-ssb-dbgen.sh\n### 2. generate ssb data. use -h for more infomations.\n ./bin/gen-ssb-data.sh -s 1\n### 3. create ssb tables. modify `conf/doris-cluster.conf` to specify Doris cluster info, then run script below.\n ./bin/create-ssb-tables.sh -s 1\n### 4. load ssb data. use -h for help.\n ./bin/load-ssb-data.sh\n### 5. run ssb queries.\n ./bin/run-ssb-queries.sh\n\n## ssb flat test, follow the steps below:\n### 1. prepare ssb data, which means ssb test steps 1 to 4 have been done.\n### 2. run ssb flat queries.\n ./bin/run-ssb-flat-queries.sh\n","tools/row_to_column/README.md":"\n\nSteps of row-to-column conversion:\n\n1. Modify the corresponding cluster configuration in `conf`.\n\n2. Run `python convert_row_to_column.py` to generate all row storage to column storage commands.\n\n3. If the script runs without any results, it means that there is no row storage table in the cluster, so nothing need to be done.\n\n4. If there is a table in the script that fails to obtain information.\n\n Please use `show create table` in the cluster to see the `storage_type` of the table. If it is row storage, please manually convert it.\n\n5. After the schema change is completed, the old schema needs to be deleted after 10 minutes.\n\n Please wait for a while to make sure that the data of the old schema has been deleted, and then upgrade.\n Otherwise, the BE process will exit with an error when it is automatically detected when it starts.\n\n6. Run `python convert_row_to_column.py` again to verify that there are no rows in the table before upgrading.\n","tools/clickbench-tools/README.md":"\n\n# Usage\n\n**These scripts are used to do [ClickBench](https://benchmark.clickhouse.com/) test, more info [there](https://github.com/ClickHouse/ClickBench).**\n\n## follow the steps below:\n\n### 1. create table\n ./create-clickbench-table.sh\n\n### 2. load data\n ./load-clickbench-data.sh\n\n### 3. run queries\n ./run-clickbench-queries.sh\n","tools/tpch-tools/README.md":"# tpch-tools Documentation\n\nPath: `tools/tpch-tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/tools/tpch-tools/README.md)","be/src/io/tools/readme.md":"# tools Documentation\n\nPath: `be/src/io/tools/readme.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/be/src/io/tools/readme.md)","fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md":"# fe-authentication-plugin-ldap Documentation\n\nPath: `fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md)","fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md":"# fe-authentication-plugin-password Documentation\n\nPath: `fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md)","extension/DataX/doriswriter/src/main/resources/plugin.json":"# resources Plugin Manifest\n\nPath: `extension/DataX/doriswriter/src/main/resources/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/extension/DataX/doriswriter/src/main/resources/plugin.json)"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md — Apache Doris\n\nThis is the codebase for Apache Doris, an MPP OLAP database. It primarily consists of the Backend module BE (`be/`, execution and storage engine), the Frontend module FE (`fe/`, optimizer and transaction core), and the Cloud module (`cloud/`, storage-compute separation). Your basic development workflow is: modify code, build using standard procedures, add and run tests, and submit relevant changes.\n\n## Security Threat Model\n\nFor security scans, vulnerability triage, security reviews, and changes involving authentication, authorization, network boundaries, external catalogs, cloud tenancy, or other security-sensitive behavior, read `SECURITY.md` first, then follow it to `threat-model.md`. Use `threat-model.md` to determine in-scope components, trust boundaries, attacker roles, explicit non-goals, and triage classification. Findings that are out of model or by design under `threat-model.md` should be reported as such, not treated as Doris vulnerabilities. Such reviews will not be conducted unless explicitly requested by the user, and no conclusions regarding this matter need to be provided.\n\n## Remote Artifacts and Dynamic Code Loading\n\nDo not add any new code path that fetches an artifact from a user-supplied URL and loads it into a Doris process. This covers `driver_url`-style catalog/resource properties, plugin and UDF locations, and any other property whose value reaches a `URLClassLoader`, `Class.forName`, `System.load`/`dlopen`, or an equivalent BE-side loader. A user who can write such a property gets code execution inside the process that loads it, so the property's effective privilege is that process, not the DDL privilege it was gated on.\n\nThe existing `driver_url` paths (jdbc / iceberg-jdbc / paimon-jdbc catalogs) remain for backward compatibility. They are grandfathered, not a precedent — do not copy the pattern into a new connector or feature.\n\nWhen a feature genuinely needs an operator-supplied artifact, read it from a local, operator-controlled directory (the `jdbc_drivers_dir` / plugin-directory pattern), not from a URL. The ADBC catalog's `AdbcDriverPathResolver` is the model: remote schemes rejected outright, a scheme-less value restricted to a bare file name so no path separator can escape the configured directory, and the check placed where the artifact is loaded rather than only at CREATE.\n\nEvery consumer of the **jdbc-flavored** `driver_url` (the jdbc / iceberg-jdbc / paimon-jdbc catalogs, which do accept a URL) must route the raw value through `JdbcDriverUrlSecurity.check` (`fe-foundation`, `foundation.security`) from its property holder's statement-time validation — `JdbcCatalogProperties.checkCreateTimeOnlyRules` and the iceberg/paimon JDBC metastore holders' `validate()` — which the engine reaches on both CREATE and ALTER CATALOG and never on replay or a catalog rebuild. That class is the single source of truth for the rule; do not fork or re-derive it per connector, and never call it from a holder's `of()`. Such a property must also get a row in the key table of `PluginDrivenExternalCatalog#driverUrlKeysOf` (one table drives both the gate trigger set and the checked values), which is how the operator's `jdbc_driver_secure_path` / `jdbc_driver_url_white_list` policy reaches the ALTER CATALOG path — ALTER never runs `Connector.preCreateValidation`, where CREATE applies that policy, so a key missing from that table lets an ALTER repoint the jar past the operator's configuration.\n\nSeparately, any new outbound HTTP request the FE or BE issues to a host named by a non-SUPER user is an SSRF surface. Call it out explicitly in the PR description together with the privilege required to reach it.\n\n## Thrift and Protobuf Field Ids\n\nEvery change under `gensrc/thrift` and `gensrc/proto` must keep every upgrade compatible (branch-3.1 → 4.0 → 4.1 → 4.2 → master): a field id or enum value means the same thing on every branch, an id is never reused, and a new field is optional. The rule and how to review against it are in `gensrc/AGENTS.md`; read it before touching those directories.\n\n## When running in a WORKTREE directory\n\nTo ensure smooth test execution without interference between worktrees, the first thing to do upon entering a worktree directory is to check if `.worktree_initialized` exists. If not, execute `hooks/setup_worktree.sh`, setting `$ROOT_WORKSPACE_PATH` to the base directory (typically `${DORIS_REPO}`) beforehand. After successful execution, verify that `.worktree_initialized` has been touched and that `thirdparty/installed` dependencies exist correctly.\n\nSubmodules are not part of the default worktree setup. If the task does not compile Doris or run tests that require submodules, leave all submodules uninitialized. Build and test entry points initialize the submodule paths they require. Only initialize a submodule manually when the task actually needs it and an entry point does not handle it; initialize the exact required path instead of running an unscoped recursive update for every submodule.\n\nWhen working in worktree mode, all operations must be confined to the current worktree directory. Do not enter `${DORIS_REPO}` or use any resources there. Compilation and execution must be done within the current worktree directory. The compiled Doris cluster must use random ports not used by other worktrees (modify BE and FE conf before compilation, using a uniform offset of `${DORIS_PORT_OFFSET_RANGE}` from default ports without conflicting with other worktrees' ports). Run from the `output` directory within the worktree. To run regression tests, modify `regression-test/conf/regression-conf.groovy` and set the port numbers in `jdbcUrl` and other configuration items to your new ports so the corresponding worktree cluster can be used for regression testing.\n\n## Coding Standards\n\nAssert correctness only—never use defensive programming with `if` or similar constructs. Any `if` check for errors must have a clearly known inevitable failure path (not speculation). If no such scenario is found, strictly avoid using `if(valid)` checks. However, you may use the `DORIS_CHECK` macro for precondition assertions (if inside performance-sensitive areas like loops, it can only be `DCHECK`). For example, if logically A=true should always imply B=true, then strictly avoid `if (A && B)` and instead use `if (A) { DORIS_CHECK(B); ... }`. In short, the principle is: upon discovering errors or unexpected situations, report errors or crash—never allow the process to continue.\n\nFor `PaddedPODArray` and its peripheral packaging types, such as some certain Column, negative alignment allows the use of -1 as a valid index. No additional special handling is needed when the index may be -1.\n\nWhen adding code, strictly follow existing similar code in similar contexts, including interface usage, error handling, and locking patterns. When adding any code, first try to reference existing functionality. Second, examine the relevant context paragraphs to fully understand the logic.\n\nAfter adding code, conduct self-review and refactoring attempts to ensure good abstraction and reuse as much as possible.\n\n### Code Style Enforcement\n\nAll code must pass style checks before committing. Use the corresponding skill for detailed step-by-step procedures.\n\n**BE (C++) Formatting**: Run `build-support/clang-format.sh` to auto-fix formatting. This script enforces clang-format v16; do not use other versions. Run `build-support/check-format.sh` to check without modifying files. See the `be-code-style` skill for details.\n\n**BE (C++) Static Analysis**: After building BE (which generates `compile_commands.json`), run `build-support/run-clang-tidy.sh` to check modified C++ files against the `.clang-tidy` config. The script parses `git diff` to filter warnings to changed lines where possible, reducing noise from pre-existing code (diagnostics from included headers may still appear). For Cloud C++ files, pass `--build-dir` pointing to the Cloud compilation database (e.g., `cloud/build_ASAN`). Try to fix all reported warnings; if a warning cannot be reasonably fixed, add a `// NOLINT` comment with justification and report it. See the `clang-tidy-check` skill for details.\n\n**BE (C++) Header Hygiene**: BE configure enforces compile-time hygiene gates (header layering rules, include-closure/reach budgets, pch whitelist, extern-template pairing, unity-skip coverage) via `build-support/check-build-hygiene.sh` (~1s, pure text). Run it directly after touching BE headers, includes, template instantiation lists, or unity skip lists — do not wait for CI. Violations are build errors whose messages carry the mechanism and fix; deliberate budget/whitelist changes are one-line table edits in the same commit with justification. Rules and review checkpoints: `be/AGENTS.md`; mechanics: `be/README.md`.\n\n**FE (Java) Style**: Checkstyle is integrated into the Maven build (`maven-checkstyle-plugin`). Running `build.sh --fe` automatically validates style via `mvn validate`. If checkstyle fails, fix the reported issues according to `fe/check/checkstyle/checkstyle.xml`. See the `fe-code-style` skill for details.\n\n## Code Review\n\nWhen conducting code review (including self-review and review tasks), complete the key checkpoints from the `code-review` skill and provide conclusions for each key checkpoint when applicable. Other content does not require item-by-item responses; check them during the review process.\n\n## Build and Run Standards\n\nAlways use only the `build.sh` script with its correct parameters to build Doris BE and FE. For example, the simplest BE+FE build command is `./build.sh --be --fe`.\nBuild type can be set via `BUILD_TYPE` in `custom_env.sh`, but only set it to `RELEASE` when explicitly required for performance testing; otherwise, keep it as `ASAN`.\nYou may modify BE and FE ports and network settings in `conf/` before compilation to ensure correctness and avoid conflicts.\nBuild artifacts are in the current directory's `output/`. If starting the service, ensure all process artifacts have their conf set with appropriate non-conflicting ports and the correct `priority_networks` that match the current network environment, for example `priority_networks = 10.16.10.3/24`. Use `--daemon` when starting. Cluster startup is slow; wait at least 30s for success. If still not ready after waiting, continue waiting. If not ready after a long time, check BE and FE logs to investigate.\nFor first-time cluster startup, you may need to manually add the backend.\n\n## Testing Standards\n\nAll kernel features must have corresponding tests. Prioritize adding regression tests under `regression-test/`, while also having BE unit tests (`be/test/`) and FE unit tests (`fe/fe-core/src/test/`) where possible. Interface usage in test cases must first reference similar cases.\n\nYou must use the preset scripts in the codebase with their correct parameters to run tests (`run-regression-test.sh`, `run-be-ut.sh`, `run-fe-ut.sh`). Regression test result files must not be handwritten; they must be auto-generated via test scripts. When running regression tests, if using `-s` to specify a case, also try to use `-d` to specify the parent directory for faster execution. For example, for cases under `nereids_p0`, you can use `-d nereids_p0 -s xxx`, where `xxx` is the name from `suite(\"xxx\")` in the groovy file.\n\nKey utility functions in BE code, as well as the core logic of complete features, must have corresponding unit tests. If it is inconvenient to add unit tests, revisit the module design and function decomposition to ensure high cohesion and low coupling.\n\nAdded regression tests must comply with the following standards:\n\n1. Use `order_qt` prefix or manually add `order by` to ensure ordered results\n2. For cases expected to error, use the `test{sql,exception}` pattern\n3. After completing tests, do not drop tables; instead drop tables before using them in tests, to preserve the environment for debugging\n4. For ordinary single test tables, do not use `def tableName` form; instead hardcode your table name in all SQL\n5. Except for variables you explicitly need to adjust for testing current functionality, other variables do not need extra setup before testing. For example, nereids optimizer and pipeline engine settings can use default states\n6. For determined expected results, do not using methods like `assert` in test groovy files, but instead generate the `.out` file using `qt_sql` and similar methods.\n\n## Commit Standards\n\nFiles in git commit should only be related to the current modification task. Environment modifications for running (e.g., `conf/`, `AGENTS.md`, `hooks/`, etc.) must not be `git add`ed. When delivering the final task, you must ensure all actual code modifications have been committed.\n\nCommit messages must follow the format below, which mirrors the PR template (`.github/PULL_REQUEST_TEMPLATE.md`):\n\n```\n[]() \n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary: \n\n### Release note\n\n\n\n### Check List (For Author)\n\n- Test: \n - Regression test / Unit Test / Manual test / No need to test (with reason)\n- Behavior changed: No / Yes (with explanation)\n- Does this need documentation: No / Yes (with doc PR link)\n```\n\nKey rules for commit messages:\n\n1. The title must follow the `[type](module)` format validated by the PR title checker (`.github/workflows/title-checker.yml`). Common types include: `fix`, `feature`, `improvement`, `refactor`, `chore`, `test`, `doc`. Common modules include: `fe`, `be`, `cloud`, `regression`, `build`\n2. The short summary must be concise and written in imperative mood (e.g., `[fix](fe) Fix null pointer in scan node` not `[fix](fe) Fixed null pointer`)\n3. The `Issue Number` field must reference the corresponding GitHub Issue with `close #xxx` syntax when applicable\n4. The `Release note` section must be filled in for any user-visible behavior or feature change; write \"None\" for internal refactoring or test-only changes\n5. The `Problem Summary` section should cover the following content when available: problem reproduction method, root cause in code, end-to-end results/phenomena before and after repair, and the fix. If it's a refactoring, explain the reason. If it's a performance improvement, specify the case and the exact improvement amount. DO NOT mention any specific JIRA numbers. The background of the problem should be fully understandable through this section alone.\n6. The test section must honestly reflect the testing performed; do not claim tests that were not actually run\n\nFiles in a git commit should only be related to the current modification task. Environment modifications for running (for example `conf/`, `AGENTS.md`, `hooks/`) must not be `git add`ed. When delivering the final task, ensure all actual code modifications have been committed.\n","isInternal":false,"tokens":3243,"sizeBytes":15063},{"name":"SKILL.md","path":".claude/skills/code-review/SKILL.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/.claude/skills/code-review/SKILL.md","title":"code-review","category":"anthropic-skill","format":"markdown","content":"---\nname: code-review\ndescription: Create consistent releases and changelogs\ncompatibility: opencode\n---\n\n## What I do\n\nComplete code review work to ensure code quality. Due to the length of content, you can read sections as needed based on the actual modifications being reviewed.\n\n## When to use me\n\nUse this when you need to review code, whether it's code you just completed or directly reviewing target code.\n\n---\n\n## Part 1: General Principles\n\n### 1.1 Core Invariants\n\nAlways focus on the following core invariants during review:\n1. **Data Correctness**: Data from any committed transaction must be visible to subsequent queries and not lost; data from uncommitted transactions must not be visible\n2. **Version Consistency**: Partition visible version is the sole standard for data visibility; BE must strictly read data not exceeding this version\n3. **Delete Bitmap Consistency** (MoW tables): delete bitmap must be strictly aligned with rowset versions; sentinel mark / TEMP_VERSION must be replaced with actual versions before use\n4. **Memory Safety**: All significant memory allocations must be tracked by MemTracker; orphan memory is not allowed\n5. **Error Means Failure**: Invariant violations should trigger exceptions or non-OK Status; silent continuation is never allowed\n\n### 1.2 Review Principles\n\n- **Defensive Programming Prohibited**: Do not use `if(valid)` style defensive checks to mask logic errors. If logically A=true should imply B=true, write `if(A) { DORIS_CHECK(B); ... }`, never `if(A && B)`\n- **Follow Context Conventions**: When adding code, strictly follow patterns in adjacent code—same error handling, same interface call order, same lock acquisition patterns, unless a clearly correct and more reasonable approach is identified.\n- **Prioritize Reuse**: Before adding new functionality, search for similar implementations that can be reused or extended; after adding, ensure good abstraction for shared code.\n- **Code First**: When this SKILL conflicts with actual code behavior, defer to your understanding of actual code behavior and explain\n- **Performance First**: All obviously redundant operations should be optimized away, all obvious performance optimizations must be applied, and obvious anti-patterns must be eliminated.\n\n### 1.3 Critical Checkpoints (Self-Review and Review Priority)\n\nThe following checkpoints must be **individually confirmed with conclusions** during self-review and review:\n\n- What is the goal of the current task? Does the current code accomplish this goal? Is there a test that proves it?\n- Is this modification as small, clear, and focused as possible?\n- Does the code segment involve concurrency? If yes:\n - Which threads introduce concurrency, what are the critical variables? What locks protect them?\n - Are operations within locks as lightweight as possible? Are all heavy operations outside locks while maintaining concurrency safety?\n - If multiple locks exist, is the locking order consistent? Is there deadlock risk?\n- Is there special or non-intuitive lifecycle management? If yes:\n - Understand the complete lifecycle and relationships of related variables. Are there circular references? When is each released? Can all lifecycles end normally?\n- Are configuration items added?\n - Should the configuration allow dynamic changes, and does it actually allow them? If yes:\n - Can affected processes detect the change promptly without restart?\n- Does it involve incompatible changes like function symbols or storage formats? If yes:\n - Is compatibility code added? Can it correctly handle requests during rolling upgrades?\n- Are there functionally parallel code paths to the modified one? If yes:\n - Should this modification be applied to other paths, and has it been?\n- Are there special conditional checks? If yes:\n - Does the condition have clear comments explaining why this check is necessary, simplest, and most viable?\n - Are there similar paths with similar issues?\n- What is the test coverage?\n - Are end-to-end functional tests comprehensive?\n - Are there comprehensive negative test cases from various angles?\n - For complex features, are key functions sufficiently modular with unit tests?\n- Does the feature need increased observability? If yes:\n - When bugs occur, are existing logs and VLOGs sufficient to investigate key issues?\n - Are INFO logs lightweight enough?\n - Are Metrics added for critical paths? Referencing similar features, are all necessary observability values covered?\n- Does it involve transaction and persistence-related modifications? If yes:\n - Are all EditLog reads and writes correctly covered? Is all information requiring persistent storage persisted? Does it follow our standard approach?\n - In transaction processing logic, can master failover at any time point be handled correctly?\n- Does it involve data writes and modifications? If yes:\n - Are transactionality and atomicity guaranteed? Are there issues with concurrency?\n - Would FE or BE crashes cause leaks, deadlocks, or incorrect data?\n- Are new variables added that need to be passed between FE and BE? If yes:\n - Is variable passing modified in all paths? For example, various scattered thrift sending points like constant folding, point queries.\n- Analyze deeply from all angles (including but not limited to time complexity, anti-patterns, CPU and memory friendliness, redundant calculations, etc.) based on the code context and available information. Are there any performance issues or optimization opportunities?\n- Based on your understanding, are there any other issues with the current modification?\n\nAfter checking all the above items with code. Use the remaining parts of this skill as needed. The following content does not need individual responses; just check during the review process.\n\n#### 1.3.1 Concurrency and Thread Safety (Highest Priority)\n\n- [ ] **New thread/bthread entry**: Is `SCOPED_ATTACH_TASK` used? Omission causes memory to be counted in orphan tracker, debug builds crash directly\n- [ ] **Lock protection scope**: Is shared data (version map, delete bitmap, transaction state) accessed within correct locks? Is lock type correct (shared vs exclusive)?\n- [ ] **Lock order**: Are there nested acquisitions of multiple locks? Must follow known lock order (e.g., `_txn_lock` -> `_txn_map_lock`), otherwise deadlock\n- [ ] **Atomic operation memory order**: Use `relaxed` for statistics, at least `acquire/release` for flag/signal\n- [ ] **COW column modification**: Is exclusive ownership ensured before `mutate()`? Is `assume_mutable_ref()` only used when exclusive ownership is confirmed?\n\n#### 1.3.2 Error Handling (Highest Priority)\n\n- [ ] **Status checking**: `Status` return values **must** be checked, `static_cast(...)` silent discard is prohibited\n- [ ] **Exception propagation chain**: Is there `RETURN_IF_CATCH_EXCEPTION` above `THROW_IF_ERROR` to catch?\n- [ ] **Error messages**: Are they clear, include context (table/partition/TabletID/transaction ID), provide solution direction?\n- [ ] **DORIS_CHECK usage**: Only for invariant assertions (e.g., `DORIS_CHECK(hash_table != nullptr)`), not for defensive programming\n\n#### 1.3.3 Memory Safety (High Priority)\n\n- [ ] **Large memory reservation**: Is `try_reserve()` called before allocation? Is there `DEFER_RELEASE_RESERVED()` to ensure release?\n- [ ] **Cache Handle**: Is `cache->release(handle)` called after using `Cache::Handle*`?\n- [ ] **shared_ptr circular references**: Do scenarios like `Dependency` use raw pointers or `weak_ptr` to break cycles?\n- [ ] **Object creation**: Do classes with `ENABLE_FACTORY_CREATOR` use `create_shared()` / `create_unique()`? Raw `new` is prohibited\n\n#### 1.3.4 Data Correctness (High Priority)\n\n- [ ] **Delete Bitmap version**: In Cloud mode, is `TEMP_VERSION_COMMON` manually replaced before use?\n- [ ] **Version consistency**: Does data reading strictly not exceed Partition visible version?\n- [ ] **MoW setting**: Is `SegmentWriterOptions::enable_unique_key_merge_on_write` correctly set to `true`?\n- [ ] **EditLog persistence**: Do metadata modifications call `editLog.logXxx()`? Is replay logic strictly equivalent to master?\n\n#### 1.3.5 Observability (Medium Priority)\n\n- [ ] **Metrics addition**: Are `ADD_TIMER/ADD_COUNTER` added to critical paths? Does naming follow `module_xxx` convention?\n- [ ] **Log levels**: Use `LOG(ERROR)` for error logs, `LOG(INFO)` for critical paths, `VLOG` or `DVLOG` for debug info\n- [ ] **Trace information**: Do distributed operations (RPC, transactions) include trace ID for tracking?\n\n### 1.4 Test Coverage Principles\n\n- All kernel features **must** have corresponding tests\n- End-to-end SQL functionality should prioritize adding regression tests under `regression-test/` (Groovy DSL)\n- Also add BE unit tests (`be/test/`, Google Test) and FE unit tests (`fe/fe-core/src/test/`, JUnit 5) where possible. For complex features, must be modular with unit tests.\n- Regression test result files (`.out`) **must not be handwritten**, must be auto-generated via `-genOut`\n- Regression tests must use `order_qt_` prefix or manual `ORDER BY` to ensure ordered results\n- Expected error cases use `test { sql \"...\"; exception \"...\" }` pattern\n- Tables should be **DROPped before use** (not after tests end), to preserve environment for debugging\n- Simple tests should hardcode table names directly, not use `def tableName`\n\n---\n\n## Part 2: BE Module Review Guide\n\n### 2.1 Error Handling (Highest Priority)\n\n#### 2.1.1 Status / Exception / Result\\ Relationship\n\n| Type | Usage | Propagation Macro |\n|------|-------|------------------|\n| `Status` | Default error return type, `[[nodiscard]]` | `RETURN_IF_ERROR` |\n| `Exception` | Vectorized execution engine hot path, expression evaluation | `RETURN_IF_CATCH_EXCEPTION` |\n| `Result` | Return value or error (replaces old output parameter pattern) | `DORIS_TRY` |\n\n**Review Checkpoints:**\n\n- [ ] Are `Status` return values checked? Search for `static_cast(...)` pattern—this is the **most dangerous anti-pattern**, silently swallowing errors without logging. Currently 171+ occurrences in codebase. Never allow this pattern in new code\n- [ ] Where `THROW_IF_ERROR` is used, is there `RETURN_IF_CATCH_EXCEPTION` or `RETURN_IF_ERROR_OR_CATCH_EXCEPTION` above in the call chain to catch? If not, exceptions propagate uncaught causing crashes. Current version has related catch in upper layers when Pipeline executes specific operators.\n- [ ] **`THROW_IF_ERROR` prohibited in Defer/RAII destructor lambdas**: Throwing exceptions during stack unwinding triggers `std::terminate`. Known issues: `pipeline_task.cpp:412-413, 638-639`. Use `WARN_IF_ERROR` or `static_cast` + logging in Defer\n- [ ] **`THROW_IF_ERROR` prohibited in Thrift RPC handlers without try/catch protection**: Thrift framework doesn't catch Doris exceptions, causing connection drops. Known issues: `backend_service.cpp:1323,1326`. RPC handlers should use `RETURN_IF_ERROR` pattern or outermost try/catch\n- [ ] Is `WARN_IF_ERROR` usage reasonable? Only allowed in destructors, cleanup paths, best-effort operations. Warning message must not be empty string\n- [ ] Is `DORIS_CHECK` used for **invariant assertions** (not defensive programming)? Correct usage example: `DORIS_CHECK(hash_table)` asserts hash table is initialized\n- [ ] Do new catch blocks correctly convert exceptions? Should follow standard pattern from `internal_service.cpp:348-357`: catch `doris::Exception` first, then `std::exception`, finally `...`\n\n#### 2.1.2 Cloud RPC Error Handling\n\nThe `retry_rpc` template in `cloud_meta_mgr.cpp` is the standard pattern for RPC error handling in Cloud mode:\n- [ ] Do new Cloud RPC calls use `retry_rpc`?\n- [ ] Does `INVALID_ARGUMENT` return directly without retry? (Correct behavior)\n- [ ] Does `KV_TXN_CONFLICT` use an independent conflict retry counter?\n- [ ] Do timeout and retry counts use config variables instead of hardcoding?\n\n#### 2.1.3 compile_check Mechanism\n\n`compile_check_begin.h` / `compile_check_end.h` elevate `-Wconversion` to **compile error** (excluding sign-conversion and float-conversion).\n\n- [ ] Do new `.h` files include paired `compile_check_begin.h` / `compile_check_end.h` in declaration regions?\n- [ ] **Coverage status**: Currently only 31.1% (1,049/3,372) of source files use compile_check. Additionally, 218 files contain `compile_check_begin.h` but lack corresponding `compile_check_end.h` (unpaired). New files should always include complete pairing\n- [ ] Are there implicit narrowing conversions? Pay special attention to `int64_t -> int32_t`, `size_t -> int`, `uint64_t -> int64_t`\n- [ ] If bypassing due to third-party code is necessary, are `compile_check_avoid_begin.h` / `compile_check_avoid_end.h` used?\n\n### 2.2 Memory Management\n\n#### 2.2.1 MemTracker Hierarchy\n\nDoris uses two-level memory tracking:\n- **`MemTrackerLimiter`**: Heavy-weight tracker with limits (QUERY / LOAD / COMPACTION / SCHEMA_CHANGE / METADATA / CACHE types)\n- **`MemTracker`**: Lightweight tracker without limits, for fine-grained sub-component tracking\n\nThread-level tracking is implemented via `ThreadMemTrackerMgr`, each thread has one active `MemTrackerLimiter` and a `MemTracker` stack.\n\n**Review Checkpoints:**\n\n- [ ] Do newly started threads/bthreads use `SCOPED_ATTACH_TASK` at entry? Without it, memory is counted in orphan tracker, debug builds trigger assertions\n- [ ] Does temporary tracker switching use `SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER` (supports nesting)?\n- [ ] **`ThreadPool::submit_func` callbacks**: Do lambdas submitted to thread pools use `SCOPED_ATTACH_TASK` internally? Currently 87+ `submit_func` calls in codebase, many rely on \"indirect attach\" (attach happens deep in call chain), which is fragile. Correct approach references gold-standard wrapper pattern from `calc_delete_bitmap_executor.h:67-83`: explicit attach at callback entry. Known problematic files: `s3_file_bufferpool.cpp:114`, `wal_manager.cpp:414`, `download_action.cpp:166`\n- [ ] Is `try_reserve()` called before large memory allocation? Is there `DEFER_RELEASE_RESERVED()` to ensure release after allocation?\n- [ ] Does memory hook path (`ThreadMemTrackerMgr::consume`) avoid operations that may trigger memory allocation (e.g., LOG, string formatting)? `_stop_consume` flag exists to prevent recursion\n\n#### 2.2.2 Object Lifecycle\n\n- [ ] Are classes using `ENABLE_FACTORY_CREATOR` created via `create_shared()` / `create_unique()`? Raw `new` is prohibited\n- [ ] Are there `shared_ptr` circular references? Standard ways to break cycles:\n - `Dependency::_shared_state` uses **raw pointer** (non-owning)\n - `Dependency::_blocked_task` uses `vector>`\n - `TrackerLimiterGroup::trackers` uses `list>`\n- [ ] Is Rowset's dual reference counting used correctly? `acquire()` / `release()` is manual reference counting (for performance), `shared_ptr` manages ownership. When `release()` decrements to 0 and in UNLOADING state, triggers `do_close()`\n- [ ] Are rowsets in `_unused_rowsets` judged deletable via `use_count() == 1`?\n- [ ] Are there suspicious `shared_ptr` circular reference issues? For clear lifecycles, prefer `unique_ptr` + raw pointer observer pattern.\n\n#### 2.2.3 COW Column Semantics\n\nVectorized columns (`IColumn`) use custom intrusive reference-counted Copy-on-Write mechanism:\n\n- [ ] Is exclusive ownership ensured when calling `mutate()`? When `use_count() > 1`, triggers deep copy, which is a performance hotspot\n- [ ] Is `assume_mutable_ref()` only used when exclusive ownership is confirmed? This method **does not check reference count**, using it in shared state causes data corruption\n- [ ] Is `set_columns()` called to restore after `Block::mutate_columns()`? After calling, original Block's column pointers are null\n- [ ] Can the pointer returned by `convert_to_full_column_if_const()` share with original column? For `ColumnConst` it materializes a new column, but for normal columns returns shared pointer\n\n#### 2.2.4 Cache Lifecycle\n\n- [ ] Is `cache->release(handle)` called after using `Cache::Handle*`? Omission causes memory leaks\n- [ ] Do cache values inherit from `LRUCacheValueBase`? This base class automatically releases tracking bytes on destruction\n- [ ] Are new cache types registered with `CacheManager`? Unregistered caches don't participate in global GC\n\n### 2.3 Concurrency and Locks\n\n#### 2.3.1 Tablet Lock Hierarchy\n\nThe `Tablet` class has 8+ independent locks, with confirmed usage rules:\n\n| Lock | Type | Rule |\n|----|------|------|\n| `_meta_lock` | `shared_mutex` | Modifying version map requires EXCLUSIVE; reading requires SHARED |\n| `_rowset_update_lock` | `mutex` | Serializes delete bitmap updates during MoW table publish_txn, **independent of _meta_lock** (avoids blocking read path) |\n| `_base_compaction_lock` | `mutex` | Serializes base compaction |\n| `_cumulative_compaction_lock` | `mutex` | Serializes cumulative compaction |\n| `_migration_lock` | `shared_timed_mutex` | Protects tablet migration |\n| `_cooldown_conf_lock` | `shared_mutex` | Protects cooldown configuration |\n\n**Review Checkpoints:**\n\n- [ ] Do operations on `_rs_version_map` or `_stale_rs_version_map` hold `_meta_lock` (shared or exclusive)?\n- [ ] Do `TabletMeta` modification operations hold the owning Tablet's EXCLUSIVE `_meta_lock`?\n- [ ] Is there acquisition of other tablet locks while holding `_meta_lock`? This may cause deadlock\n\n#### 2.3.2 TxnManager Lock Order\n\nConfirmed consistent lock order: `_txn_lock` -> `_txn_map_lock` (applies to commit, publish, delete, rollback).\n- [ ] Do new TxnManager operations follow this lock order?\n\n#### 2.3.3 Pipeline Dependency Concurrency\n\n`Dependency` uses double-checked locking + atomic `_ready` flag + mutex `_task_lock`:\n- `set_ready()`: First check `_ready` (fast path), then lock to set `_ready = true` and wake up\n- `is_blocked_by()`: Lock, check `_ready`, add to wait list if not ready\n\n- [ ] Do new `Dependency` subclasses correctly call `block()` / `set_ready()`?\n- [ ] Do `CountedFinishDependency`'s `add()` / `sub()` operate under `_mtx` protection?\n\n#### 2.3.4 Atomic Operation Memory Order\n\nMemory order usage patterns in codebase:\n- `memory_order_relaxed`: Statistics counters (correct)\n- `memory_order_acquire/release`: Lifecycle flags like `ExecEnv::_s_ready` (correct)\n- `memory_order_acq_rel`: CAS close operations (correct)\n\n- [ ] Do new atomic operations use appropriate memory order? Use relaxed for statistics counters, at least acquire/release for flag/signal\n- [ ] Does `std::atomic` as stop signal use at least acquire/release? Relaxed may theoretically delay propagation\n\n### 2.4 Type System and Serialization\n\n#### 2.4.1 Upgrade/Downgrade Compatibility\n\nUpgrade/downgrade compatibility no longer uses be_exec_version for control. Instead, each feature (e.g., functions, compression algorithms, etc.) adds its own option for confirmation. BE uses is_set&&is_true to branch or replace functionality at appropriate nodes.\n\n#### 2.4.2 Decimal Precision Overflow\n\n- [ ] Do Decimal operation results check precision upper limit? `decimal_result_type()` may promote result to DECIMAL256, causing unexpected memory layout changes\n- [ ] Are DecimalV2's `original_precision` / `original_scale` correctly set? Default value `UINT32_MAX` means \"unknown\", `get_format_scale()` returns different value from `scale` in this case\n\n#### 2.4.3 Thrift Type Mapping\n\n- [ ] Are `TScalarType` optional fields (`precision`, `scale`, `len`) set when needed? DECIMAL needs precision/scale, CHAR/VARCHAR needs len, DATETIMEV2 needs scale\n- [ ] Is depth-first flattened representation of `TTypeDesc.types` correctly traversed? Off-by-one errors are common with nested complex types (ARRAY>)\n\n#### 2.4.4 Block Merge Nullable Handling\n\n`MutableBlock::merge_impl()` assumes target is Nullable and source is non-Nullable when types mismatch, using C-style cast `(DataTypeNullable*)` instead of `dynamic_cast`.\n\n- [ ] Does new Block merge logic correctly handle Nullable promotion? DCHECK only protects in debug builds\n\n### 2.5 Pipeline Execution Engine\n\n#### 2.5.1 Operator Lifecycle\n\nPipeline Task state machine: `INITED -> RUNNABLE -> BLOCKED -> FINISHED -> FINALIZED`\n\n- [ ] Are `SharedState`'s source_deps and sink_deps correctly connected? `inject_shared_state()` is responsible for connection\n\n#### 2.5.2 Memory Reservation and Spill\n\n- [ ] Do memory-intensive operations (Hash Join build, aggregation) use `_try_to_reserve_memory()`?\n- [ ] Is there `_memory_sufficient_dependency` for backpressure when memory is insufficient?\n- [ ] Does `revoke_memory()` correctly implement spill logic?\n\n### 2.6 Storage Engine\n\n#### 2.6.1 Rowset Version Management\n\n- [ ] Are `add_rowset()` and `modify_rowsets()` executed under exclusive `_meta_lock`?\n- [ ] Are version ranges continuous? There should be no version gaps in `_rs_version_map`\n- [ ] Does compaction output rowset's version range correctly cover input rowsets?\n\n#### 2.6.2 Delete Bitmap (MoW Tables, Extremely High Priority)\n\n- [ ] In Cloud mode, is **TEMP_VERSION_COMMON manually replaced** for bitmaps obtained from `CloudTxnDeleteBitmapCache`? Code comments explicitly warn about this issue (`cloud_txn_delete_bitmap_cache.h:72-75`). Omission causes bitmap to be applied to wrong version\n- [ ] Is `_rowset_update_lock` held during delete bitmap calculation? Prevents concurrent calculation on same (txn_id, tablet_id)\n- [ ] Does compaction's delete bitmap calculation use **latest** compaction stats (`ms_base_compaction_cnt`, etc.)? Stale stats will miss recently compacted rowsets\n\n#### 2.6.3 Segment Writing\n\n- [ ] Is `SegmentWriterOptions::enable_unique_key_merge_on_write` correctly set for MoW tables? Default value is `false`, omission causes deduplication to be silently skipped\n- [ ] Is performance impact of row-stored columns (`_serialize_block_to_row_column`) on wide tables within expectations?\n\n### 2.7 IO Subsystem\n\n- [ ] Do `FileSystem` operations use `FILESYSTEM_M` macro to handle bthread async IO?\n- [ ] Do remote storage (S3/HDFS) reads use `create_cached_file_reader()` for file caching?\n\n### 2.8 Inverted Index Subsystem\n\n#### 2.8.1 Lifecycle and Destruction Order\n\n- [ ] **Field declaration order is destruction order**: In `inverted_index_writer.h:92-95`, `_dir` must be declared **before** `_index_writer`, because `_index_writer`'s `write.lock` is created by `_dir.lockFactory`, C++ destructs in reverse declaration order. New fields must follow this constraint\n- [ ] **Cache handle destruction extends LRU retention**: `inverted_index_cache.h:142-152`, `InvertedIndexCacheHandle` destructor sets `last_visit_time` to future timestamp (`UnixMillis() + config::index_cache_entry_stay_time_after_lookup_s * 1000`), making actually queried searchers survive longer in LRU. Be aware of this extension mechanism when modifying cache logic\n\n#### 2.8.2 CLucene Exception Handling\n\nCLucene doesn't use C++ exceptions, `inverted_index_common.h:65-111` implements `ErrorContext` + FINALLY macro system:\n- `FINALLY`: Catches exceptions and returns `Status` error\n- `FINALLY_EXCEPTION`: Catches and rethrows\n- All three macros use `static_assert` to enforce existence of `error_context` local variable in scope\n\n- [ ] Does new CLucene-related code use `ErrorContext` + `FINALLY` pattern instead of raw try/catch?\n\n#### 2.8.3 Three-Valued Logic Bitmap\n\n`InvertedIndexResultBitmap` (`inverted_index_reader.h:80-214`) implements SQL three-valued logic (TRUE/FALSE/NULL), storing separately via `_data_bitmap` + `_null_bitmap`:\n- [ ] **`op_not` is const but actually modifies data**: Modifies pointed object via `shared_ptr` to bypass const qualifier. Callers should not assume const reference returned results are immutable\n- [ ] `operator|=` implements complete SQL three-valued OR semantics (lines 148-157), same for `operator&=` and `operator-=`. New bitmap operations must follow three-valued logic\n\n### 2.9 Schema Change Subsystem\n\n#### 2.9.1 Three-Strategy Selection and Lock Order\n\n`_get_sc_procedure()` (`schema_change.h:303-315`) selects among three strategies based on change type:\n- `LinkedSchemaChange`: Column name/type alias changes only, zero-copy\n- `VSchemaChangeDirectly`: Direct conversion without sorting\n- `VLocalSchemaChangeWithSorting`: Full rewrite with sorting\n\n**Lock acquisition order** (`schema_change.cpp:972-976`): `base_tablet push_lock → new_tablet push_lock → base_tablet header_lock → new_tablet header_lock`\n\n- [ ] Do schema change lock acquisitions follow this four-level order? Out-of-order causes deadlock\n\n#### 2.9.2 MOW Delete Bitmap Four-Step Flow (Extremely High Priority)\n\n`schema_change.cpp:1595-1657` describes critical correctness flow for MOW table schema change:\n1. Newly imported rowsets during dual-write **skip** delete bitmap calculation\n2. After conversion completes, calculate delete bitmap for rowsets during dual-write period (lines 1618-1634)\n3. **Block new publish**, calculate delete bitmap for incremental rowsets (lines 1636-1652)\n4. Switch tablet state to `TABLET_RUNNING` (lines 1653-1656)\n\n- [ ] When modifying schema change delete bitmap path, are the four-step order and blocking logic complete? Step 3's blocking is key to ensuring consistency\n\n#### 2.9.3 Column Deletion Order Trap\n\n`return_columns` is calculated **before** `merge_dropped_columns()` (`schema_change.cpp:959-964` vs line 1062), this is intentional: `return_columns` only covers non-dropped columns, dropped columns are handled internally by `SegmentIterator` for delete predicate evaluation.\n\n- [ ] When modifying schema change column processing logic, is the timing of `return_columns` and dropped columns calculation understood?\n- [ ] `_check_row_nums` can be bypassed by `config::ignore_schema_change_check` (default false) (`schema_change.h:155-171`), this config should remain off in production\n\n### 2.10 Workload Group Memory Tracking\n\n`WorkloadGroup` (`workload_group.h`) uses dual-lock pattern: `_mutex` protects name/version/memory limit/resource_ctxs/shutdown state, `_task_sched_lock` protects scheduler and cgroup pointers.\n\n- [ ] **Memory tracking inconsistency**: `check_mem_used()` (lines 115-123) calculates `_total_mem_used + _wg_refresh_interval_memory_growth` for watermark check, but `exceed_limit()` (lines 132-135) **only compares `_total_mem_used > _memory_limit`**, ignoring `_wg_refresh_interval_memory_growth`. This means during refresh interval, workload group may exceed hard limit without being detected by `exceed_limit()`\n- [ ] Does new memory check logic use the correct method? If precise limit checking is needed, should reference `check_mem_used()` approach\n\n---\n\n## Part 3: FE Module Review Guide\n\n### 3.1 Metadata Persistence (EditLog)\n\n#### 3.1.1 Write Path\n\nAll metadata modifications must go through EditLog persistence. Flow:\n1. Modify in-memory state\n2. Call `editLog.logXxx()` to write log\n3. Log is replicated to Follower via BDBJE\n4. Caller blocks until persistence confirmation\n\n**Review Checkpoints:**\n\n- [ ] Do new DDL/DML operations call corresponding `editLog.logXxx()` methods? Omission causes Follower state inconsistency\n- [ ] Do new OperationTypes have corresponding replay handling in `EditLog.loadJournal()` switch?\n- [ ] Is replay logic **strictly equivalent** to master execution logic? Any difference causes Follower state drift\n- [ ] Do new serializable objects correctly implement Gson serialization? Are `@SerializedName` annotations complete?\n- [ ] Can Image checkpoint correctly include new metadata?\n- [ ] Can master failover at any moment guarantee no duplicate or lost data? Short-term task failures are tolerable, but memory/transaction leaks and data inconsistencies are not allowed.\n\n#### 3.1.2 EditLog Batch and Outside-Lock Commit Mode\n\n`DatabaseTransactionMgr` implements two EditLog commit modes, controlled by `Config.enable_txn_log_outside_lock`:\n\n- **Inside-lock mode** (default): Directly calls `editLog.logInsertTransactionState()` within transaction lock\n- **Outside-lock mode**: Only modifies in-memory state and calls `submitEdit()` to submit to queue within lock, waits for persistence completion via `awaitTransactionState()` outside lock (`DatabaseTransactionMgr.java:383-395`)\n\nFuzzy testing (`DorisFE.java:612-615`) randomly switches between these two modes, so both paths must be correct.\n\n**Review Checkpoints:**\n\n- [ ] Do modifications involving transaction EditLog consider both inside-lock and outside-lock commit paths?\n- [ ] In outside-lock mode, are `awaitTransactionState()` timeout and exception handling correct? Timeout doesn't mean failure—transaction may still be committing\n- [ ] Do new transaction state modifications complete all in-memory state changes before `submitEdit()` submission? After submission, other threads may immediately see new state\n\n#### 3.1.3 EditLog Fatal Error Handling\n\nWhen EditLog flush fails, FE calls `System.exit(-1)` to terminate process—this is **intentional**, preventing split-brain.\n- [ ] Does new code catch and swallow exceptions in EditLog-related paths? This may cause metadata loss\n\n### 3.2 Lock Hierarchy and Concurrency\n\n#### 3.2.1 Confirmed Lock Orders\n\n| Scenario | Lock Order |\n|------|------|\n| ConsistencyChecker | `jobs_lock` -> `synchronized(job)` -> `db_lock` |\n| AlterJobV2 | `synchronized(job)` -> `db_lock` |\n| TabletScheduler | `synchronized(this)` **internal** operations, then **release**, then acquire `db_lock` (two-step method avoids deadlock) |\n| DatabaseTransactionMgr | Its RWLock is **leaf lock**, should not acquire other locks internally |\n| MTMVTask | Acquire table locks after sorting by table ID (avoids deadlock) |\n\n**Review Checkpoints:**\n\n- [ ] Do new lock acquisitions follow the above lock orders?\n- [ ] Is `tryLock(timeout)` used instead of infinite-wait `lock()`? FE's standard pattern is `tryLock` + timeout logging\n- [ ] **Lock timeout standard values**: Env lock = 5s, DB write lock = 100s, Table read lock = 1min. Operations involving multiple tables must lock in ascending table ID order (implemented via `PriorityQueue`) to prevent deadlock\n- [ ] Does `Database.tryWriteLock()` log current owner thread on timeout?\n- [ ] Are there paths acquiring database lock within `synchronized` block? This is a known deadlock risk, should use two-step method\n\n#### 3.2.2 ConcurrentModificationException\n\n- [ ] Is traversal of shared collections under lock protection? `CatalogRecycleBin.write()` added `synchronized` because dump image API concurrent access caused CME\n- [ ] Can there be concurrent modifications during traversal of `Map` / `List`? Prefer `ConcurrentHashMap` or snapshot before traversal\n\n#### 3.2.3 Known Deadlock Cases (New Code Must Avoid Similar Patterns)\n\nThe following are confirmed deadlock risk points in codebase, be extra vigilant when reviewing similar patterns:\n\n| Location | Pattern | Description |\n|------|------|------|\n| `MasterCatalogExecutor.java:70-78` | Initiating RPC while holding catalog lock | RPC timeout blocks catalog operations for long time |\n| `ExternalDatabase.java:137-158` | Nested acquisition of db lock + makeSureInitialized() | Comments explicitly mark deadlock avoidance strategy |\n| `HiveMetaStoreCache.java:126-130` | Acquiring cache lock in thread pool callback | Comment marks \"this may cause a deadlock\" |\n| `CacheFactory.java:85-92` | Circular dependency in cache initialization | Avoided via lazy initialization |\n\n- [ ] Does new code initiate RPC or IO operations while holding locks? This is the most common source of deadlock and performance issues\n- [ ] Does new code acquire locks in callbacks (e.g., thread pool, cache loader) that callers may hold?\n\n### 3.3 Nereids Optimizer\n\n#### 3.3.1 Rule Implementation Specifications\n\n**Review Checkpoints:**\n\n- [ ] Is new Rule's `RuleType` added to `RuleType.java` enum?\n- [ ] Does Rule correctly handle **all Join types**? Pay special attention to ASOF variants (`ASOF_LEFT_INNER_JOIN`, etc.). `COULD_PUSH_THROUGH_LEFT/RIGHT` lists in `PushDownFilterThroughJoin` explicitly include ASOF types\n- [ ] Are constant predicates (`slots.isEmpty()`) correctly handled? Inner Join can push to both sides, Outer Join cannot\n- [ ] Is there **infinite loop risk**? Can created nodes be matched again by another rule in same batch? Code comments have multiple \"separate topDown passes to avoid dead loops\" warnings\n- [ ] Does `OneRewriteRuleFactory` pattern match correct wrapper nodes? For example, `EliminateJoinByFK` only matches `project(join)` not bare `join`\n- [ ] Are predicates checked for `containsUniqueFunction()`? Predicates containing unique functions cannot be pushed down\n\n#### 3.3.2 Rewriter Stage Order\n\nRewriter organizes rules into multi-stage pipeline, key dependencies:\n- `PUSH_DOWN_FILTERS` is called 5-6 times throughout pipeline, because it interacts with many other transformations\n- `InferPredicates` is called twice (with `PUSH_DOWN_FILTERS` in between), because eliminating outer join exposes new inference opportunities\n- `costBased(...)` rules are controlled by `runCboRules` flag\n\n- [ ] Is new rewrite rule placed in correct stage? For example, rules needing predicate pushdown should be after `PUSH_DOWN_FILTERS`\n- [ ] Is new exploration rule registered in `RuleSet`?\n\n#### 3.3.3 Property Derivation\n\n`RequestPropertyDeriver` implements top-down property requests:\n\n- [ ] Does new Physical Operator have visit method in `RequestPropertyDeriver`?\n- [ ] Are multiple alternative property requirements provided (e.g., HashJoin provides both shuffle and broadcast choices)? Single requirement narrows search space\n- [ ] Is `PhysicalProject` alias penetration logic correct? Only `Alias(SlotRef)` and bare `SlotRef` can penetrate\n\n### 3.4 Transaction Management\n\n#### 3.4.1 Transaction Lifecycle\n\nState machine: `PREPARE -> COMMITTED -> VISIBLE` (or `ABORTED`)\n\n- [ ] After transaction commit, can `PublishVersionDaemon` correctly send `PublishVersionTask` to all BEs?\n- [ ] Does publish wait for quorum confirmation?\n- [ ] Does transaction correctly abort after timeout?\n\n**Known race condition—`TransactionState.tableIdList`**: At `TransactionState.java:632`, TODO comment marks that `addTableId()` / `removeTableId()` lack synchronization protection. `tableIdList` is plain `ArrayList`, has race condition (CME or data loss) when multiple threads concurrently add/remove table IDs.\n\n- [ ] Are operations on `TransactionState.tableIdList` under external lock protection? Or should it be replaced with thread-safe collection?\n\n#### 3.4.2 Cloud Mode Differences\n\nIn Cloud mode, transactions are centrally managed by Meta-Service, not FE:\n- [ ] Do Cloud mode transaction operations use `CloudGlobalTransactionMgr` instead of local `GlobalTransactionMgr`?\n- [ ] **Unsafe downcasting prohibited**: Current code has 4 places forcibly casting `GlobalTransactionMgrIface` to `CloudGlobalTransactionMgr` (e.g., `InternalCatalog.java:994`). These casts cause ClassCastException in non-Cloud mode. New code should add required methods by extending `GlobalTransactionMgrIface` interface, not rely on downcasting\n- [ ] Does version retrieval use `getVisibleVersion()` (RPC + cache in Cloud mode)?\n- [ ] Is `getVisibleVersion()`'s `RpcException` correctly handled? Multiple callers catch and return 0, may cause MV refresh misjudgment\n\n### 3.5 Exception System\n\n```\nUserException (checked)\n ├── AnalysisException -- SQL analysis error\n ├── DdlException -- DDL execution failure\n ├── MetaNotFoundException -- Metadata lookup failure\n ├── LoadException -- Load failure\n └── ...\n\nNereidsException (unchecked, RuntimeException)\n ├── nereids/AnalysisException -- Nereids-specific, different from common/\n └── nereids/ParseException\n```\n\n- [ ] **Note Nereids has its own `AnalysisException`** (`nereids/exceptions/AnalysisException.java`, extends RuntimeException), which is a **different class** from `common/AnalysisException` (checked)\n- [ ] Does `ErrorReport.reportAnalysisException()` use correct `ErrorCode`? Custom Doris error codes start from 5000\n- [ ] Does new RPC error handling use `Status` pattern (`TStatusCode` / `PStatus`)?\n\n### 3.6 OlapTable Version Visibility\n\n- [ ] `getVisibleVersion()` behaves differently in Cloud vs non-Cloud mode:\n - Non-Cloud: Returns local `tableAttributes.getVisibleVersion()`\n - Cloud: RPC to meta-service, has cache TTL\n- [ ] Can Cloud mode's version cache TTL (`cloudTableVersionCacheTtlMs`) cause reading stale versions?\n- [ ] `VERSION_NOT_FOUND` is forcibly converted to version=1 (`PARTITION_INIT_VERSION`)—cannot distinguish \"never loaded\" from \"loaded to version 1\"\n\n### 3.7 External Catalog Concurrency Traps\n\n#### 3.7.1 Initialization Race Condition\n\n`ExternalCatalog.makeSureInitialized()` (`ExternalCatalog.java:354-379`) is synchronized, but has the following traps:\n\n- [ ] **`isInitializing` guard is dead code in `ExternalCatalog`**: Although code checks `isInitializing` (line 358), it's never set to `true` (only set to `false` in finally). Compare with `ExternalDatabase.makeSureInitialized()` (`ExternalDatabase.java:137-158`) which **correctly** sets `isInitializing = true`. This means ExternalCatalog has no reentrancy protection\n- [ ] **`lowerCaseToDatabaseName.clear()` race window**: `getFilteredDatabaseNames()` (line 504) first `clear()` then refills `ConcurrentMap`, this method is **not synchronized**. Between clear and refill completion, concurrent `get()` calls return null\n\n#### 3.7.2 resetToUninitialized() NPE Risk\n\n`resetToUninitialized()` (`ExternalCatalog.java:581-590`) sets `objectCreated` and `initialized` to false within synchronized block, then calls `onClose()` (lines 773-784) to set `executionAuthenticator` and `transactionManager` to null.\n\n- [ ] Concurrent queries that have passed `makeSureInitialized()` may be using these fields being set to null. synchronized only protects callers of other synchronized methods, not non-synchronized read paths already in execution\n\n### 3.8 Backup/Restore State Machine\n\n#### 3.8.1 BackupJob State Machine and EditLog Timing\n\nEditLog write timing in `BackupJob` state machine is **uneven**, this is the most important non-obvious convention:\n\n- [ ] **PENDING→SNAPSHOTING doesn't write EditLog** (`BackupJob.java:620-624`): After FE restart, state reverts to PENDING, all snapshot tasks are regenerated and resent. Don't add EditLog when modifying this transition—restart recovery relies on this behavior\n- [ ] **BarrierLog mechanism**: `prepareSnapshotTaskForOlapTableWithoutLock()` creates `BarrierLog` for each table and writes via `editLog.logBarrier()` (lines 648-657), returned `commitSeq` is stored in properties map, used to ensure snapshot consistency\n- [ ] **UPLOAD_INFO stage errors only retry, not cancel** (`BackupJob.java:509-514`): This is the final stage, code comments explicitly state continuous retry until timeout even when encountering unrecoverable errors. Don't call `cancelInternal()` in this stage when modifying error handling\n\n#### 3.8.2 RestoreJob state vs showState\n\n`RestoreJob` has two independent state fields (`RestoreJob.java:157,162`):\n- `state`: Serialized persistent actual state, master sets to FINISHED **before** writing EditLog\n- `showState`: Non-serialized, only for display, master sets to FINISHED **after** writing EditLog (line 2191)\n\n- [ ] This prevents follower from seeing FINISHED state before EditLog replication completes. After deserialization, `gsonPostProcess()` syncs `showState = state`. Must update both fields when modifying state transitions\n\n---\n\n## Part 4: Cloud Module Review Guide\n\n### 4.1 FDB Transaction Mode\n\nCloud module uses FoundationDB (FDB) as the sole metadata storage. All metadata operations must be completed within FDB transactions.\n\n#### 4.1.1 TxnKv Interface Specifications\n\n- [ ] Is `TxnErrorCode` return value from each `Transaction::get()` / `commit()` checked? Marked as `[[nodiscard]]`\n- [ ] Is `TXN_CONFLICT` correctly handled? `commit_txn_immediately` returns error without retry, `commit_txn_eventually` has lazy committer path\n- [ ] Can transaction exceed FDB's 10MB limit? Large transactions need to use `TxnLazyCommitter` for batched commits\n- [ ] Is `TXN_MAYBE_COMMITTED` correctly handled? This indicates commit may have succeeded or failed, requires idempotent retry\n\n#### 4.1.2 Key Encoding Specifications\n\nThree key spaces:\n- `0x01`: User space (mutable metadata)\n- `0x02`: System space (service registration, encryption keys)\n- `0x03`: Version space (multi-version metadata with versionstamp, for snapshots)\n\n- [ ] Do new keys use correct key space prefix?\n- [ ] Does key encoding use `encode_bytes()` (order-preserving byte encoding) and `encode_int64()` (big-endian + sign marker)?\n- [ ] Does compound key field order match query pattern? Range scan relies on key prefix\n\n#### 4.1.3 Commit Path\n\nTwo commit paths:\n- **`commit_txn_immediately`**: Single FDB transaction completes all operations (read txn info, move tmp_rowset, bump version, update stats)\n- **`commit_txn_eventually`**: Uses lazy committer for batched commits when transaction is too large\n\n- [ ] Are the three config switches for lazy commit (`is_2pc`, `enable_txn_lazy_commit`, `config::enable_cloud_txn_lazy_commit`) correctly combined?\n- [ ] Is fallback path from `commit_txn_immediately` to `commit_txn_eventually` (triggered by `TXN_BYTES_TOO_LARGE`) correct?\n\n**Known risk—`commit_txn` infinite loop**: The `commit_txn` entry in `meta_service_txn.cpp:3257-3341` uses `do { ... } while(true)` loop to dispatch `commit_txn_immediately` and `commit_txn_eventually`. When pending lazy-committed transactions are found, it retries, but this loop **has no independent retry count limit**. If lazy committer continuously fails to complete (e.g., FDB continuous conflicts), theoretically infinite loop.\n\n- [ ] Do modifications involving `commit_txn` dispatch loop consider maximum retry count or total timeout?\n\n#### 4.1.4 Versionstamp Usage\n\nTransaction ID is derived from FDB versionstamp (`get_txn_id_from_fdb_ts`).\n\n- [ ] Does code depending on FDB versionstamp correctly use `atomic_set_ver_key()` / `atomic_set_ver_value()`?\n- [ ] Is `get_versionstamp()` only called after `commit()` succeeds?\n\n### 4.2 Meta-Service RPC Specifications\n\n#### 4.2.1 RPC Preprocessing\n\nEach RPC uses `RPC_PREPROCESS` macro for preprocessing:\n- StopWatch timing\n- DORIS_CLOUD_DEFER sets response status and records metrics on return\n- `RPC_RATE_LIMIT` performs per-instance QPS throttling\n\n- [ ] Do new RPCs use `RPC_PREPROCESS` and `RPC_RATE_LIMIT`?\n- [ ] Is response `MetaServiceCode` correctly set?\n\n#### 4.2.2 MetaServiceProxy Automatic Retry\n\n`MetaServiceProxy` decorator automatically retries (exponential backoff) on following error codes:\n- `KV_TXN_STORE_*_RETRYABLE`, `KV_TXN_TOO_OLD`, `KV_TXN_CONFLICT`\n\n- [ ] Do new RPCs need to be idempotent? Retry means same request may execute multiple times\n- [ ] Do non-idempotent operations correctly handle `TXN_MAYBE_COMMITTED`?\n\n### 4.3 Recycler Safety Specifications\n\nRecycler is responsible for GC of expired/deleted data:\n\n- [ ] Before deleting rowset data, are associated transactions aborted first (`abort_txn_for_related_rowset`)? Prevents commit-after-delete data loss\n- [ ] Are recycle operations best-effort + next-round retry mode? Should not stop entire recycle flow due to single failure\n- [ ] Is packed file reference count recycling correct? (`recycle_packed_files`)\n\n### 4.4 Delete Bitmap (Cloud MoW, Extremely High Priority)\n\nCloud mode delete bitmap management fundamentally differs from shared-nothing mode:\n\n- [ ] Bitmaps in `CloudTxnDeleteBitmapCache` contain sentinel marks and `TEMP_VERSION_COMMON`, **must manually replace version** before use. This is a critical correctness constraint explicitly marked in code comments\n- [ ] Empty rowset marks returned by `is_empty_rowset()` are not actively cleared (rely on expiration cleanup), does this affect correctness?\n- [ ] **`CloudTxnDeleteBitmapCache` LRU eviction side effect**: When cache entry is LRU evicted, `publish_status` is reset to `INIT`, causing next access to **completely recalculate** delete bitmap (not incremental update). This is correct but expensive. Additionally, marks written by `mark_empty_rowset()` are never actively cleared—they only disappear when entire cache entry is evicted. On high-frequency MoW tables with insufficient cache capacity, causes repeated recalculation\n- [ ] Do modifications involving `CloudTxnDeleteBitmapCache` consider impact of state reset after eviction?\n- [ ] Are `CalcDeleteBitmapTask`'s compaction stats consistent with latest state in meta-service?\n\n---\n\n## Part 5: Cross-Module Concerns\n\n### 5.1 FE-BE Protocol Compatibility\n\n- [ ] Do new `TPlanNodeType`s have corresponding branches in BE's processing switch?\n- [ ] Can deprecated node types (e.g., `MERGE_JOIN_NODE`, `KUDU_SCAN_NODE`) still be gracefully handled?\n- [ ] If new variables are added, are they correctly set in all paths where FE sends execution plans to BE?\n\n### 5.2 Cloud Mode vs Shared-Nothing Mode\n\nCode extensively uses `Config.isNotCloudMode()` / `Config.isCloudMode()` branches.\n\n- [ ] Do new features consider both modes?\n- [ ] In Cloud mode, tablets are not bound to specific BE—any compute node can load any tablet's metadata\n- [ ] In Cloud mode, rowsets go through two-phase protocol (`prepare_rowset` -> `commit_rowset`), with recycle marker protection in between\n- [ ] In Cloud mode, partition version is centrally managed by Meta-Service, not FE\n\n### 5.3 Merge-on-Write Unique Key Tables\n\nMoW tables are one of the most complex data paths, require special attention during review:\n\n- [ ] Write path: Does `SegmentWriter::probe_key_for_mow()` correctly probe existing rowsets?\n- [ ] Publish path: Is delete bitmap calculation under `_rowset_update_lock` protection?\n- [ ] Query path: Is delete bitmap version-aligned before reading?\n- [ ] Compaction path: Does `calc_delete_bitmap_for_compaction()` correctly handle `allow_delete_in_cumu_compaction`?\n- [ ] Cloud mode: Does delete bitmap cache have expiration mechanism? Is `_unused_delete_bitmap` cleaned up promptly?\n\n### 5.4 Performance Hotspot Concerns\n\n- [ ] Does `SendBatchThreadPool` (64 threads, 102K queue) have saturation risk in heavy shuffle scenarios?\n- [ ] Does `SegmentPrefetchThreadPool` (max 2000 threads) create too many OS threads in intensive scan scenarios?\n- [ ] In Cloud mode, does label deduplication check perform linear scan on labels with many aborted transactions, causing performance cliff?\n- [ ] Is serialization overhead of `_serialize_block_to_row_column()` for wide tables within expectations?\n- [ ] COW columns trigger deep copy on `mutate()` when `use_count > 1`—are there unnecessary copies in high-concurrency scenarios?\n\n---\n\n## Part 6: Testing Review Guide\n\n### 6.1 Regression Test Specifications\n\n- [ ] Is `order_qt_` prefix or manual `ORDER BY` used to ensure ordered results?\n- [ ] Do expected error cases use `test { sql \"...\"; exception \"...\" }` pattern?\n- [ ] Are tables **DROPped before use** (`DROP TABLE IF EXISTS xxx`), not after test ends?\n- [ ] Do simple tests directly hardcode table names instead of using `def tableName`?\n- [ ] Do tests cover these seven scenarios: empty table, all null, mixed null, pure non-null, constant, constant null, nullable wrapper (`nullable()`)?\n- [ ] Are result files (`.out`) auto-generated via `-genOut`? **Handwriting prohibited**\n\n### 6.2 BE Unit Test Specifications\n\n- [ ] Are Google Test's `TEST` / `TEST_F` macros used?\n- [ ] Are tests independent? Not dependent on other tests' execution order\n- [ ] Are `ASSERT_*` (failure terminates test) and `EXPECT_*` (failure continues execution) used correctly?\n- [ ] Do memory-intensive tests use `MemTracker` to check for leaks?\n\n### 6.3 FE Unit Test Specifications\n\n- [ ] Is JUnit 5's `@Test` annotation used?\n- [ ] Do test method names clearly describe test scenarios?\n- [ ] Is `Assertions.assertXxx()` used instead of JUnit 4's static imports?\n- [ ] Do tests involving concurrency use `CountDownLatch` / `CyclicBarrier` for synchronization?\n\n### 6.4 FE-BE Integration Test Specifications\n---\n\n## Part 7: Known High-Risk Code Patterns Quick Reference\n\n| Pattern | Risk Level | Location | Description |\n|------|---------|------|------|\n| `static_cast(status_returning_call)` | **CRITICAL** | BE global (195+ places) | Silently swallows errors, no logging |\n| Delete bitmap TEMP_VERSION not replaced | **CRITICAL** | `cloud_txn_delete_bitmap_cache.h:72-75` | Data visibility error |\n| `be_exec_version` mismatch | **CRITICAL** | FE-BE communication all paths | Serialization format incompatibility causes data corruption |\n| `(DataTypeNullable*)` C-style cast | **HIGH** | `block.h merge_impl` | No protection in release builds |\n| `sizeof(size_t*)` vs `sizeof(size_t)` | **HIGH** | `data_type.cpp:195` | Happens to work on 64-bit, crashes on 32-bit |\n| Lazy commit three-switch combination | **HIGH** | `meta_service_txn.cpp` | Missing config causes large transaction failure |\n| DecimalV2 original_precision default UINT32_MAX | **HIGH** | `data_type_decimal.h` | Affects string representation correctness |\n| `MOW` default false | **MEDIUM** | `segment_writer.h:70` | Omission causes deduplication to be silently skipped |\n| Cloud version cache TTL | **MEDIUM** | `OlapTable.getVisibleVersion()` | May read stale version |\n| Label scan linear complexity | **MEDIUM** | `meta_service_txn.cpp` | Many aborted txns cause performance cliff |\n| `THROW_IF_ERROR` no outer catch | **MEDIUM** | vec/ execution engine | Uncaught exception propagation causes crash |\n| `THROW_IF_ERROR` in Defer lambda | **CRITICAL** | `pipeline_task.cpp:412-413, 638-639` | Throwing exception during stack unwinding causes `std::terminate` |\n| `THROW_IF_ERROR` in Thrift RPC handler | **HIGH** | `backend_service.cpp:1323,1326` | No try/catch protection, connection drops |\n| `DCHECK` disappears in RELEASE builds | **HIGH** | BE global (2,872 places) | No invariant protection in production, should migrate","frontmatter":{"name":"code-review","description":"Create consistent releases and changelogs","compatibility":"opencode"},"isInternal":false,"tokens":11577,"sizeBytes":50072},{"name":"readme.md","path":"be/src/io/tools/readme.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/be/src/io/tools/readme.md","title":"tools Documentation","category":"plugin-manifest","format":"markdown","content":"# tools Documentation\n\nPath: `be/src/io/tools/readme.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/be/src/io/tools/readme.md)","tokens":300,"sizeBytes":165},{"name":"plugin.json","path":"extension/DataX/doriswriter/src/main/resources/plugin.json","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/extension/DataX/doriswriter/src/main/resources/plugin.json","title":"resources Plugin Manifest","category":"plugin-manifest","format":"json","content":"# resources Plugin Manifest\n\nPath: `extension/DataX/doriswriter/src/main/resources/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/extension/DataX/doriswriter/src/main/resources/plugin.json)","tokens":300,"sizeBytes":237},{"name":"README.md","path":"fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md","title":"fe-authentication-plugin-ldap Documentation","category":"plugin-manifest","format":"markdown","content":"# fe-authentication-plugin-ldap Documentation\n\nPath: `fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md)","tokens":300,"sizeBytes":311},{"name":"README.md","path":"fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md","title":"fe-authentication-plugin-password Documentation","category":"plugin-manifest","format":"markdown","content":"# fe-authentication-plugin-password Documentation\n\nPath: `fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md)","tokens":300,"sizeBytes":323},{"name":"README.md","path":"tools/clickbench-tools/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/clickbench-tools/README.md","title":"clickbench-tools Documentation","category":"plugin-manifest","format":"markdown","content":"\n\n# Usage\n\n**These scripts are used to do [ClickBench](https://benchmark.clickhouse.com/) test, more info [there](https://github.com/ClickHouse/ClickBench).**\n\n## follow the steps below:\n\n### 1. create table\n ./create-clickbench-table.sh\n\n### 2. load data\n ./load-clickbench-data.sh\n\n### 3. run queries\n ./run-clickbench-queries.sh\n","isInternal":false,"tokens":256,"sizeBytes":1104},{"name":"README.md","path":"tools/coffeebench-tools/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/coffeebench-tools/README.md","title":"coffeebench-tools Documentation","category":"plugin-manifest","format":"markdown","content":"\n\n## Usage\n\nThese scripts are used to make coffee-bench test.\nfollow the steps below:\n\n### 1. create tables. modify `conf/doris-cluster.conf` to specify doris info, then run script below.\n\n ./bin/create-tables.sh\n\n### 2. load coffee-bench data. use -s for scale. \"500m/1b/5b\"\n\n ./bin/load-data.sh -s 500m\n\n **Note: The data loading process uses S3 data located in the US East (us-east-1) region.**\n If you cannot access this region due to network restrictions or other reasons, you can:\n 1. Use AWS CLI to copy the data to an S3 bucket in your preferred region\n 2. Download the data locally and use alternative methods to import it into Doris\n\n### 3. run queries.\n\n ./bin/run-queries.sh\n\n NOTICE: At present, we rewrite some queries in coffee bench to adapt to Doris' execution framework, but it does not affect the correctness of the results.\n","isInternal":false,"tokens":385,"sizeBytes":1631},{"name":"README.MD","path":"tools/cost_model_evaluate/README.MD","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/cost_model_evaluate/README.MD","title":"cost_model_evaluate Documentation","category":"plugin-manifest","format":"markdown","content":"\n\nThis code is used to evaluate the cost model in doris.\nYou can config the query in config of main.py \n\nBefore running, you should install the libraries in requirements.txt","isInternal":false,"tokens":198,"sizeBytes":937},{"name":"README.md","path":"tools/emr_storage_regression/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/emr_storage_regression/README.md","title":"emr_storage_regression Documentation","category":"plugin-manifest","format":"markdown","content":"\n\n# Data Lake Regression Testing Tool For External Table\n\nUsed to test the doris external table on object storage for cloud vendors\n\n> Supported storage formats: HDFS, Alibaba Cloud OSS, Tencent Cloud COS, Huawei Cloud OBS\n\n> Supported data lake table formats: Iceberg\n\nThe following provides the example of the command line options:\n\n```\nsh tools/emr_storage_regression/emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\n```\nsh tools/emr_storage_regression/emr_tools.sh --case CASE --endpoint ENDPOINT --region REGION --service SERVICE --ak AK --sk SK --host HOST --user USER --port PORT\n```\n\nThe usage of each option is described below.\n\n## Connectivity Test\n\nWhen the `--case` option is set to `ping`, will check Doris's connectivity on EMR:\n\n- `--endpoint`, Object Storage Endpoint.\n\n- `--region`, Object Storage Region.\n\n- `--ak`, Object Storage Access Key.\n\n- `--sk`, Object Storage Secret Key.\n\n- `--host`, Doris Mysql Client IP.\n\n- `--user`, Doris Mysql Client Username.\n\n- `--port`, Doris Mysql Client Port.\n\n- `--service`, EMR cloud vendors: ali(Alibaba), hw(Huawei), tx(tencent).\n\n### Environment Variables\n\nNeed modify the environment variable in `default_emr_env.sh`, the script will execute `source default_emr_env.sh` to make the environment variable take effect.\n\nIf environment variables are configured, you can run the test script directly with the following command:\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\n### The Script Execution Steps For Connectivity Test\n\n1. Create Spark and Hive tables on EMR\n2. Use Spark and Hive command lines to insert sample data\n3. Doris creates the Catalog for connectivity test \n4. Execute SQL for connectivity test: `ping.sql`\n\n### Alibaba Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service` to `ali`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint oss-cn-beijing-internal.aliyuncs.com --region cn-beijing --service ali --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log\n```\n\nAlibaba Cloud EMR also supports testing connectivity for both Doris with DLF metadata and Doris on OSS-HDFS storage.\n\n- The DLF metadata connectivity test needs to be performed on the EMR cluster where the DLF serves as the metadata store, Default value of `DLF_ENDPOINT` is `datalake-vpc.cn-beijing.aliyuncs.com`, configured at ping_test/ping_poc.sh.\n\n- To test the OSS-HDFS storage connectivity, need to [enable the HDFS service on the OSS storage and configure](https://www.alibabacloud.com/help/en/e-mapreduce/latest/oss-hdfsnew), Default value of `JINDO_ENDPOINT` is `cn-beijing.oss-dls.aliyuncs.com`, configured at ping_test/ping_poc.sh.\n\n### Tencent Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service` to `tx`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint cos.ap-beijing.myqcloud.com --region ap-beijing --service tx --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log\n```\n\n### Huawei Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service`to `hw`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint obs.cn-north-4.myhuaweicloud.com --region cn-north-4 --service hw --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log \n```\n\n## Performance Testing on Standard Test Set\n\nWhen the `--case` option is set to `data_set`, will test the query performance of Doris external table:\n\n- `--test` test data set: ssb, ssb_flat, tpch, clickbench and all. Default `all`.\n\n- `--service`, EMR cloud vendors: ali(Alibaba), hw(Huawei), tx(tencent).\n\n- `--host`, Doris Mysql Client IP.\n\n- `--user`, Doris Mysql Client Username.\n\n- `--port`, Doris Mysql Client Port.\n\n### Environment Variables\n\nJust modify the above environment variable in `default_emr_env.sh`, the script will execute `source default_emr_env.sh` to make the environment variable take effect.\n\nIf environment variables are configured, you can run the test script directly with the following command:\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\n### Prepare Data\n\n1. To run the standard test set using the `emr_tools.sh` script, you need to rewrite the object storage bucket specified by the `BUCKET` variable, and then prepare data in advance and put them under the bucket. The script will generate table creation statements based on the bucket.\n\n2. Now the `emr_tools.sh` script supports iceberg, parquet and orc data for ssb, ssb_flat, tpch, clickbench.\n\n### Execution Steps\n\n1. After the connectivity test, the Doris Catalog corresponding to the standard test set is created\n2. Prepare the test set data based on the object storage bucket specified by the `BUCKET` variable\n3. Generate Spark table creation statements and create Spark object storage tables on EMR\n4. Create the spark table in the local HDFS directory: `hdfs:///benchmark-hdfs`\n5. You can choose to analyze Doris tables ahead of time and manually execute the statements in `analyze.sql` in the Doris Catalog\n6. Execute standard test set scripts: `run_standard_set.sh`\n\n### Standard data set: ssb, ssb_flat, tpch, clickbench\n\n- Full test. After executing the test command, Doris will run ssb, ssb_flat, tpch, clickbench tests in sequence, and the test results will include the cases on HDFS and on the object storage specified by `--service`.\n\n```\nsh emr_tools.sh --case data_set --service ali --host 127.0.0.1 --user root --port 9030 > log\n```\n\n- Specify a single test. `--test` option can be set to one of ssb, ssb_flat, tpch and clickbench.\n\n```\nsh emr_tools.sh --case data_set --test ssb --service ali --host 127.0.0.1 --user root --port 9030 > log\n```\n","isInternal":false,"tokens":1623,"sizeBytes":6440},{"name":"README.md","path":"tools/lzo/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/lzo/README.md","title":"lzo Documentation","category":"plugin-manifest","format":"markdown","content":"\n\n# LZO Writer\n\nThis is simple LZO writer, you can use it to generate lzo compressed file.\n\nSee `main()` for details\n\n## Compilation\n\n```\nsh build.sh\n```\n\n## Generate file\n\n```\n./lzo_writer file_name.lzo\n```\n\n","isInternal":false,"tokens":220,"sizeBytes":972},{"name":"README.md","path":"tools/pipeline-tracing/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/pipeline-tracing/README.md","title":"pipeline-tracing Documentation","category":"plugin-manifest","format":"markdown","content":"\n\n# Pipeline Tracing and Display Tool\n\nIn the Pipeline execution engine, we split the execution plan tree of each Instance into multiple small Pipeline Tasks and execute them under our custom Pipeline scheduler. Therefore, in an environment with a large number of Pipeline Tasks executing, how these Tasks are scheduled across threads and CPU cores is an important factor for execution performance. We have developed a specialised tool to observe the scheduling process on a particular query or time period, which we call \"Pipeline Tracing\".\n\nThis tool converts record files to proper JSON format for visualization.\n\n## How to Use\n\n```shell\npython3 origin-to-show.py -s -d .json\n```\nto transfer record file `` to `.json`. Then it could be visualized.\n\n```shell\npython3 origin-to-show.py --help\n```\nfor help details.","isInternal":false,"tokens":338,"sizeBytes":1617},{"name":"README.md","path":"tools/row_to_column/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/row_to_column/README.md","title":"row_to_column Documentation","category":"plugin-manifest","format":"markdown","content":"\n\nSteps of row-to-column conversion:\n\n1. Modify the corresponding cluster configuration in `conf`.\n\n2. Run `python convert_row_to_column.py` to generate all row storage to column storage commands.\n\n3. If the script runs without any results, it means that there is no row storage table in the cluster, so nothing need to be done.\n\n4. If there is a table in the script that fails to obtain information.\n\n Please use `show create table` in the cluster to see the `storage_type` of the table. If it is row storage, please manually convert it.\n\n5. After the schema change is completed, the old schema needs to be deleted after 10 minutes.\n\n Please wait for a while to make sure that the data of the old schema has been deleted, and then upgrade.\n Otherwise, the BE process will exit with an error when it is automatically detected when it starts.\n\n6. Run `python convert_row_to_column.py` again to verify that there are no rows in the table before upgrading.\n","isInternal":false,"tokens":371,"sizeBytes":1727},{"name":"README.md","path":"tools/show_segment_status/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/show_segment_status/README.md","title":"show_segment_status Documentation","category":"plugin-manifest","format":"markdown","content":"\n\nThis tool is used to get the progress of all current table transitions \nduring the online `segment_2` function.\n\nCurrently, you can specify 3 dimensions (in the conf file) to view the \nresults, you can specify one of them individually, or you can customize \nthe combination (that is, specify multiple at the same time).\n\n# Note\nWe use MySQLdb python lib to fetch meta from FE, so you must install it.\n\nYou can get MySQLdb lib from https://pypi.python.org/pypi/MySQL-python, \nthen you can install it as follows:\n```\n$ tar zxvf MySQL-python-*.tar.gz\n$ cd MySQL-python-*\n$ python setup.py build\n$ python setup.py install\n``` \n\n# Steps\n1. Fill in the conf according to your cluster configuration, and specify \n the table or be you want to watch.\n2. Execute `python show_segment_status.py`\n\n# Example\n1. If you want to watch the process of a table named `xxxx`, you can specify \n `table_name = xxxx` in conf file\n\n2. If you want to watch the process on be whose be_id is `xxxx`, you can specify \n `be_id = xxxx` in conf file\n\n# Output Example Format\n\nThe first number is about segment v2, the latter one is the total count.\n\n```\n==========SUMMARY()===========\nrowset_count: 289845 / 289845\nrowset_disk_size: 84627551189 / 84627551189\nrowset_row_count: 1150899153 / 1150899153\n===========================================================\n==========SUMMARY(table=xxxx)===========\nrowset_count: 289845 / 289845\nrowset_disk_size: 84627551189 / 84627551189\nrowset_row_count: 1150899153 / 1150899153\n===========================================================\n==========SUMMARY(be=10003 )===========\nrowset_count: 79650 / 79650\nrowset_disk_size: 24473921575 / 24473921575\nrowset_row_count: 331449328 / 331449328\n===========================================================\n```\n","isInternal":false,"tokens":609,"sizeBytes":2535},{"name":"README.md","path":"tools/ssb-tools/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/ssb-tools/README.md","title":"ssb-tools Documentation","category":"plugin-manifest","format":"markdown","content":"\n\n# Run SSB with one command\n\nThe Trino SSB connector generates rows while Doris reads them. It uses the same\n`ssb-dbgen` distribution as the existing tools, with stdout connected to the scan\nthrough a bounded pipe. There are no intermediate `.tbl` files, Stream Load\nrequests, or separate Trino servers.\n\n## Install the generator plugin once\n\nBuild on Linux with the same CPU architecture as the Doris nodes. The build needs\nJDK 17, Maven, GCC, Make, curl, patch, and sha256sum:\n\n```bash\n./trino-ssb/build.sh\n```\n\nCopy the resulting **entire directory** `trino-ssb/target/ssb` into the configured\nTrino plugin directory on **every FE and BE**. For current Doris installations:\n\n```bash\ncp -r trino-ssb/target/ssb /path/to/fe/plugins/trino_plugins/\ncp -r trino-ssb/target/ssb /path/to/be/plugins/trino_plugins/\n```\n\nStart or restart these nodes after installing the plugin. It consists of a JAR,\nthe native `dbgen` executable, and its distribution file. The executable must\nremain executable. No paths or executable commands are accepted as catalog properties.\n\n## Create, import, and benchmark\n\nSet the Doris connection in `conf/doris-cluster.conf`, then run:\n\n```bash\n./bin/run-ssb.sh -s 1 -d ssb_sf1\n```\n\nThis creates a new database and its `ssb_gen_ssb_sf1` catalog, creates the five\nSSB tables, and imports them using `INSERT INTO ... SELECT ...` from the catalog's\n`sf1` schema. It then builds `lineorder_flat`, collects statistics synchronously,\nand runs all 13 SSB queries and all 13 SSB Flat queries three times each.\nThe supported scale factors are 1, 100, and 1000. Use `--mode ssb` or `--mode flat`\nto select one benchmark; the default is `both`.\n\nEach run writes a new `results/-/` directory with:\n\n- `result.csv`: first-run, two repeated-run, and best repeated-run times in milliseconds.\n- `ssb/` and `ssb-flat/`: each query's output and errors.\n- `prepare.log` and `environment.txt`: import diagnostics and Doris settings.\n\nTiming includes the mysql client connection and result transfer, as in the old\nscripts. The `cold` column means the first execution; the script does not flush\ndata caches. SQL result caching and BE query caching are disabled for every measured\nsession. A query or import error stops the run with a nonzero exit status.\n\nPreparation requires a **new database**, so rerunning a failed or completed load\ncannot append duplicate data. Tables and the catalog are preserved for inspection.\nTo rerun only the queries against completed data:\n\n```bash\n./bin/run-ssb.sh -d ssb_sf1 --queries-only\n```\n\nUse `-c /path/to/cluster.conf` for a separate connection configuration, and\n`--result-dir /path/to/new-directory` for a custom output directory. Existing\nresult directories are never overwritten.\n\n## Generator consistency and tests\n\nThe connector keeps the old tools' default of ten `lineorder` generator partitions.\nEach dimension uses one generator. Changing the original dbgen partition count\nchanges some random streams, so partition count is intentionally fixed here.\nThe date generator runs in UTC on every BE. The native build enables dbgen's\nexisting 64-bit code path for large scale factors and avoids an out-of-bounds\nusage-counter update for its SSB-only random streams. Neither change replaces\nthe data-generation algorithm.\n\nThe existing benchmark SQL is preserved. In particular, SSB Q1.3 filters the\ngenerator's `d_weeknuminyear`, whereas SSB Flat Q1.3 uses `weekofyear()`; these week\ndefinitions differ, so their aggregate results are not expected to match.\n\nThe plugin build runs its Java tests, including full SF1 checksums of all five tables,\nhandle serialization, projected columns, large keys, producer failures, and early\ncursor closure. Run script orchestration tests with:\n\n```bash\npython3 -m unittest discover -s ../benchmark/tests -v\n```\n\nThe shared runner also supports [TPCH and TPCDS](../benchmark/README.md).\n\n# Original file-based workflow\n\n These scripts are used to make ssb and ssb flat test.\n The ssb flat data comes from ssb tables by way of 'INSERT INTO ... SELECT ...'.\n\n## ssb test, follow the steps below:\n### 1. build ssb dbgen tool.\n ./bin/build-ssb-dbgen.sh\n### 2. generate ssb data. use -h for more infomations.\n ./bin/gen-ssb-data.sh -s 1\n### 3. create ssb tables. modify `conf/doris-cluster.conf` to specify Doris cluster info, then run script below.\n ./bin/create-ssb-tables.sh -s 1\n### 4. load ssb data. use -h for help.\n ./bin/load-ssb-data.sh\n### 5. run ssb queries.\n ./bin/run-ssb-queries.sh\n\n## ssb flat test, follow the steps below:\n### 1. prepare ssb data, which means ssb test steps 1 to 4 have been done.\n### 2. run ssb flat queries.\n ./bin/run-ssb-flat-queries.sh\n","isInternal":false,"tokens":1337,"sizeBytes":5454},{"name":"README.md","path":"tools/tpcds-tools/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/tpcds-tools/README.md","title":"tpcds-tools Documentation","category":"plugin-manifest","format":"markdown","content":"\n\n# Run TPCDS with one command\n\nInstall the Trino TPCDS generator on every FE and BE using the\n[shared installation instructions](../benchmark/README.md#install-the-generators-once),\nthen configure `conf/doris-cluster.conf` and run:\n\n```bash\n./bin/run-tpcds.sh -s 1 -d tpcds_sf1\n```\n\nThis streams generated rows directly into Doris, creates the existing benchmark\ntables, collects statistics, and records three executions per query. It supports\nSF1, SF100, SF1000, and SF10000. Preparation requires a new database.\nUse `--queries-only` to measure existing data again, with the same scale factor.\n\nSee [shared workflow options, result files, and timing semantics](../benchmark/README.md).\nTPCDS uses the original scale-specific table DDL and multi-table queries.\nInstall the plugin before running the command.\n\n# Original file-based workflow\n\n## Usage\n\nThese scripts are used to make tpc-ds test.\nfollow the steps below:\n\n### 1. build tpc-ds dsdgen dsqgen tool.\n\n ./bin/build-tpcds-tools.sh\n\n If the build failed in dbgen tools' compilation, update your GCC version or change all \"TPC-DS_Tools_v3.2.0new.zip\" in build-tpcds-dbgen.sh to \"TPC-DS_Tools_v3.2.0.zip\"\n\n### 2. generate tpc-ds data. use -h for more infomations.\n\n ./bin/gen-tpcds-data.sh -s 1\n\n### 3. create tpc-ds tables. modify `conf/doris-cluster.conf` to specify doris info, then run script below.\n\n ./bin/create-tpcds-tables.sh -s 1\n\n### 4. load tpc-ds data. use -h for help.\n\n ./bin/load-tpcds-data.sh\n\n### 5. run tpc-ds queries.\n\n ./bin/run-tpcds-queries.sh -s 1\n","isInternal":false,"tokens":604,"sizeBytes":2309},{"name":"README.md","path":"tools/tpch-tools/README.md","rawUrl":"https://raw.githubusercontent.com/apache/doris/HEAD/tools/tpch-tools/README.md","title":"tpch-tools Documentation","category":"plugin-manifest","format":"markdown","content":"# tpch-tools Documentation\n\nPath: `tools/tpch-tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/tools/tpch-tools/README.md)","tokens":300,"sizeBytes":172}],"systemPromptSnippet":"\n\n# AGENTS.md — Apache Doris\n\nThis is the codebase for Apache Doris, an MPP OLAP database. It primarily consists of the Backend module BE (`be/`, execution and storage engine), the Frontend module FE (`fe/`, optimizer and transaction core), and the Cloud module (`cloud/`, storage-compute separation). Your basic development workflow is: modify code, build using standard procedures, add and run tests, and submit relevant changes.\n\n## Security Threat Model\n\nFor security scans, vulnerability triage, security reviews, and changes involving authentication, authorization, network boundaries, external catalogs, cloud tenancy, or other security-sensitive behavior, read `SECURITY.md` first, then follow it to `threat-model.md`. Use `threat-model.md` to determine in-scope components, trust boundaries, attacker roles, explicit non-goals, and triage classification. Findings that are out of model or by design under `threat-model.md` should be reported as such, not treated as Doris vulnerabilities. Such reviews will not be conducted unless explicitly requested by the user, and no conclusions regarding this matter need to be provided.\n\n## Remote Artifacts and Dynamic Code Loading\n\nDo not add any new code path that fetches an artifact from a user-supplied URL and loads it into a Doris process. This covers `driver_url`-style catalog/resource properties, plugin and UDF locations, and any other property whose value reaches a `URLClassLoader`, `Class.forName`, `System.load`/`dlopen`, or an equivalent BE-side loader. A user who can write such a property gets code execution inside the process that loads it, so the property's effective privilege is that process, not the DDL privilege it was gated on.\n\nThe existing `driver_url` paths (jdbc / iceberg-jdbc / paimon-jdbc catalogs) remain for backward compatibility. They are grandfathered, not a precedent — do not copy the pattern into a new connector or feature.\n\nWhen a feature genuinely needs an operator-supplied artifact, read it from a local, operator-controlled directory (the `jdbc_drivers_dir` / plugin-directory pattern), not from a URL. The ADBC catalog's `AdbcDriverPathResolver` is the model: remote schemes rejected outright, a scheme-less value restricted to a bare file name so no path separator can escape the configured directory, and the check placed where the artifact is loaded rather than only at CREATE.\n\nEvery consumer of the **jdbc-flavored** `driver_url` (the jdbc / iceberg-jdbc / paimon-jdbc catalogs, which do accept a URL) must route the raw value through `JdbcDriverUrlSecurity.check` (`fe-foundation`, `foundation.security`) from its property holder's statement-time validation — `JdbcCatalogProperties.checkCreateTimeOnlyRules` and the iceberg/paimon JDBC metastore holders' `validate()` — which the engine reaches on both CREATE and ALTER CATALOG and never on replay or a catalog rebuild. That class is the single source of truth for the rule; do not fork or re-derive it per connector, and never call it from a holder's `of()`. Such a property must also get a row in the key table of `PluginDrivenExternalCatalog#driverUrlKeysOf` (one table drives both the gate trigger set and the checked values), which is how the operator's `jdbc_driver_secure_path` / `jdbc_driver_url_white_list` policy reaches the ALTER CATALOG path — ALTER never runs `Connector.preCreateValidation`, where CREATE applies that policy, so a key missing from that table lets an ALTER repoint the jar past the operator's configuration.\n\nSeparately, any new outbound HTTP request the FE or BE issues to a host named by a non-SUPER user is an SSRF surface. Call it out explicitly in the PR description together with the privilege required to reach it.\n\n## Thrift and Protobuf Field Ids\n\nEvery change under `gensrc/thrift` and `gensrc/proto` must keep every upgrade compatible (branch-3.1 → 4.0 → 4.1 → 4.2 → master): a field id or enum value means the same thing on every branch, an id is never reused, and a new field is optional. The rule and how to review against it are in `gensrc/AGENTS.md`; read it before touching those directories.\n\n## When running in a WORKTREE directory\n\nTo ensure smooth test execution without interference between worktrees, the first thing to do upon entering a worktree directory is to check if `.worktree_initialized` exists. If not, execute `hooks/setup_worktree.sh`, setting `$ROOT_WORKSPACE_PATH` to the base directory (typically `${DORIS_REPO}`) beforehand. After successful execution, verify that `.worktree_initialized` has been touched and that `thirdparty/installed` dependencies exist correctly.\n\nSubmodules are not part of the default worktree setup. If the task does not compile Doris or run tests that require submodules, leave all submodules uninitialized. Build and test entry points initialize the submodule paths they require. Only initialize a submodule manually when the task actually needs it and an entry point does not handle it; initialize the exact required path instead of running an unscoped recursive update for every submodule.\n\nWhen working in worktree mode, all operations must be confined to the current worktree directory. Do not enter `${DORIS_REPO}` or use any resources there. Compilation and execution must be done within the current worktree directory. The compiled Doris cluster must use random ports not used by other worktrees (modify BE and FE conf before compilation, using a uniform offset of `${DORIS_PORT_OFFSET_RANGE}` from default ports without conflicting with other worktrees' ports). Run from the `output` directory within the worktree. To run regression tests, modify `regression-test/conf/regression-conf.groovy` and set the port numbers in `jdbcUrl` and other configuration items to your new ports so the corresponding worktree cluster can be used for regression testing.\n\n## Coding Standards\n\nAssert correctness only—never use defensive programming with `if` or similar constructs. Any `if` check for errors must have a clearly known inevitable failure path (not speculation). If no such scenario is found, strictly avoid using `if(valid)` checks. However, you may use the `DORIS_CHECK` macro for precondition assertions (if inside performance-sensitive areas like loops, it can only be `DCHECK`). For example, if logically A=true should always imply B=true, then strictly avoid `if (A && B)` and instead use `if (A) { DORIS_CHECK(B); ... }`. In short, the principle is: upon discovering errors or unexpected situations, report errors or crash—never allow the process to continue.\n\nFor `PaddedPODArray` and its peripheral packaging types, such as some certain Column, negative alignment allows the use of -1 as a valid index. No additional special handling is needed when the index may be -1.\n\nWhen adding code, strictly follow existing similar code in similar contexts, including interface usage, error handling, and locking patterns. When adding any code, first try to reference existing functionality. Second, examine the relevant context paragraphs to fully understand the logic.\n\nAfter adding code, conduct self-review and refactoring attempts to ensure good abstraction and reuse as much as possible.\n\n### Code Style Enforcement\n\nAll code must pass style checks before committing. Use the corresponding skill for detailed step-by-step procedures.\n\n**BE (C++) Formatting**: Run `build-support/clang-format.sh` to auto-fix formatting. This script enforces clang-format v16; do not use other versions. Run `build-support/check-format.sh` to check without modifying files. See the `be-code-style` skill for details.\n\n**BE (C++) Static Analysis**: After building BE (which generates `compile_commands.json`), run `build-support/run-clang-tidy.sh` to check modified C++ files against the `.clang-tidy` config. The script parses `git diff` to filter warnings to changed lines where possible, reducing noise from pre-existing code (diagnostics from included headers may still appear). For Cloud C++ files, pass `--build-dir` pointing to the Cloud compilation database (e.g., `cloud/build_ASAN`). Try to fix all reported warnings; if a warning cannot be reasonably fixed, add a `// NOLINT` comment with justification and report it. See the `clang-tidy-check` skill for details.\n\n**BE (C++) Header Hygiene**: BE configure enforces compile-time hygiene gates (header layering rules, include-closure/reach budgets, pch whitelist, extern-template pairing, unity-skip coverage) via `build-support/check-build-hygiene.sh` (~1s, pure text). Run it directly after touching BE headers, includes, template instantiation lists, or unity skip lists — do not wait for CI. Violations are build errors whose messages carry the mechanism and fix; deliberate budget/whitelist changes are one-line table edits in the same commit with justification. Rules and review checkpoints: `be/AGENTS.md`; mechanics: `be/README.md`.\n\n**FE (Java) Style**: Checkstyle is integrated into the Maven build (`maven-checkstyle-plugin`). Running `build.sh --fe` automatically validates style via `mvn validate`. If checkstyle fails, fix the reported issues according to `fe/check/checkstyle/checkstyle.xml`. See the `fe-code-style` skill for details.\n\n## Code Review\n\nWhen conducting code review (including self-review and review tasks), complete the key checkpoints from the `code-review` skill and provide conclusions for each key checkpoint when applicable. Other content does not require item-by-item responses; check them during the review process.\n\n## Build and Run Standards\n\nAlways use only the `build.sh` script with its correct parameters to build Doris BE and FE. For example, the simplest BE+FE build command is `./build.sh --be --fe`.\nBuild type can be set via `BUILD_TYPE` in `custom_env.sh`, but only set it to `RELEASE` when explicitly required for performance testing; otherwise, keep it as `ASAN`.\nYou may modify BE and FE ports and network settings in `conf/` before compilation to ensure correctness and avoid conflicts.\nBuild artifacts are in the current directory's `output/`. If starting the service, ensure all process artifacts have their conf set with appropriate non-conflicting ports and the correct `priority_networks` that match the current network environment, for example `priority_networks = 10.16.10.3/24`. Use `--daemon` when starting. Cluster startup is slow; wait at least 30s for success. If still not ready after waiting, continue waiting. If not ready after a long time, check BE and FE logs to investigate.\nFor first-time cluster startup, you may need to manually add the backend.\n\n## Testing Standards\n\nAll kernel features must have corresponding tests. Prioritize adding regression tests under `regression-test/`, while also having BE unit tests (`be/test/`) and FE unit tests (`fe/fe-core/src/test/`) where possible. Interface usage in test cases must first reference similar cases.\n\nYou must use the preset scripts in the codebase with their correct parameters to run tests (`run-regression-test.sh`, `run-be-ut.sh`, `run-fe-ut.sh`). Regression test result files must not be handwritten; they must be auto-generated via test scripts. When running regression tests, if using `-s` to specify a case, also try to use `-d` to specify the parent directory for faster execution. For example, for cases under `nereids_p0`, you can use `-d nereids_p0 -s xxx`, where `xxx` is the name from `suite(\"xxx\")` in the groovy file.\n\nKey utility functions in BE code, as well as the core logic of complete features, must have corresponding unit tests. If it is inconvenient to add unit tests, revisit the module design and function decomposition to ensure high cohesion and low coupling.\n\nAdded regression tests must comply with the following standards:\n\n1. Use `order_qt` prefix or manually add `order by` to ensure ordered results\n2. For cases expected to error, use the `test{sql,exception}` pattern\n3. After completing tests, do not drop tables; instead drop tables before using them in tests, to preserve the environment for debugging\n4. For ordinary single test tables, do not use `def tableName` form; instead hardcode your table name in all SQL\n5. Except for variables you explicitly need to adjust for testing current functionality, other variables do not need extra setup before testing. For example, nereids optimizer and pipeline engine settings can use default states\n6. For determined expected results, do not using methods like `assert` in test groovy files, but instead generate the `.out` file using `qt_sql` and similar methods.\n\n## Commit Standards\n\nFiles in git commit should only be related to the current modification task. Environment modifications for running (e.g., `conf/`, `AGENTS.md`, `hooks/`, etc.) must not be `git add`ed. When delivering the final task, you must ensure all actual code modifications have been committed.\n\nCommit messages must follow the format below, which mirrors the PR template (`.github/PULL_REQUEST_TEMPLATE.md`):\n\n```\n[]() \n\n### What problem does this PR solve?\n\nIssue Number: close #xxx\n\nRelated PR: #xxx\n\nProblem Summary: \n\n### Release note\n\n\n\n### Check List (For Author)\n\n- Test: \n - Regression test / Unit Test / Manual test / No need to test (with reason)\n- Behavior changed: No / Yes (with explanation)\n- Does this need documentation: No / Yes (with doc PR link)\n```\n\nKey rules for commit messages:\n\n1. The title must follow the `[type](module)` format validated by the PR title checker (`.github/workflows/title-checker.yml`). Common types include: `fix`, `feature`, `improvement`, `refactor`, `chore`, `test`, `doc`. Common modules include: `fe`, `be`, `cloud`, `regression`, `build`\n2. The short summary must be concise and written in imperative mood (e.g., `[fix](fe) Fix null pointer in scan node` not `[fix](fe) Fixed null pointer`)\n3. The `Issue Number` field must reference the corresponding GitHub Issue with `close #xxx` syntax when applicable\n4. The `Release note` section must be filled in for any user-visible behavior or feature change; write \"None\" for internal refactoring or test-only changes\n5. The `Problem Summary` section should cover the following content when available: problem reproduction method, root cause in code, end-to-end results/phenomena before and after repair, and the fix. If it's a refactoring, explain the reason. If it's a performance improvement, specify the case and the exact improvement amount. DO NOT mention any specific JIRA numbers. The background of the problem should be fully understandable through this section alone.\n6. The test section must honestly reflect the testing performed; do not claim tests that were not actually run\n\nFiles in a git commit should only be related to the current modification task. Environment modifications for running (for example `conf/`, `AGENTS.md`, `hooks/`) must not be `git add`ed. When delivering the final task, ensure all actual code modifications have been committed.\n\n\n---\nname: code-review\ndescription: Create consistent releases and changelogs\ncompatibility: opencode\n---\n\n## What I do\n\nComplete code review work to ensure code quality. Due to the length of content, you can read sections as needed based on the actual modifications being reviewed.\n\n## When to use me\n\nUse this when you need to review code, whether it's code you just completed or directly reviewing target code.\n\n---\n\n## Part 1: General Principles\n\n### 1.1 Core Invariants\n\nAlways focus on the following core invariants during review:\n1. **Data Correctness**: Data from any committed transaction must be visible to subsequent queries and not lost; data from uncommitted transactions must not be visible\n2. **Version Consistency**: Partition visible version is the sole standard for data visibility; BE must strictly read data not exceeding this version\n3. **Delete Bitmap Consistency** (MoW tables): delete bitmap must be strictly aligned with rowset versions; sentinel mark / TEMP_VERSION must be replaced with actual versions before use\n4. **Memory Safety**: All significant memory allocations must be tracked by MemTracker; orphan memory is not allowed\n5. **Error Means Failure**: Invariant violations should trigger exceptions or non-OK Status; silent continuation is never allowed\n\n### 1.2 Review Principles\n\n- **Defensive Programming Prohibited**: Do not use `if(valid)` style defensive checks to mask logic errors. If logically A=true should imply B=true, write `if(A) { DORIS_CHECK(B); ... }`, never `if(A && B)`\n- **Follow Context Conventions**: When adding code, strictly follow patterns in adjacent code—same error handling, same interface call order, same lock acquisition patterns, unless a clearly correct and more reasonable approach is identified.\n- **Prioritize Reuse**: Before adding new functionality, search for similar implementations that can be reused or extended; after adding, ensure good abstraction for shared code.\n- **Code First**: When this SKILL conflicts with actual code behavior, defer to your understanding of actual code behavior and explain\n- **Performance First**: All obviously redundant operations should be optimized away, all obvious performance optimizations must be applied, and obvious anti-patterns must be eliminated.\n\n### 1.3 Critical Checkpoints (Self-Review and Review Priority)\n\nThe following checkpoints must be **individually confirmed with conclusions** during self-review and review:\n\n- What is the goal of the current task? Does the current code accomplish this goal? Is there a test that proves it?\n- Is this modification as small, clear, and focused as possible?\n- Does the code segment involve concurrency? If yes:\n - Which threads introduce concurrency, what are the critical variables? What locks protect them?\n - Are operations within locks as lightweight as possible? Are all heavy operations outside locks while maintaining concurrency safety?\n - If multiple locks exist, is the locking order consistent? Is there deadlock risk?\n- Is there special or non-intuitive lifecycle management? If yes:\n - Understand the complete lifecycle and relationships of related variables. Are there circular references? When is each released? Can all lifecycles end normally?\n- Are configuration items added?\n - Should the configuration allow dynamic changes, and does it actually allow them? If yes:\n - Can affected processes detect the change promptly without restart?\n- Does it involve incompatible changes like function symbols or storage formats? If yes:\n - Is compatibility code added? Can it correctly handle requests during rolling upgrades?\n- Are there functionally parallel code paths to the modified one? If yes:\n - Should this modification be applied to other paths, and has it been?\n- Are there special conditional checks? If yes:\n - Does the condition have clear comments explaining why this check is necessary, simplest, and most viable?\n - Are there similar paths with similar issues?\n- What is the test coverage?\n - Are end-to-end functional tests comprehensive?\n - Are there comprehensive negative test cases from various angles?\n - For complex features, are key functions sufficiently modular with unit tests?\n- Does the feature need increased observability? If yes:\n - When bugs occur, are existing logs and VLOGs sufficient to investigate key issues?\n - Are INFO logs lightweight enough?\n - Are Metrics added for critical paths? Referencing similar features, are all necessary observability values covered?\n- Does it involve transaction and persistence-related modifications? If yes:\n - Are all EditLog reads and writes correctly covered? Is all information requiring persistent storage persisted? Does it follow our standard approach?\n - In transaction processing logic, can master failover at any time point be handled correctly?\n- Does it involve data writes and modifications? If yes:\n - Are transactionality and atomicity guaranteed? Are there issues with concurrency?\n - Would FE or BE crashes cause leaks, deadlocks, or incorrect data?\n- Are new variables added that need to be passed between FE and BE? If yes:\n - Is variable passing modified in all paths? For example, various scattered thrift sending points like constant folding, point queries.\n- Analyze deeply from all angles (including but not limited to time complexity, anti-patterns, CPU and memory friendliness, redundant calculations, etc.) based on the code context and available information. Are there any performance issues or optimization opportunities?\n- Based on your understanding, are there any other issues with the current modification?\n\nAfter checking all the above items with code. Use the remaining parts of this skill as needed. The following content does not need individual responses; just check during the review process.\n\n#### 1.3.1 Concurrency and Thread Safety (Highest Priority)\n\n- [ ] **New thread/bthread entry**: Is `SCOPED_ATTACH_TASK` used? Omission causes memory to be counted in orphan tracker, debug builds crash directly\n- [ ] **Lock protection scope**: Is shared data (version map, delete bitmap, transaction state) accessed within correct locks? Is lock type correct (shared vs exclusive)?\n- [ ] **Lock order**: Are there nested acquisitions of multiple locks? Must follow known lock order (e.g., `_txn_lock` -> `_txn_map_lock`), otherwise deadlock\n- [ ] **Atomic operation memory order**: Use `relaxed` for statistics, at least `acquire/release` for flag/signal\n- [ ] **COW column modification**: Is exclusive ownership ensured before `mutate()`? Is `assume_mutable_ref()` only used when exclusive ownership is confirmed?\n\n#### 1.3.2 Error Handling (Highest Priority)\n\n- [ ] **Status checking**: `Status` return values **must** be checked, `static_cast(...)` silent discard is prohibited\n- [ ] **Exception propagation chain**: Is there `RETURN_IF_CATCH_EXCEPTION` above `THROW_IF_ERROR` to catch?\n- [ ] **Error messages**: Are they clear, include context (table/partition/TabletID/transaction ID), provide solution direction?\n- [ ] **DORIS_CHECK usage**: Only for invariant assertions (e.g., `DORIS_CHECK(hash_table != nullptr)`), not for defensive programming\n\n#### 1.3.3 Memory Safety (High Priority)\n\n- [ ] **Large memory reservation**: Is `try_reserve()` called before allocation? Is there `DEFER_RELEASE_RESERVED()` to ensure release?\n- [ ] **Cache Handle**: Is `cache->release(handle)` called after using `Cache::Handle*`?\n- [ ] **shared_ptr circular references**: Do scenarios like `Dependency` use raw pointers or `weak_ptr` to break cycles?\n- [ ] **Object creation**: Do classes with `ENABLE_FACTORY_CREATOR` use `create_shared()` / `create_unique()`? Raw `new` is prohibited\n\n#### 1.3.4 Data Correctness (High Priority)\n\n- [ ] **Delete Bitmap version**: In Cloud mode, is `TEMP_VERSION_COMMON` manually replaced before use?\n- [ ] **Version consistency**: Does data reading strictly not exceed Partition visible version?\n- [ ] **MoW setting**: Is `SegmentWriterOptions::enable_unique_key_merge_on_write` correctly set to `true`?\n- [ ] **EditLog persistence**: Do metadata modifications call `editLog.logXxx()`? Is replay logic strictly equivalent to master?\n\n#### 1.3.5 Observability (Medium Priority)\n\n- [ ] **Metrics addition**: Are `ADD_TIMER/ADD_COUNTER` added to critical paths? Does naming follow `module_xxx` convention?\n- [ ] **Log levels**: Use `LOG(ERROR)` for error logs, `LOG(INFO)` for critical paths, `VLOG` or `DVLOG` for debug info\n- [ ] **Trace information**: Do distributed operations (RPC, transactions) include trace ID for tracking?\n\n### 1.4 Test Coverage Principles\n\n- All kernel features **must** have corresponding tests\n- End-to-end SQL functionality should prioritize adding regression tests under `regression-test/` (Groovy DSL)\n- Also add BE unit tests (`be/test/`, Google Test) and FE unit tests (`fe/fe-core/src/test/`, JUnit 5) where possible. For complex features, must be modular with unit tests.\n- Regression test result files (`.out`) **must not be handwritten**, must be auto-generated via `-genOut`\n- Regression tests must use `order_qt_` prefix or manual `ORDER BY` to ensure ordered results\n- Expected error cases use `test { sql \"...\"; exception \"...\" }` pattern\n- Tables should be **DROPped before use** (not after tests end), to preserve environment for debugging\n- Simple tests should hardcode table names directly, not use `def tableName`\n\n---\n\n## Part 2: BE Module Review Guide\n\n### 2.1 Error Handling (Highest Priority)\n\n#### 2.1.1 Status / Exception / Result\\ Relationship\n\n| Type | Usage | Propagation Macro |\n|------|-------|------------------|\n| `Status` | Default error return type, `[[nodiscard]]` | `RETURN_IF_ERROR` |\n| `Exception` | Vectorized execution engine hot path, expression evaluation | `RETURN_IF_CATCH_EXCEPTION` |\n| `Result` | Return value or error (replaces old output parameter pattern) | `DORIS_TRY` |\n\n**Review Checkpoints:**\n\n- [ ] Are `Status` return values checked? Search for `static_cast(...)` pattern—this is the **most dangerous anti-pattern**, silently swallowing errors without logging. Currently 171+ occurrences in codebase. Never allow this pattern in new code\n- [ ] Where `THROW_IF_ERROR` is used, is there `RETURN_IF_CATCH_EXCEPTION` or `RETURN_IF_ERROR_OR_CATCH_EXCEPTION` above in the call chain to catch? If not, exceptions propagate uncaught causing crashes. Current version has related catch in upper layers when Pipeline executes specific operators.\n- [ ] **`THROW_IF_ERROR` prohibited in Defer/RAII destructor lambdas**: Throwing exceptions during stack unwinding triggers `std::terminate`. Known issues: `pipeline_task.cpp:412-413, 638-639`. Use `WARN_IF_ERROR` or `static_cast` + logging in Defer\n- [ ] **`THROW_IF_ERROR` prohibited in Thrift RPC handlers without try/catch protection**: Thrift framework doesn't catch Doris exceptions, causing connection drops. Known issues: `backend_service.cpp:1323,1326`. RPC handlers should use `RETURN_IF_ERROR` pattern or outermost try/catch\n- [ ] Is `WARN_IF_ERROR` usage reasonable? Only allowed in destructors, cleanup paths, best-effort operations. Warning message must not be empty string\n- [ ] Is `DORIS_CHECK` used for **invariant assertions** (not defensive programming)? Correct usage example: `DORIS_CHECK(hash_table)` asserts hash table is initialized\n- [ ] Do new catch blocks correctly convert exceptions? Should follow standard pattern from `internal_service.cpp:348-357`: catch `doris::Exception` first, then `std::exception`, finally `...`\n\n#### 2.1.2 Cloud RPC Error Handling\n\nThe `retry_rpc` template in `cloud_meta_mgr.cpp` is the standard pattern for RPC error handling in Cloud mode:\n- [ ] Do new Cloud RPC calls use `retry_rpc`?\n- [ ] Does `INVALID_ARGUMENT` return directly without retry? (Correct behavior)\n- [ ] Does `KV_TXN_CONFLICT` use an independent conflict retry counter?\n- [ ] Do timeout and retry counts use config variables instead of hardcoding?\n\n#### 2.1.3 compile_check Mechanism\n\n`compile_check_begin.h` / `compile_check_end.h` elevate `-Wconversion` to **compile error** (excluding sign-conversion and float-conversion).\n\n- [ ] Do new `.h` files include paired `compile_check_begin.h` / `compile_check_end.h` in declaration regions?\n- [ ] **Coverage status**: Currently only 31.1% (1,049/3,372) of source files use compile_check. Additionally, 218 files contain `compile_check_begin.h` but lack corresponding `compile_check_end.h` (unpaired). New files should always include complete pairing\n- [ ] Are there implicit narrowing conversions? Pay special attention to `int64_t -> int32_t`, `size_t -> int`, `uint64_t -> int64_t`\n- [ ] If bypassing due to third-party code is necessary, are `compile_check_avoid_begin.h` / `compile_check_avoid_end.h` used?\n\n### 2.2 Memory Management\n\n#### 2.2.1 MemTracker Hierarchy\n\nDoris uses two-level memory tracking:\n- **`MemTrackerLimiter`**: Heavy-weight tracker with limits (QUERY / LOAD / COMPACTION / SCHEMA_CHANGE / METADATA / CACHE types)\n- **`MemTracker`**: Lightweight tracker without limits, for fine-grained sub-component tracking\n\nThread-level tracking is implemented via `ThreadMemTrackerMgr`, each thread has one active `MemTrackerLimiter` and a `MemTracker` stack.\n\n**Review Checkpoints:**\n\n- [ ] Do newly started threads/bthreads use `SCOPED_ATTACH_TASK` at entry? Without it, memory is counted in orphan tracker, debug builds trigger assertions\n- [ ] Does temporary tracker switching use `SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER` (supports nesting)?\n- [ ] **`ThreadPool::submit_func` callbacks**: Do lambdas submitted to thread pools use `SCOPED_ATTACH_TASK` internally? Currently 87+ `submit_func` calls in codebase, many rely on \"indirect attach\" (attach happens deep in call chain), which is fragile. Correct approach references gold-standard wrapper pattern from `calc_delete_bitmap_executor.h:67-83`: explicit attach at callback entry. Known problematic files: `s3_file_bufferpool.cpp:114`, `wal_manager.cpp:414`, `download_action.cpp:166`\n- [ ] Is `try_reserve()` called before large memory allocation? Is there `DEFER_RELEASE_RESERVED()` to ensure release after allocation?\n- [ ] Does memory hook path (`ThreadMemTrackerMgr::consume`) avoid operations that may trigger memory allocation (e.g., LOG, string formatting)? `_stop_consume` flag exists to prevent recursion\n\n#### 2.2.2 Object Lifecycle\n\n- [ ] Are classes using `ENABLE_FACTORY_CREATOR` created via `create_shared()` / `create_unique()`? Raw `new` is prohibited\n- [ ] Are there `shared_ptr` circular references? Standard ways to break cycles:\n - `Dependency::_shared_state` uses **raw pointer** (non-owning)\n - `Dependency::_blocked_task` uses `vector>`\n - `TrackerLimiterGroup::trackers` uses `list>`\n- [ ] Is Rowset's dual reference counting used correctly? `acquire()` / `release()` is manual reference counting (for performance), `shared_ptr` manages ownership. When `release()` decrements to 0 and in UNLOADING state, triggers `do_close()`\n- [ ] Are rowsets in `_unused_rowsets` judged deletable via `use_count() == 1`?\n- [ ] Are there suspicious `shared_ptr` circular reference issues? For clear lifecycles, prefer `unique_ptr` + raw pointer observer pattern.\n\n#### 2.2.3 COW Column Semantics\n\nVectorized columns (`IColumn`) use custom intrusive reference-counted Copy-on-Write mechanism:\n\n- [ ] Is exclusive ownership ensured when calling `mutate()`? When `use_count() > 1`, triggers deep copy, which is a performance hotspot\n- [ ] Is `assume_mutable_ref()` only used when exclusive ownership is confirmed? This method **does not check reference count**, using it in shared state causes data corruption\n- [ ] Is `set_columns()` called to restore after `Block::mutate_columns()`? After calling, original Block's column pointers are null\n- [ ] Can the pointer returned by `convert_to_full_column_if_const()` share with original column? For `ColumnConst` it materializes a new column, but for normal columns returns shared pointer\n\n#### 2.2.4 Cache Lifecycle\n\n- [ ] Is `cache->release(handle)` called after using `Cache::Handle*`? Omission causes memory leaks\n- [ ] Do cache values inherit from `LRUCacheValueBase`? This base class automatically releases tracking bytes on destruction\n- [ ] Are new cache types registered with `CacheManager`? Unregistered caches don't participate in global GC\n\n### 2.3 Concurrency and Locks\n\n#### 2.3.1 Tablet Lock Hierarchy\n\nThe `Tablet` class has 8+ independent locks, with confirmed usage rules:\n\n| Lock | Type | Rule |\n|----|------|------|\n| `_meta_lock` | `shared_mutex` | Modifying version map requires EXCLUSIVE; reading requires SHARED |\n| `_rowset_update_lock` | `mutex` | Serializes delete bitmap updates during MoW table publish_txn, **independent of _meta_lock** (avoids blocking read path) |\n| `_base_compaction_lock` | `mutex` | Serializes base compaction |\n| `_cumulative_compaction_lock` | `mutex` | Serializes cumulative compaction |\n| `_migration_lock` | `shared_timed_mutex` | Protects tablet migration |\n| `_cooldown_conf_lock` | `shared_mutex` | Protects cooldown configuration |\n\n**Review Checkpoints:**\n\n- [ ] Do operations on `_rs_version_map` or `_stale_rs_version_map` hold `_meta_lock` (shared or exclusive)?\n- [ ] Do `TabletMeta` modification operations hold the owning Tablet's EXCLUSIVE `_meta_lock`?\n- [ ] Is there acquisition of other tablet locks while holding `_meta_lock`? This may cause deadlock\n\n#### 2.3.2 TxnManager Lock Order\n\nConfirmed consistent lock order: `_txn_lock` -> `_txn_map_lock` (applies to commit, publish, delete, rollback).\n- [ ] Do new TxnManager operations follow this lock order?\n\n#### 2.3.3 Pipeline Dependency Concurrency\n\n`Dependency` uses double-checked locking + atomic `_ready` flag + mutex `_task_lock`:\n- `set_ready()`: First check `_ready` (fast path), then lock to set `_ready = true` and wake up\n- `is_blocked_by()`: Lock, check `_ready`, add to wait list if not ready\n\n- [ ] Do new `Dependency` subclasses correctly call `block()` / `set_ready()`?\n- [ ] Do `CountedFinishDependency`'s `add()` / `sub()` operate under `_mtx` protection?\n\n#### 2.3.4 Atomic Operation Memory Order\n\nMemory order usage patterns in codebase:\n- `memory_order_relaxed`: Statistics counters (correct)\n- `memory_order_acquire/release`: Lifecycle flags like `ExecEnv::_s_ready` (correct)\n- `memory_order_acq_rel`: CAS close operations (correct)\n\n- [ ] Do new atomic operations use appropriate memory order? Use relaxed for statistics counters, at least acquire/release for flag/signal\n- [ ] Does `std::atomic` as stop signal use at least acquire/release? Relaxed may theoretically delay propagation\n\n### 2.4 Type System and Serialization\n\n#### 2.4.1 Upgrade/Downgrade Compatibility\n\nUpgrade/downgrade compatibility no longer uses be_exec_version for control. Instead, each feature (e.g., functions, compression algorithms, etc.) adds its own option for confirmation. BE uses is_set&&is_true to branch or replace functionality at appropriate nodes.\n\n#### 2.4.2 Decimal Precision Overflow\n\n- [ ] Do Decimal operation results check precision upper limit? `decimal_result_type()` may promote result to DECIMAL256, causing unexpected memory layout changes\n- [ ] Are DecimalV2's `original_precision` / `original_scale` correctly set? Default value `UINT32_MAX` means \"unknown\", `get_format_scale()` returns different value from `scale` in this case\n\n#### 2.4.3 Thrift Type Mapping\n\n- [ ] Are `TScalarType` optional fields (`precision`, `scale`, `len`) set when needed? DECIMAL needs precision/scale, CHAR/VARCHAR needs len, DATETIMEV2 needs scale\n- [ ] Is depth-first flattened representation of `TTypeDesc.types` correctly traversed? Off-by-one errors are common with nested complex types (ARRAY>)\n\n#### 2.4.4 Block Merge Nullable Handling\n\n`MutableBlock::merge_impl()` assumes target is Nullable and source is non-Nullable when types mismatch, using C-style cast `(DataTypeNullable*)` instead of `dynamic_cast`.\n\n- [ ] Does new Block merge logic correctly handle Nullable promotion? DCHECK only protects in debug builds\n\n### 2.5 Pipeline Execution Engine\n\n#### 2.5.1 Operator Lifecycle\n\nPipeline Task state machine: `INITED -> RUNNABLE -> BLOCKED -> FINISHED -> FINALIZED`\n\n- [ ] Are `SharedState`'s source_deps and sink_deps correctly connected? `inject_shared_state()` is responsible for connection\n\n#### 2.5.2 Memory Reservation and Spill\n\n- [ ] Do memory-intensive operations (Hash Join build, aggregation) use `_try_to_reserve_memory()`?\n- [ ] Is there `_memory_sufficient_dependency` for backpressure when memory is insufficient?\n- [ ] Does `revoke_memory()` correctly implement spill logic?\n\n### 2.6 Storage Engine\n\n#### 2.6.1 Rowset Version Management\n\n- [ ] Are `add_rowset()` and `modify_rowsets()` executed under exclusive `_meta_lock`?\n- [ ] Are version ranges continuous? There should be no version gaps in `_rs_version_map`\n- [ ] Does compaction output rowset's version range correctly cover input rowsets?\n\n#### 2.6.2 Delete Bitmap (MoW Tables, Extremely High Priority)\n\n- [ ] In Cloud mode, is **TEMP_VERSION_COMMON manually replaced** for bitmaps obtained from `CloudTxnDeleteBitmapCache`? Code comments explicitly warn about this issue (`cloud_txn_delete_bitmap_cache.h:72-75`). Omission causes bitmap to be applied to wrong version\n- [ ] Is `_rowset_update_lock` held during delete bitmap calculation? Prevents concurrent calculation on same (txn_id, tablet_id)\n- [ ] Does compaction's delete bitmap calculation use **latest** compaction stats (`ms_base_compaction_cnt`, etc.)? Stale stats will miss recently compacted rowsets\n\n#### 2.6.3 Segment Writing\n\n- [ ] Is `SegmentWriterOptions::enable_unique_key_merge_on_write` correctly set for MoW tables? Default value is `false`, omission causes deduplication to be silently skipped\n- [ ] Is performance impact of row-stored columns (`_serialize_block_to_row_column`) on wide tables within expectations?\n\n### 2.7 IO Subsystem\n\n- [ ] Do `FileSystem` operations use `FILESYSTEM_M` macro to handle bthread async IO?\n- [ ] Do remote storage (S3/HDFS) reads use `create_cached_file_reader()` for file caching?\n\n### 2.8 Inverted Index Subsystem\n\n#### 2.8.1 Lifecycle and Destruction Order\n\n- [ ] **Field declaration order is destruction order**: In `inverted_index_writer.h:92-95`, `_dir` must be declared **before** `_index_writer`, because `_index_writer`'s `write.lock` is created by `_dir.lockFactory`, C++ destructs in reverse declaration order. New fields must follow this constraint\n- [ ] **Cache handle destruction extends LRU retention**: `inverted_index_cache.h:142-152`, `InvertedIndexCacheHandle` destructor sets `last_visit_time` to future timestamp (`UnixMillis() + config::index_cache_entry_stay_time_after_lookup_s * 1000`), making actually queried searchers survive longer in LRU. Be aware of this extension mechanism when modifying cache logic\n\n#### 2.8.2 CLucene Exception Handling\n\nCLucene doesn't use C++ exceptions, `inverted_index_common.h:65-111` implements `ErrorContext` + FINALLY macro system:\n- `FINALLY`: Catches exceptions and returns `Status` error\n- `FINALLY_EXCEPTION`: Catches and rethrows\n- All three macros use `static_assert` to enforce existence of `error_context` local variable in scope\n\n- [ ] Does new CLucene-related code use `ErrorContext` + `FINALLY` pattern instead of raw try/catch?\n\n#### 2.8.3 Three-Valued Logic Bitmap\n\n`InvertedIndexResultBitmap` (`inverted_index_reader.h:80-214`) implements SQL three-valued logic (TRUE/FALSE/NULL), storing separately via `_data_bitmap` + `_null_bitmap`:\n- [ ] **`op_not` is const but actually modifies data**: Modifies pointed object via `shared_ptr` to bypass const qualifier. Callers should not assume const reference returned results are immutable\n- [ ] `operator|=` implements complete SQL three-valued OR semantics (lines 148-157), same for `operator&=` and `operator-=`. New bitmap operations must follow three-valued logic\n\n### 2.9 Schema Change Subsystem\n\n#### 2.9.1 Three-Strategy Selection and Lock Order\n\n`_get_sc_procedure()` (`schema_change.h:303-315`) selects among three strategies based on change type:\n- `LinkedSchemaChange`: Column name/type alias changes only, zero-copy\n- `VSchemaChangeDirectly`: Direct conversion without sorting\n- `VLocalSchemaChangeWithSorting`: Full rewrite with sorting\n\n**Lock acquisition order** (`schema_change.cpp:972-976`): `base_tablet push_lock → new_tablet push_lock → base_tablet header_lock → new_tablet header_lock`\n\n- [ ] Do schema change lock acquisitions follow this four-level order? Out-of-order causes deadlock\n\n#### 2.9.2 MOW Delete Bitmap Four-Step Flow (Extremely High Priority)\n\n`schema_change.cpp:1595-1657` describes critical correctness flow for MOW table schema change:\n1. Newly imported rowsets during dual-write **skip** delete bitmap calculation\n2. After conversion completes, calculate delete bitmap for rowsets during dual-write period (lines 1618-1634)\n3. **Block new publish**, calculate delete bitmap for incremental rowsets (lines 1636-1652)\n4. Switch tablet state to `TABLET_RUNNING` (lines 1653-1656)\n\n- [ ] When modifying schema change delete bitmap path, are the four-step order and blocking logic complete? Step 3's blocking is key to ensuring consistency\n\n#### 2.9.3 Column Deletion Order Trap\n\n`return_columns` is calculated **before** `merge_dropped_columns()` (`schema_change.cpp:959-964` vs line 1062), this is intentional: `return_columns` only covers non-dropped columns, dropped columns are handled internally by `SegmentIterator` for delete predicate evaluation.\n\n- [ ] When modifying schema change column processing logic, is the timing of `return_columns` and dropped columns calculation understood?\n- [ ] `_check_row_nums` can be bypassed by `config::ignore_schema_change_check` (default false) (`schema_change.h:155-171`), this config should remain off in production\n\n### 2.10 Workload Group Memory Tracking\n\n`WorkloadGroup` (`workload_group.h`) uses dual-lock pattern: `_mutex` protects name/version/memory limit/resource_ctxs/shutdown state, `_task_sched_lock` protects scheduler and cgroup pointers.\n\n- [ ] **Memory tracking inconsistency**: `check_mem_used()` (lines 115-123) calculates `_total_mem_used + _wg_refresh_interval_memory_growth` for watermark check, but `exceed_limit()` (lines 132-135) **only compares `_total_mem_used > _memory_limit`**, ignoring `_wg_refresh_interval_memory_growth`. This means during refresh interval, workload group may exceed hard limit without being detected by `exceed_limit()`\n- [ ] Does new memory check logic use the correct method? If precise limit checking is needed, should reference `check_mem_used()` approach\n\n---\n\n## Part 3: FE Module Review Guide\n\n### 3.1 Metadata Persistence (EditLog)\n\n#### 3.1.1 Write Path\n\nAll metadata modifications must go through EditLog persistence. Flow:\n1. Modify in-memory state\n2. Call `editLog.logXxx()` to write log\n3. Log is replicated to Follower via BDBJE\n4. Caller blocks until persistence confirmation\n\n**Review Checkpoints:**\n\n- [ ] Do new DDL/DML operations call corresponding `editLog.logXxx()` methods? Omission causes Follower state inconsistency\n- [ ] Do new OperationTypes have corresponding replay handling in `EditLog.loadJournal()` switch?\n- [ ] Is replay logic **strictly equivalent** to master execution logic? Any difference causes Follower state drift\n- [ ] Do new serializable objects correctly implement Gson serialization? Are `@SerializedName` annotations complete?\n- [ ] Can Image checkpoint correctly include new metadata?\n- [ ] Can master failover at any moment guarantee no duplicate or lost data? Short-term task failures are tolerable, but memory/transaction leaks and data inconsistencies are not allowed.\n\n#### 3.1.2 EditLog Batch and Outside-Lock Commit Mode\n\n`DatabaseTransactionMgr` implements two EditLog commit modes, controlled by `Config.enable_txn_log_outside_lock`:\n\n- **Inside-lock mode** (default): Directly calls `editLog.logInsertTransactionState()` within transaction lock\n- **Outside-lock mode**: Only modifies in-memory state and calls `submitEdit()` to submit to queue within lock, waits for persistence completion via `awaitTransactionState()` outside lock (`DatabaseTransactionMgr.java:383-395`)\n\nFuzzy testing (`DorisFE.java:612-615`) randomly switches between these two modes, so both paths must be correct.\n\n**Review Checkpoints:**\n\n- [ ] Do modifications involving transaction EditLog consider both inside-lock and outside-lock commit paths?\n- [ ] In outside-lock mode, are `awaitTransactionState()` timeout and exception handling correct? Timeout doesn't mean failure—transaction may still be committing\n- [ ] Do new transaction state modifications complete all in-memory state changes before `submitEdit()` submission? After submission, other threads may immediately see new state\n\n#### 3.1.3 EditLog Fatal Error Handling\n\nWhen EditLog flush fails, FE calls `System.exit(-1)` to terminate process—this is **intentional**, preventing split-brain.\n- [ ] Does new code catch and swallow exceptions in EditLog-related paths? This may cause metadata loss\n\n### 3.2 Lock Hierarchy and Concurrency\n\n#### 3.2.1 Confirmed Lock Orders\n\n| Scenario | Lock Order |\n|------|------|\n| ConsistencyChecker | `jobs_lock` -> `synchronized(job)` -> `db_lock` |\n| AlterJobV2 | `synchronized(job)` -> `db_lock` |\n| TabletScheduler | `synchronized(this)` **internal** operations, then **release**, then acquire `db_lock` (two-step method avoids deadlock) |\n| DatabaseTransactionMgr | Its RWLock is **leaf lock**, should not acquire other locks internally |\n| MTMVTask | Acquire table locks after sorting by table ID (avoids deadlock) |\n\n**Review Checkpoints:**\n\n- [ ] Do new lock acquisitions follow the above lock orders?\n- [ ] Is `tryLock(timeout)` used instead of infinite-wait `lock()`? FE's standard pattern is `tryLock` + timeout logging\n- [ ] **Lock timeout standard values**: Env lock = 5s, DB write lock = 100s, Table read lock = 1min. Operations involving multiple tables must lock in ascending table ID order (implemented via `PriorityQueue`) to prevent deadlock\n- [ ] Does `Database.tryWriteLock()` log current owner thread on timeout?\n- [ ] Are there paths acquiring database lock within `synchronized` block? This is a known deadlock risk, should use two-step method\n\n#### 3.2.2 ConcurrentModificationException\n\n- [ ] Is traversal of shared collections under lock protection? `CatalogRecycleBin.write()` added `synchronized` because dump image API concurrent access caused CME\n- [ ] Can there be concurrent modifications during traversal of `Map` / `List`? Prefer `ConcurrentHashMap` or snapshot before traversal\n\n#### 3.2.3 Known Deadlock Cases (New Code Must Avoid Similar Patterns)\n\nThe following are confirmed deadlock risk points in codebase, be extra vigilant when reviewing similar patterns:\n\n| Location | Pattern | Description |\n|------|------|------|\n| `MasterCatalogExecutor.java:70-78` | Initiating RPC while holding catalog lock | RPC timeout blocks catalog operations for long time |\n| `ExternalDatabase.java:137-158` | Nested acquisition of db lock + makeSureInitialized() | Comments explicitly mark deadlock avoidance strategy |\n| `HiveMetaStoreCache.java:126-130` | Acquiring cache lock in thread pool callback | Comment marks \"this may cause a deadlock\" |\n| `CacheFactory.java:85-92` | Circular dependency in cache initialization | Avoided via lazy initialization |\n\n- [ ] Does new code initiate RPC or IO operations while holding locks? This is the most common source of deadlock and performance issues\n- [ ] Does new code acquire locks in callbacks (e.g., thread pool, cache loader) that callers may hold?\n\n### 3.3 Nereids Optimizer\n\n#### 3.3.1 Rule Implementation Specifications\n\n**Review Checkpoints:**\n\n- [ ] Is new Rule's `RuleType` added to `RuleType.java` enum?\n- [ ] Does Rule correctly handle **all Join types**? Pay special attention to ASOF variants (`ASOF_LEFT_INNER_JOIN`, etc.). `COULD_PUSH_THROUGH_LEFT/RIGHT` lists in `PushDownFilterThroughJoin` explicitly include ASOF types\n- [ ] Are constant predicates (`slots.isEmpty()`) correctly handled? Inner Join can push to both sides, Outer Join cannot\n- [ ] Is there **infinite loop risk**? Can created nodes be matched again by another rule in same batch? Code comments have multiple \"separate topDown passes to avoid dead loops\" warnings\n- [ ] Does `OneRewriteRuleFactory` pattern match correct wrapper nodes? For example, `EliminateJoinByFK` only matches `project(join)` not bare `join`\n- [ ] Are predicates checked for `containsUniqueFunction()`? Predicates containing unique functions cannot be pushed down\n\n#### 3.3.2 Rewriter Stage Order\n\nRewriter organizes rules into multi-stage pipeline, key dependencies:\n- `PUSH_DOWN_FILTERS` is called 5-6 times throughout pipeline, because it interacts with many other transformations\n- `InferPredicates` is called twice (with `PUSH_DOWN_FILTERS` in between), because eliminating outer join exposes new inference opportunities\n- `costBased(...)` rules are controlled by `runCboRules` flag\n\n- [ ] Is new rewrite rule placed in correct stage? For example, rules needing predicate pushdown should be after `PUSH_DOWN_FILTERS`\n- [ ] Is new exploration rule registered in `RuleSet`?\n\n#### 3.3.3 Property Derivation\n\n`RequestPropertyDeriver` implements top-down property requests:\n\n- [ ] Does new Physical Operator have visit method in `RequestPropertyDeriver`?\n- [ ] Are multiple alternative property requirements provided (e.g., HashJoin provides both shuffle and broadcast choices)? Single requirement narrows search space\n- [ ] Is `PhysicalProject` alias penetration logic correct? Only `Alias(SlotRef)` and bare `SlotRef` can penetrate\n\n### 3.4 Transaction Management\n\n#### 3.4.1 Transaction Lifecycle\n\nState machine: `PREPARE -> COMMITTED -> VISIBLE` (or `ABORTED`)\n\n- [ ] After transaction commit, can `PublishVersionDaemon` correctly send `PublishVersionTask` to all BEs?\n- [ ] Does publish wait for quorum confirmation?\n- [ ] Does transaction correctly abort after timeout?\n\n**Known race condition—`TransactionState.tableIdList`**: At `TransactionState.java:632`, TODO comment marks that `addTableId()` / `removeTableId()` lack synchronization protection. `tableIdList` is plain `ArrayList`, has race condition (CME or data loss) when multiple threads concurrently add/remove table IDs.\n\n- [ ] Are operations on `TransactionState.tableIdList` under external lock protection? Or should it be replaced with thread-safe collection?\n\n#### 3.4.2 Cloud Mode Differences\n\nIn Cloud mode, transactions are centrally managed by Meta-Service, not FE:\n- [ ] Do Cloud mode transaction operations use `CloudGlobalTransactionMgr` instead of local `GlobalTransactionMgr`?\n- [ ] **Unsafe downcasting prohibited**: Current code has 4 places forcibly casting `GlobalTransactionMgrIface` to `CloudGlobalTransactionMgr` (e.g., `InternalCatalog.java:994`). These casts cause ClassCastException in non-Cloud mode. New code should add required methods by extending `GlobalTransactionMgrIface` interface, not rely on downcasting\n- [ ] Does version retrieval use `getVisibleVersion()` (RPC + cache in Cloud mode)?\n- [ ] Is `getVisibleVersion()`'s `RpcException` correctly handled? Multiple callers catch and return 0, may cause MV refresh misjudgment\n\n### 3.5 Exception System\n\n```\nUserException (checked)\n ├── AnalysisException -- SQL analysis error\n ├── DdlException -- DDL execution failure\n ├── MetaNotFoundException -- Metadata lookup failure\n ├── LoadException -- Load failure\n └── ...\n\nNereidsException (unchecked, RuntimeException)\n ├── nereids/AnalysisException -- Nereids-specific, different from common/\n └── nereids/ParseException\n```\n\n- [ ] **Note Nereids has its own `AnalysisException`** (`nereids/exceptions/AnalysisException.java`, extends RuntimeException), which is a **different class** from `common/AnalysisException` (checked)\n- [ ] Does `ErrorReport.reportAnalysisException()` use correct `ErrorCode`? Custom Doris error codes start from 5000\n- [ ] Does new RPC error handling use `Status` pattern (`TStatusCode` / `PStatus`)?\n\n### 3.6 OlapTable Version Visibility\n\n- [ ] `getVisibleVersion()` behaves differently in Cloud vs non-Cloud mode:\n - Non-Cloud: Returns local `tableAttributes.getVisibleVersion()`\n - Cloud: RPC to meta-service, has cache TTL\n- [ ] Can Cloud mode's version cache TTL (`cloudTableVersionCacheTtlMs`) cause reading stale versions?\n- [ ] `VERSION_NOT_FOUND` is forcibly converted to version=1 (`PARTITION_INIT_VERSION`)—cannot distinguish \"never loaded\" from \"loaded to version 1\"\n\n### 3.7 External Catalog Concurrency Traps\n\n#### 3.7.1 Initialization Race Condition\n\n`ExternalCatalog.makeSureInitialized()` (`ExternalCatalog.java:354-379`) is synchronized, but has the following traps:\n\n- [ ] **`isInitializing` guard is dead code in `ExternalCatalog`**: Although code checks `isInitializing` (line 358), it's never set to `true` (only set to `false` in finally). Compare with `ExternalDatabase.makeSureInitialized()` (`ExternalDatabase.java:137-158`) which **correctly** sets `isInitializing = true`. This means ExternalCatalog has no reentrancy protection\n- [ ] **`lowerCaseToDatabaseName.clear()` race window**: `getFilteredDatabaseNames()` (line 504) first `clear()` then refills `ConcurrentMap`, this method is **not synchronized**. Between clear and refill completion, concurrent `get()` calls return null\n\n#### 3.7.2 resetToUninitialized() NPE Risk\n\n`resetToUninitialized()` (`ExternalCatalog.java:581-590`) sets `objectCreated` and `initialized` to false within synchronized block, then calls `onClose()` (lines 773-784) to set `executionAuthenticator` and `transactionManager` to null.\n\n- [ ] Concurrent queries that have passed `makeSureInitialized()` may be using these fields being set to null. synchronized only protects callers of other synchronized methods, not non-synchronized read paths already in execution\n\n### 3.8 Backup/Restore State Machine\n\n#### 3.8.1 BackupJob State Machine and EditLog Timing\n\nEditLog write timing in `BackupJob` state machine is **uneven**, this is the most important non-obvious convention:\n\n- [ ] **PENDING→SNAPSHOTING doesn't write EditLog** (`BackupJob.java:620-624`): After FE restart, state reverts to PENDING, all snapshot tasks are regenerated and resent. Don't add EditLog when modifying this transition—restart recovery relies on this behavior\n- [ ] **BarrierLog mechanism**: `prepareSnapshotTaskForOlapTableWithoutLock()` creates `BarrierLog` for each table and writes via `editLog.logBarrier()` (lines 648-657), returned `commitSeq` is stored in properties map, used to ensure snapshot consistency\n- [ ] **UPLOAD_INFO stage errors only retry, not cancel** (`BackupJob.java:509-514`): This is the final stage, code comments explicitly state continuous retry until timeout even when encountering unrecoverable errors. Don't call `cancelInternal()` in this stage when modifying error handling\n\n#### 3.8.2 RestoreJob state vs showState\n\n`RestoreJob` has two independent state fields (`RestoreJob.java:157,162`):\n- `state`: Serialized persistent actual state, master sets to FINISHED **before** writing EditLog\n- `showState`: Non-serialized, only for display, master sets to FINISHED **after** writing EditLog (line 2191)\n\n- [ ] This prevents follower from seeing FINISHED state before EditLog replication completes. After deserialization, `gsonPostProcess()` syncs `showState = state`. Must update both fields when modifying state transitions\n\n---\n\n## Part 4: Cloud Module Review Guide\n\n### 4.1 FDB Transaction Mode\n\nCloud module uses FoundationDB (FDB) as the sole metadata storage. All metadata operations must be completed within FDB transactions.\n\n#### 4.1.1 TxnKv Interface Specifications\n\n- [ ] Is `TxnErrorCode` return value from each `Transaction::get()` / `commit()` checked? Marked as `[[nodiscard]]`\n- [ ] Is `TXN_CONFLICT` correctly handled? `commit_txn_immediately` returns error without retry, `commit_txn_eventually` has lazy committer path\n- [ ] Can transaction exceed FDB's 10MB limit? Large transactions need to use `TxnLazyCommitter` for batched commits\n- [ ] Is `TXN_MAYBE_COMMITTED` correctly handled? This indicates commit may have succeeded or failed, requires idempotent retry\n\n#### 4.1.2 Key Encoding Specifications\n\nThree key spaces:\n- `0x01`: User space (mutable metadata)\n- `0x02`: System space (service registration, encryption keys)\n- `0x03`: Version space (multi-version metadata with versionstamp, for snapshots)\n\n- [ ] Do new keys use correct key space prefix?\n- [ ] Does key encoding use `encode_bytes()` (order-preserving byte encoding) and `encode_int64()` (big-endian + sign marker)?\n- [ ] Does compound key field order match query pattern? Range scan relies on key prefix\n\n#### 4.1.3 Commit Path\n\nTwo commit paths:\n- **`commit_txn_immediately`**: Single FDB transaction completes all operations (read txn info, move tmp_rowset, bump version, update stats)\n- **`commit_txn_eventually`**: Uses lazy committer for batched commits when transaction is too large\n\n- [ ] Are the three config switches for lazy commit (`is_2pc`, `enable_txn_lazy_commit`, `config::enable_cloud_txn_lazy_commit`) correctly combined?\n- [ ] Is fallback path from `commit_txn_immediately` to `commit_txn_eventually` (triggered by `TXN_BYTES_TOO_LARGE`) correct?\n\n**Known risk—`commit_txn` infinite loop**: The `commit_txn` entry in `meta_service_txn.cpp:3257-3341` uses `do { ... } while(true)` loop to dispatch `commit_txn_immediately` and `commit_txn_eventually`. When pending lazy-committed transactions are found, it retries, but this loop **has no independent retry count limit**. If lazy committer continuously fails to complete (e.g., FDB continuous conflicts), theoretically infinite loop.\n\n- [ ] Do modifications involving `commit_txn` dispatch loop consider maximum retry count or total timeout?\n\n#### 4.1.4 Versionstamp Usage\n\nTransaction ID is derived from FDB versionstamp (`get_txn_id_from_fdb_ts`).\n\n- [ ] Does code depending on FDB versionstamp correctly use `atomic_set_ver_key()` / `atomic_set_ver_value()`?\n- [ ] Is `get_versionstamp()` only called after `commit()` succeeds?\n\n### 4.2 Meta-Service RPC Specifications\n\n#### 4.2.1 RPC Preprocessing\n\nEach RPC uses `RPC_PREPROCESS` macro for preprocessing:\n- StopWatch timing\n- DORIS_CLOUD_DEFER sets response status and records metrics on return\n- `RPC_RATE_LIMIT` performs per-instance QPS throttling\n\n- [ ] Do new RPCs use `RPC_PREPROCESS` and `RPC_RATE_LIMIT`?\n- [ ] Is response `MetaServiceCode` correctly set?\n\n#### 4.2.2 MetaServiceProxy Automatic Retry\n\n`MetaServiceProxy` decorator automatically retries (exponential backoff) on following error codes:\n- `KV_TXN_STORE_*_RETRYABLE`, `KV_TXN_TOO_OLD`, `KV_TXN_CONFLICT`\n\n- [ ] Do new RPCs need to be idempotent? Retry means same request may execute multiple times\n- [ ] Do non-idempotent operations correctly handle `TXN_MAYBE_COMMITTED`?\n\n### 4.3 Recycler Safety Specifications\n\nRecycler is responsible for GC of expired/deleted data:\n\n- [ ] Before deleting rowset data, are associated transactions aborted first (`abort_txn_for_related_rowset`)? Prevents commit-after-delete data loss\n- [ ] Are recycle operations best-effort + next-round retry mode? Should not stop entire recycle flow due to single failure\n- [ ] Is packed file reference count recycling correct? (`recycle_packed_files`)\n\n### 4.4 Delete Bitmap (Cloud MoW, Extremely High Priority)\n\nCloud mode delete bitmap management fundamentally differs from shared-nothing mode:\n\n- [ ] Bitmaps in `CloudTxnDeleteBitmapCache` contain sentinel marks and `TEMP_VERSION_COMMON`, **must manually replace version** before use. This is a critical correctness constraint explicitly marked in code comments\n- [ ] Empty rowset marks returned by `is_empty_rowset()` are not actively cleared (rely on expiration cleanup), does this affect correctness?\n- [ ] **`CloudTxnDeleteBitmapCache` LRU eviction side effect**: When cache entry is LRU evicted, `publish_status` is reset to `INIT`, causing next access to **completely recalculate** delete bitmap (not incremental update). This is correct but expensive. Additionally, marks written by `mark_empty_rowset()` are never actively cleared—they only disappear when entire cache entry is evicted. On high-frequency MoW tables with insufficient cache capacity, causes repeated recalculation\n- [ ] Do modifications involving `CloudTxnDeleteBitmapCache` consider impact of state reset after eviction?\n- [ ] Are `CalcDeleteBitmapTask`'s compaction stats consistent with latest state in meta-service?\n\n---\n\n## Part 5: Cross-Module Concerns\n\n### 5.1 FE-BE Protocol Compatibility\n\n- [ ] Do new `TPlanNodeType`s have corresponding branches in BE's processing switch?\n- [ ] Can deprecated node types (e.g., `MERGE_JOIN_NODE`, `KUDU_SCAN_NODE`) still be gracefully handled?\n- [ ] If new variables are added, are they correctly set in all paths where FE sends execution plans to BE?\n\n### 5.2 Cloud Mode vs Shared-Nothing Mode\n\nCode extensively uses `Config.isNotCloudMode()` / `Config.isCloudMode()` branches.\n\n- [ ] Do new features consider both modes?\n- [ ] In Cloud mode, tablets are not bound to specific BE—any compute node can load any tablet's metadata\n- [ ] In Cloud mode, rowsets go through two-phase protocol (`prepare_rowset` -> `commit_rowset`), with recycle marker protection in between\n- [ ] In Cloud mode, partition version is centrally managed by Meta-Service, not FE\n\n### 5.3 Merge-on-Write Unique Key Tables\n\nMoW tables are one of the most complex data paths, require special attention during review:\n\n- [ ] Write path: Does `SegmentWriter::probe_key_for_mow()` correctly probe existing rowsets?\n- [ ] Publish path: Is delete bitmap calculation under `_rowset_update_lock` protection?\n- [ ] Query path: Is delete bitmap version-aligned before reading?\n- [ ] Compaction path: Does `calc_delete_bitmap_for_compaction()` correctly handle `allow_delete_in_cumu_compaction`?\n- [ ] Cloud mode: Does delete bitmap cache have expiration mechanism? Is `_unused_delete_bitmap` cleaned up promptly?\n\n### 5.4 Performance Hotspot Concerns\n\n- [ ] Does `SendBatchThreadPool` (64 threads, 102K queue) have saturation risk in heavy shuffle scenarios?\n- [ ] Does `SegmentPrefetchThreadPool` (max 2000 threads) create too many OS threads in intensive scan scenarios?\n- [ ] In Cloud mode, does label deduplication check perform linear scan on labels with many aborted transactions, causing performance cliff?\n- [ ] Is serialization overhead of `_serialize_block_to_row_column()` for wide tables within expectations?\n- [ ] COW columns trigger deep copy on `mutate()` when `use_count > 1`—are there unnecessary copies in high-concurrency scenarios?\n\n---\n\n## Part 6: Testing Review Guide\n\n### 6.1 Regression Test Specifications\n\n- [ ] Is `order_qt_` prefix or manual `ORDER BY` used to ensure ordered results?\n- [ ] Do expected error cases use `test { sql \"...\"; exception \"...\" }` pattern?\n- [ ] Are tables **DROPped before use** (`DROP TABLE IF EXISTS xxx`), not after test ends?\n- [ ] Do simple tests directly hardcode table names instead of using `def tableName`?\n- [ ] Do tests cover these seven scenarios: empty table, all null, mixed null, pure non-null, constant, constant null, nullable wrapper (`nullable()`)?\n- [ ] Are result files (`.out`) auto-generated via `-genOut`? **Handwriting prohibited**\n\n### 6.2 BE Unit Test Specifications\n\n- [ ] Are Google Test's `TEST` / `TEST_F` macros used?\n- [ ] Are tests independent? Not dependent on other tests' execution order\n- [ ] Are `ASSERT_*` (failure terminates test) and `EXPECT_*` (failure continues execution) used correctly?\n- [ ] Do memory-intensive tests use `MemTracker` to check for leaks?\n\n### 6.3 FE Unit Test Specifications\n\n- [ ] Is JUnit 5's `@Test` annotation used?\n- [ ] Do test method names clearly describe test scenarios?\n- [ ] Is `Assertions.assertXxx()` used instead of JUnit 4's static imports?\n- [ ] Do tests involving concurrency use `CountDownLatch` / `CyclicBarrier` for synchronization?\n\n### 6.4 FE-BE Integration Test Specifications\n---\n\n## Part 7: Known High-Risk Code Patterns Quick Reference\n\n| Pattern | Risk Level | Location | Description |\n|------|---------|------|------|\n| `static_cast(status_returning_call)` | **CRITICAL** | BE global (195+ places) | Silently swallows errors, no logging |\n| Delete bitmap TEMP_VERSION not replaced | **CRITICAL** | `cloud_txn_delete_bitmap_cache.h:72-75` | Data visibility error |\n| `be_exec_version` mismatch | **CRITICAL** | FE-BE communication all paths | Serialization format incompatibility causes data corruption |\n| `(DataTypeNullable*)` C-style cast | **HIGH** | `block.h merge_impl` | No protection in release builds |\n| `sizeof(size_t*)` vs `sizeof(size_t)` | **HIGH** | `data_type.cpp:195` | Happens to work on 64-bit, crashes on 32-bit |\n| Lazy commit three-switch combination | **HIGH** | `meta_service_txn.cpp` | Missing config causes large transaction failure |\n| DecimalV2 original_precision default UINT32_MAX | **HIGH** | `data_type_decimal.h` | Affects string representation correctness |\n| `MOW` default false | **MEDIUM** | `segment_writer.h:70` | Omission causes deduplication to be silently skipped |\n| Cloud version cache TTL | **MEDIUM** | `OlapTable.getVisibleVersion()` | May read stale version |\n| Label scan linear complexity | **MEDIUM** | `meta_service_txn.cpp` | Many aborted txns cause performance cliff |\n| `THROW_IF_ERROR` no outer catch | **MEDIUM** | vec/ execution engine | Uncaught exception propagation causes crash |\n| `THROW_IF_ERROR` in Defer lambda | **CRITICAL** | `pipeline_task.cpp:412-413, 638-639` | Throwing exception during stack unwinding causes `std::terminate` |\n| `THROW_IF_ERROR` in Thrift RPC handler | **HIGH** | `backend_service.cpp:1323,1326` | No try/catch protection, connection drops |\n| `DCHECK` disappears in RELEASE builds | **HIGH** | BE global (2,872 places) | No invariant protection in production, should migrate\n\n\n# tools Documentation\n\nPath: `be/src/io/tools/readme.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/be/src/io/tools/readme.md)\n\n\n# resources Plugin Manifest\n\nPath: `extension/DataX/doriswriter/src/main/resources/plugin.json`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/extension/DataX/doriswriter/src/main/resources/plugin.json)\n\n\n# fe-authentication-plugin-ldap Documentation\n\nPath: `fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-ldap/README.md)\n\n\n# fe-authentication-plugin-password Documentation\n\nPath: `fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/fe/fe-authentication/fe-authentication-plugins/fe-authentication-plugin-password/README.md)\n\n\n\n\n# Usage\n\n**These scripts are used to do [ClickBench](https://benchmark.clickhouse.com/) test, more info [there](https://github.com/ClickHouse/ClickBench).**\n\n## follow the steps below:\n\n### 1. create table\n ./create-clickbench-table.sh\n\n### 2. load data\n ./load-clickbench-data.sh\n\n### 3. run queries\n ./run-clickbench-queries.sh\n\n\n\n\n## Usage\n\nThese scripts are used to make coffee-bench test.\nfollow the steps below:\n\n### 1. create tables. modify `conf/doris-cluster.conf` to specify doris info, then run script below.\n\n ./bin/create-tables.sh\n\n### 2. load coffee-bench data. use -s for scale. \"500m/1b/5b\"\n\n ./bin/load-data.sh -s 500m\n\n **Note: The data loading process uses S3 data located in the US East (us-east-1) region.**\n If you cannot access this region due to network restrictions or other reasons, you can:\n 1. Use AWS CLI to copy the data to an S3 bucket in your preferred region\n 2. Download the data locally and use alternative methods to import it into Doris\n\n### 3. run queries.\n\n ./bin/run-queries.sh\n\n NOTICE: At present, we rewrite some queries in coffee bench to adapt to Doris' execution framework, but it does not affect the correctness of the results.\n\n\n\n\nThis code is used to evaluate the cost model in doris.\nYou can config the query in config of main.py \n\nBefore running, you should install the libraries in requirements.txt\n\n\n\n\n# Data Lake Regression Testing Tool For External Table\n\nUsed to test the doris external table on object storage for cloud vendors\n\n> Supported storage formats: HDFS, Alibaba Cloud OSS, Tencent Cloud COS, Huawei Cloud OBS\n\n> Supported data lake table formats: Iceberg\n\nThe following provides the example of the command line options:\n\n```\nsh tools/emr_storage_regression/emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\n```\nsh tools/emr_storage_regression/emr_tools.sh --case CASE --endpoint ENDPOINT --region REGION --service SERVICE --ak AK --sk SK --host HOST --user USER --port PORT\n```\n\nThe usage of each option is described below.\n\n## Connectivity Test\n\nWhen the `--case` option is set to `ping`, will check Doris's connectivity on EMR:\n\n- `--endpoint`, Object Storage Endpoint.\n\n- `--region`, Object Storage Region.\n\n- `--ak`, Object Storage Access Key.\n\n- `--sk`, Object Storage Secret Key.\n\n- `--host`, Doris Mysql Client IP.\n\n- `--user`, Doris Mysql Client Username.\n\n- `--port`, Doris Mysql Client Port.\n\n- `--service`, EMR cloud vendors: ali(Alibaba), hw(Huawei), tx(tencent).\n\n### Environment Variables\n\nNeed modify the environment variable in `default_emr_env.sh`, the script will execute `source default_emr_env.sh` to make the environment variable take effect.\n\nIf environment variables are configured, you can run the test script directly with the following command:\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\n### The Script Execution Steps For Connectivity Test\n\n1. Create Spark and Hive tables on EMR\n2. Use Spark and Hive command lines to insert sample data\n3. Doris creates the Catalog for connectivity test \n4. Execute SQL for connectivity test: `ping.sql`\n\n### Alibaba Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service` to `ali`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint oss-cn-beijing-internal.aliyuncs.com --region cn-beijing --service ali --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log\n```\n\nAlibaba Cloud EMR also supports testing connectivity for both Doris with DLF metadata and Doris on OSS-HDFS storage.\n\n- The DLF metadata connectivity test needs to be performed on the EMR cluster where the DLF serves as the metadata store, Default value of `DLF_ENDPOINT` is `datalake-vpc.cn-beijing.aliyuncs.com`, configured at ping_test/ping_poc.sh.\n\n- To test the OSS-HDFS storage connectivity, need to [enable the HDFS service on the OSS storage and configure](https://www.alibabacloud.com/help/en/e-mapreduce/latest/oss-hdfsnew), Default value of `JINDO_ENDPOINT` is `cn-beijing.oss-dls.aliyuncs.com`, configured at ping_test/ping_poc.sh.\n\n### Tencent Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service` to `tx`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint cos.ap-beijing.myqcloud.com --region ap-beijing --service tx --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log\n```\n\n### Huawei Cloud\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\nOr\n\nSet `--service`to `hw`, and then test connectivity on Huawei Cloud.\n\n```\nsh emr_tools.sh --case ping --endpoint obs.cn-north-4.myhuaweicloud.com --region cn-north-4 --service hw --ak ak --sk sk --host 127.0.0.1 --user root --port 9030 > log \n```\n\n## Performance Testing on Standard Test Set\n\nWhen the `--case` option is set to `data_set`, will test the query performance of Doris external table:\n\n- `--test` test data set: ssb, ssb_flat, tpch, clickbench and all. Default `all`.\n\n- `--service`, EMR cloud vendors: ali(Alibaba), hw(Huawei), tx(tencent).\n\n- `--host`, Doris Mysql Client IP.\n\n- `--user`, Doris Mysql Client Username.\n\n- `--port`, Doris Mysql Client Port.\n\n### Environment Variables\n\nJust modify the above environment variable in `default_emr_env.sh`, the script will execute `source default_emr_env.sh` to make the environment variable take effect.\n\nIf environment variables are configured, you can run the test script directly with the following command:\n\n```\nsh emr_tools.sh --profile default_emr_env.sh\n```\n\n### Prepare Data\n\n1. To run the standard test set using the `emr_tools.sh` script, you need to rewrite the object storage bucket specified by the `BUCKET` variable, and then prepare data in advance and put them under the bucket. The script will generate table creation statements based on the bucket.\n\n2. Now the `emr_tools.sh` script supports iceberg, parquet and orc data for ssb, ssb_flat, tpch, clickbench.\n\n### Execution Steps\n\n1. After the connectivity test, the Doris Catalog corresponding to the standard test set is created\n2. Prepare the test set data based on the object storage bucket specified by the `BUCKET` variable\n3. Generate Spark table creation statements and create Spark object storage tables on EMR\n4. Create the spark table in the local HDFS directory: `hdfs:///benchmark-hdfs`\n5. You can choose to analyze Doris tables ahead of time and manually execute the statements in `analyze.sql` in the Doris Catalog\n6. Execute standard test set scripts: `run_standard_set.sh`\n\n### Standard data set: ssb, ssb_flat, tpch, clickbench\n\n- Full test. After executing the test command, Doris will run ssb, ssb_flat, tpch, clickbench tests in sequence, and the test results will include the cases on HDFS and on the object storage specified by `--service`.\n\n```\nsh emr_tools.sh --case data_set --service ali --host 127.0.0.1 --user root --port 9030 > log\n```\n\n- Specify a single test. `--test` option can be set to one of ssb, ssb_flat, tpch and clickbench.\n\n```\nsh emr_tools.sh --case data_set --test ssb --service ali --host 127.0.0.1 --user root --port 9030 > log\n```\n\n\n\n\n# LZO Writer\n\nThis is simple LZO writer, you can use it to generate lzo compressed file.\n\nSee `main()` for details\n\n## Compilation\n\n```\nsh build.sh\n```\n\n## Generate file\n\n```\n./lzo_writer file_name.lzo\n```\n\n\n\n\n# Pipeline Tracing and Display Tool\n\nIn the Pipeline execution engine, we split the execution plan tree of each Instance into multiple small Pipeline Tasks and execute them under our custom Pipeline scheduler. Therefore, in an environment with a large number of Pipeline Tasks executing, how these Tasks are scheduled across threads and CPU cores is an important factor for execution performance. We have developed a specialised tool to observe the scheduling process on a particular query or time period, which we call \"Pipeline Tracing\".\n\nThis tool converts record files to proper JSON format for visualization.\n\n## How to Use\n\n```shell\npython3 origin-to-show.py -s -d .json\n```\nto transfer record file `` to `.json`. Then it could be visualized.\n\n```shell\npython3 origin-to-show.py --help\n```\nfor help details.\n\n\n\n\nSteps of row-to-column conversion:\n\n1. Modify the corresponding cluster configuration in `conf`.\n\n2. Run `python convert_row_to_column.py` to generate all row storage to column storage commands.\n\n3. If the script runs without any results, it means that there is no row storage table in the cluster, so nothing need to be done.\n\n4. If there is a table in the script that fails to obtain information.\n\n Please use `show create table` in the cluster to see the `storage_type` of the table. If it is row storage, please manually convert it.\n\n5. After the schema change is completed, the old schema needs to be deleted after 10 minutes.\n\n Please wait for a while to make sure that the data of the old schema has been deleted, and then upgrade.\n Otherwise, the BE process will exit with an error when it is automatically detected when it starts.\n\n6. Run `python convert_row_to_column.py` again to verify that there are no rows in the table before upgrading.\n\n\n\n\nThis tool is used to get the progress of all current table transitions \nduring the online `segment_2` function.\n\nCurrently, you can specify 3 dimensions (in the conf file) to view the \nresults, you can specify one of them individually, or you can customize \nthe combination (that is, specify multiple at the same time).\n\n# Note\nWe use MySQLdb python lib to fetch meta from FE, so you must install it.\n\nYou can get MySQLdb lib from https://pypi.python.org/pypi/MySQL-python, \nthen you can install it as follows:\n```\n$ tar zxvf MySQL-python-*.tar.gz\n$ cd MySQL-python-*\n$ python setup.py build\n$ python setup.py install\n``` \n\n# Steps\n1. Fill in the conf according to your cluster configuration, and specify \n the table or be you want to watch.\n2. Execute `python show_segment_status.py`\n\n# Example\n1. If you want to watch the process of a table named `xxxx`, you can specify \n `table_name = xxxx` in conf file\n\n2. If you want to watch the process on be whose be_id is `xxxx`, you can specify \n `be_id = xxxx` in conf file\n\n# Output Example Format\n\nThe first number is about segment v2, the latter one is the total count.\n\n```\n==========SUMMARY()===========\nrowset_count: 289845 / 289845\nrowset_disk_size: 84627551189 / 84627551189\nrowset_row_count: 1150899153 / 1150899153\n===========================================================\n==========SUMMARY(table=xxxx)===========\nrowset_count: 289845 / 289845\nrowset_disk_size: 84627551189 / 84627551189\nrowset_row_count: 1150899153 / 1150899153\n===========================================================\n==========SUMMARY(be=10003 )===========\nrowset_count: 79650 / 79650\nrowset_disk_size: 24473921575 / 24473921575\nrowset_row_count: 331449328 / 331449328\n===========================================================\n```\n\n\n\n\n# Run SSB with one command\n\nThe Trino SSB connector generates rows while Doris reads them. It uses the same\n`ssb-dbgen` distribution as the existing tools, with stdout connected to the scan\nthrough a bounded pipe. There are no intermediate `.tbl` files, Stream Load\nrequests, or separate Trino servers.\n\n## Install the generator plugin once\n\nBuild on Linux with the same CPU architecture as the Doris nodes. The build needs\nJDK 17, Maven, GCC, Make, curl, patch, and sha256sum:\n\n```bash\n./trino-ssb/build.sh\n```\n\nCopy the resulting **entire directory** `trino-ssb/target/ssb` into the configured\nTrino plugin directory on **every FE and BE**. For current Doris installations:\n\n```bash\ncp -r trino-ssb/target/ssb /path/to/fe/plugins/trino_plugins/\ncp -r trino-ssb/target/ssb /path/to/be/plugins/trino_plugins/\n```\n\nStart or restart these nodes after installing the plugin. It consists of a JAR,\nthe native `dbgen` executable, and its distribution file. The executable must\nremain executable. No paths or executable commands are accepted as catalog properties.\n\n## Create, import, and benchmark\n\nSet the Doris connection in `conf/doris-cluster.conf`, then run:\n\n```bash\n./bin/run-ssb.sh -s 1 -d ssb_sf1\n```\n\nThis creates a new database and its `ssb_gen_ssb_sf1` catalog, creates the five\nSSB tables, and imports them using `INSERT INTO ... SELECT ...` from the catalog's\n`sf1` schema. It then builds `lineorder_flat`, collects statistics synchronously,\nand runs all 13 SSB queries and all 13 SSB Flat queries three times each.\nThe supported scale factors are 1, 100, and 1000. Use `--mode ssb` or `--mode flat`\nto select one benchmark; the default is `both`.\n\nEach run writes a new `results/-/` directory with:\n\n- `result.csv`: first-run, two repeated-run, and best repeated-run times in milliseconds.\n- `ssb/` and `ssb-flat/`: each query's output and errors.\n- `prepare.log` and `environment.txt`: import diagnostics and Doris settings.\n\nTiming includes the mysql client connection and result transfer, as in the old\nscripts. The `cold` column means the first execution; the script does not flush\ndata caches. SQL result caching and BE query caching are disabled for every measured\nsession. A query or import error stops the run with a nonzero exit status.\n\nPreparation requires a **new database**, so rerunning a failed or completed load\ncannot append duplicate data. Tables and the catalog are preserved for inspection.\nTo rerun only the queries against completed data:\n\n```bash\n./bin/run-ssb.sh -d ssb_sf1 --queries-only\n```\n\nUse `-c /path/to/cluster.conf` for a separate connection configuration, and\n`--result-dir /path/to/new-directory` for a custom output directory. Existing\nresult directories are never overwritten.\n\n## Generator consistency and tests\n\nThe connector keeps the old tools' default of ten `lineorder` generator partitions.\nEach dimension uses one generator. Changing the original dbgen partition count\nchanges some random streams, so partition count is intentionally fixed here.\nThe date generator runs in UTC on every BE. The native build enables dbgen's\nexisting 64-bit code path for large scale factors and avoids an out-of-bounds\nusage-counter update for its SSB-only random streams. Neither change replaces\nthe data-generation algorithm.\n\nThe existing benchmark SQL is preserved. In particular, SSB Q1.3 filters the\ngenerator's `d_weeknuminyear`, whereas SSB Flat Q1.3 uses `weekofyear()`; these week\ndefinitions differ, so their aggregate results are not expected to match.\n\nThe plugin build runs its Java tests, including full SF1 checksums of all five tables,\nhandle serialization, projected columns, large keys, producer failures, and early\ncursor closure. Run script orchestration tests with:\n\n```bash\npython3 -m unittest discover -s ../benchmark/tests -v\n```\n\nThe shared runner also supports [TPCH and TPCDS](../benchmark/README.md).\n\n# Original file-based workflow\n\n These scripts are used to make ssb and ssb flat test.\n The ssb flat data comes from ssb tables by way of 'INSERT INTO ... SELECT ...'.\n\n## ssb test, follow the steps below:\n### 1. build ssb dbgen tool.\n ./bin/build-ssb-dbgen.sh\n### 2. generate ssb data. use -h for more infomations.\n ./bin/gen-ssb-data.sh -s 1\n### 3. create ssb tables. modify `conf/doris-cluster.conf` to specify Doris cluster info, then run script below.\n ./bin/create-ssb-tables.sh -s 1\n### 4. load ssb data. use -h for help.\n ./bin/load-ssb-data.sh\n### 5. run ssb queries.\n ./bin/run-ssb-queries.sh\n\n## ssb flat test, follow the steps below:\n### 1. prepare ssb data, which means ssb test steps 1 to 4 have been done.\n### 2. run ssb flat queries.\n ./bin/run-ssb-flat-queries.sh\n\n\n\n\n# Run TPCDS with one command\n\nInstall the Trino TPCDS generator on every FE and BE using the\n[shared installation instructions](../benchmark/README.md#install-the-generators-once),\nthen configure `conf/doris-cluster.conf` and run:\n\n```bash\n./bin/run-tpcds.sh -s 1 -d tpcds_sf1\n```\n\nThis streams generated rows directly into Doris, creates the existing benchmark\ntables, collects statistics, and records three executions per query. It supports\nSF1, SF100, SF1000, and SF10000. Preparation requires a new database.\nUse `--queries-only` to measure existing data again, with the same scale factor.\n\nSee [shared workflow options, result files, and timing semantics](../benchmark/README.md).\nTPCDS uses the original scale-specific table DDL and multi-table queries.\nInstall the plugin before running the command.\n\n# Original file-based workflow\n\n## Usage\n\nThese scripts are used to make tpc-ds test.\nfollow the steps below:\n\n### 1. build tpc-ds dsdgen dsqgen tool.\n\n ./bin/build-tpcds-tools.sh\n\n If the build failed in dbgen tools' compilation, update your GCC version or change all \"TPC-DS_Tools_v3.2.0new.zip\" in build-tpcds-dbgen.sh to \"TPC-DS_Tools_v3.2.0.zip\"\n\n### 2. generate tpc-ds data. use -h for more infomations.\n\n ./bin/gen-tpcds-data.sh -s 1\n\n### 3. create tpc-ds tables. modify `conf/doris-cluster.conf` to specify doris info, then run script below.\n\n ./bin/create-tpcds-tables.sh -s 1\n\n### 4. load tpc-ds data. use -h for help.\n\n ./bin/load-tpcds-data.sh\n\n### 5. run tpc-ds queries.\n\n ./bin/run-tpcds-queries.sh -s 1\n\n\n# tpch-tools Documentation\n\nPath: `tools/tpch-tools/README.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/apache/doris/HEAD/tools/tpch-tools/README.md)\n"}