3FS Documentation

Technical Documentation: deepseek-ai/3FS

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Fire-Flyer File System

The Fire-Flyer File System (3FS) is a high-performance distributed file system designed to address the challenges of AI training and inference workloads. It leverages modern SSDs and RDMA networks to provide a shared storage layer that simplifies development of distributed applications. Key features and benefits of 3FS include:

• Performance and Usability

  • Disaggregated Architecture Combines the throughput of thousands of SSDs and the network bandwidth of hundreds of storage nodes, enabling applications to access storage resource in a locality-oblivious manner.
  • Strong Consistency Implements Chain Replication with Apportioned Queries (CRAQ) for strong consistency, making application code simple and easy to reason about.
  • File Interfaces Develops stateless metadata services backed by a transactional key-value store (e.g., FoundationDB). The file interface is well known and used everywhere. There is no need to learn a new storage API.

• Diverse Workloads

  • Data Preparation Organizes outputs of data analytics pipelines into hierarchical directory structures and manages a large volume of intermediate outputs efficiently.
  • Dataloaders Eliminates the need for prefetching or shuffling datasets by enabling random access to training samples across compute nodes.
  • Checkpointing Supports high-throughput parallel checkpointing for large-scale training.
  • KVCache for Inference Provides a cost-effective alternative to DRAM-based caching, offering high throughput and significantly larger capacity.

Documentation Copy Chapter

• Design Notes
• Setup Guide
• USRBIO API Reference
• P Specifications

Performance Copy Chapter

1. Peak throughput Copy Section

The following figure demonstrates the throughput of read stress test on a large 3FS cluster. This cluster consists of 180 storage nodes, each equipped with 2×200Gbps InfiniBand NICs and sixteen 14TiB NVMe SSDs. Approximately 500+ client nodes were used for the read stress test, with each client node configured with 1x200Gbps InfiniBand NIC. The final aggregate read throughput reached approximately 6.6 TiB/s with background traffic from training jobs.

To benchmark 3FS, please use our fio engine for USRBIO.

2. GraySort Copy Section

We evaluated smallpond using the GraySort benchmark, which measures sort performance on large-scale datasets. Our implementation adopts a two-phase approach: (1) partitioning data via shuffle using the prefix bits of keys, and (2) in-partition sorting. Both phases read/write data from/to 3FS.

The test cluster comprised 25 storage nodes (2 NUMA domains/node, 1 storage service/NUMA, 2×400Gbps NICs/node) and 50 compute nodes (2 NUMA domains, 192 physical cores, 2.2 TiB RAM, and 1×200 Gbps NIC/node). Sorting 110.5 TiB of data across 8,192 partitions completed in 30 minutes and 14 seconds, achieving an average throughput of 3.66 TiB/min.

3. KVCache Copy Section

KVCache is a technique used to optimize the LLM inference process. It avoids redundant computations by caching the key and value vectors of previous tokens in the decoder layers.
The top figure demonstrates the read throughput of all KVCache clients (1×400Gbps NIC/node), highlighting both peak and average values, with peak throughput reaching up to 40 GiB/s. The bottom figure presents the IOPS of removing ops from garbage collection (GC) during the same time period.

Check out source code Copy Chapter

Clone 3FS repository from GitHub:

git clone https://github.com/deepseek-ai/3fs

When deepseek-ai/3fs has been cloned to a local file system, run the
following commands to check out the submodules:

cd 3fs
git submodule update --init --recursive
./patches/apply.sh

Install dependencies Copy Chapter

Install dependencies:

# for Ubuntu 20.04.
apt install cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev libdwarf-dev libunwind-dev \
  libaio-dev libgflags-dev libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 clang-tidy-14 lld-14 \
  libgoogle-perftools-dev google-perftools libssl-dev libclang-rt-14-dev gcc-10 g++-10 libboost1.71-all-dev build-essential

# for Ubuntu 22.04.
apt install cmake libuv1-dev liblz4-dev liblzma-dev libdouble-conversion-dev libdwarf-dev libunwind-dev \
  libaio-dev libgflags-dev libgoogle-glog-dev libgtest-dev libgmock-dev clang-format-14 clang-14 clang-tidy-14 lld-14 \
  libgoogle-perftools-dev google-perftools libssl-dev gcc-12 g++-12 libboost-all-dev build-essential

# for openEuler 2403sp1
yum install cmake libuv-devel lz4-devel xz-devel double-conversion-devel libdwarf-devel libunwind-devel \
    libaio-devel gflags-devel glog-devel gtest-devel gmock-devel clang-tools-extra clang lld \
    gperftools-devel gperftools openssl-devel gcc gcc-c++ boost-devel

# for OpenCloudOS 9 and TencentOS 4
dnf install epol-release wget git meson cmake perl lld gcc gcc-c++ autoconf lz4 lz4-devel xz xz-devel \
    double-conversion-devel libdwarf-devel libunwind-devel libaio-devel gflags-devel glog-devel \
    libuv-devel gmock-devel gperftools gperftools-devel openssl-devel boost-static boost-devel mono-devel \
    libevent-devel libibverbs-devel numactl-devel python3-devel

Install other build prerequisites:

• libfuse 3.16.1 or newer version
• FoundationDB 7.1 or newer version
• Rust toolchain: minimal 1.75.0, recommended 1.85.0 or newer version (latest stable version)

Build 3FS Copy Chapter

Build 3FS in build folder:

# Replace <method> with 'g++10' or 'g++11' based on your environment
cmake -S . -B build \
      -DCMAKE_CXX_COMPILER=clang++-14 -DCMAKE_C_COMPILER=clang-14 \
      -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
      -DSHUFFLE_METHOD=<method>
cmake --build build -j 32

Due to the historical use of std::shuffle, binaries compiled with different compiler versions (e.g., g++10 vs. g++11 +) may be incompatible (issue). To resolve this, you must explicitly specify -DSHUFFLE_METHOD during compilation to lock in a consistent shuffle algorithm:

• Existing Clusters: Use the method corresponding to the compiler version previously used to deploy the cluster (g++10 or g++11).
• New Clusters: You can choose either g++10 or g++11. However, once the cluster is deployed, you must stay with the same configuration for all future builds to maintain compatibility.

Build 3FS use Docker Copy Section

• For TencentOS-4: docker pull docker.io/tencentos/tencentos4-deepseek3fs-build:latest
• For OpenCloudOS-9: docker pull docker.io/opencloudos/opencloudos9-deepseek3fs-build:latest

Run a test cluster Copy Chapter

Follow instructions in setup guide to run a test cluster.

Report Issues Copy Chapter

Please visit https://github.com/deepseek-ai/3fs/issues to report issues.

Overview (`deploy/README.md`) Copy Section

3FS Setup Guide

This section provides a manual deployment guide for setting up a six-node cluster with the cluster ID stage.

Installation prerequisites Copy Chapter

Hardware specifications Copy Section

RDMA Configuration

1. Assign IP addresses to RDMA NICs. Multiple RDMA NICs (InfiniBand or RoCE) are supported on each node.
2. Check RDMA connectivity between nodes using ib_write_bw.

Third-party dependencies Copy Section

In production environment, it is recommended to install FoundationDB and ClickHouse on dedicated nodes.

FoundationDB

1. Ensure that the version of FoundationDB client matches the server version, or copy the corresponding version of libfdb_c.so to maintain compatibility.
2. Find the fdb.cluster file and libfdb_c.so at /etc/foundationdb/fdb.cluster, /usr/lib/libfdb_c.so on nodes with FoundationDB installed.

Step 0: Build 3FS Copy Chapter

Follow the instructions to build 3FS. Binaries can be found in build/bin.

Services and clients Copy Section

The following steps show how to install 3FS services in /opt/3fs/bin and the config files in /opt/3fs/etc.

Step 1: Create ClickHouse tables for metrics Copy Chapter

Import the SQL file into ClickHouse:

clickhouse-client -n < ~/3fs/deploy/sql/3fs-monitor.sql

Step 2: Monitor service Copy Chapter

Install monitor_collector service on the meta node.

1. Copy monitor_collector_main to /opt/3fs/bin and config files to /opt/3fs/etc, and create log directory /var/log/3fs.
   bash Copy

   mkdir -p /opt/3fs/{bin,etc}
   mkdir -p /var/log/3fs
   cp ~/3fs/build/bin/monitor_collector_main /opt/3fs/bin
   cp ~/3fs/configs/monitor_collector_main.toml /opt/3fs/etc

2. Update monitor_collector_main.toml to add a ClickHouse connection:
   toml Copy

   [server.monitor_collector.reporter]
   type = 'clickhouse'
   
   [server.monitor_collector.reporter.clickhouse]
   db = '3fs'
   host = '<CH_HOST>'
   passwd = '<CH_PASSWD>'
   port = '<CH_PORT>'
   user = '<CH_USER>'

3. Start monitor service:
   bash Copy

   cp ~/3fs/deploy/systemd/monitor_collector_main.service /usr/lib/systemd/system
   systemctl start monitor_collector_main

Note that

• Multiple instances of monitor services can be deployed behind a virtual IP address to share the traffic.
• Other services communicate with the monitor service over a TCP connection.

Step 3: Admin client Copy Chapter

Install admin_cli on all nodes.

1. Copy admin_cli to /opt/3fs/bin and config files to /opt/3fs/etc.
   bash Copy

   mkdir -p /opt/3fs/{bin,etc}
   rsync -avz meta:~/3fs/build/bin/admin_cli /opt/3fs/bin
   rsync -avz meta:~/3fs/configs/admin_cli.toml /opt/3fs/etc
   rsync -avz meta:/etc/foundationdb/fdb.cluster /opt/3fs/etc

2. Update admin_cli.toml to set cluster_id and clusterFile:
   toml Copy

   cluster_id = "stage"
   
   [fdb]
   clusterFile = '/opt/3fs/etc/fdb.cluster'

The full help documentation for admin_cli can be displayed by running the following command:

/opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml help

Step 4: Mgmtd service Copy Chapter

Install mgmtd service on meta node.

1. Copy mgmtd_main to /opt/3fs/bin and config files to /opt/3fs/etc.

   bash Copy

   cp ~/3fs/build/bin/mgmtd_main /opt/3fs/bin
   cp ~/3fs/configs/{mgmtd_main.toml,mgmtd_main_launcher.toml,mgmtd_main_app.toml} /opt/3fs/etc

2. Update config files:

   • Set mgmtd node_id = 1 in mgmtd_main_app.toml.
   • Edit mgmtd_main_launcher.toml to set the cluster_id and clusterFile:
   toml Copy

   cluster_id = "stage"
   
   [fdb]
   clusterFile = '/opt/3fs/etc/fdb.cluster'

   • Set monitor address in mgmtd_main.toml:
   toml Copy

   [common.monitor.reporters.monitor_collector]
   remote_ip = "192.168.1.1:10000"

3. Initialize the cluster:

   bash Copy

   /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml "init-cluster --mgmtd /opt/3fs/etc/mgmtd_main.toml 1 1048576 16"

   The parameters of admin_cli:

   • 1 the chain table ID
   • 1048576 the chunk size in bytes
   • 16 the file strip size

   Run help init-cluster for full documentation.

4. Start mgmtd service:

   bash Copy

   cp ~/3fs/deploy/systemd/mgmtd_main.service /usr/lib/systemd/system
   systemctl start mgmtd_main

5. Run list-nodes command to check if the cluster has been successfully initialized:

   bash Copy

   /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "list-nodes"

If multiple instances of mgmtd services deployed, one of the mgmtd services is elected as the primary; others are secondaries. Automatic failover occurs when the primary fails.

Step 5: Meta service Copy Chapter

Install meta service on meta node.

1. Copy meta_main to /opt/3fs/bin and config files to /opt/3fs/etc.
   bash Copy

   cp ~/3fs/build/bin/meta_main /opt/3fs/bin
   cp ~/3fs/configs/{meta_main_launcher.toml,meta_main.toml,meta_main_app.toml} /opt/3fs/etc

2. Update config files:
   • Set meta node_id = 100 in meta_main_app.toml.
   • Set cluster_id, clusterFile and mgmtd address in meta_main_launcher.toml:
   toml Copy

   cluster_id = "stage"
   
   [mgmtd_client]
   mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]

   • Set mgmtd and monitor addresses in meta_main.toml.
   toml Copy

   [server.mgmtd_client]
   mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]
   
   [common.monitor.reporters.monitor_collector]
   remote_ip = "192.168.1.1:10000"
   
   [server.fdb]
   clusterFile = '/opt/3fs/etc/fdb.cluster'

3. Config file of meta service is managed by mgmtd service. Use admin_cli to upload the config file to mgmtd:
   bash Copy

   /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "set-config --type META --file /opt/3fs/etc/meta_main.toml"

4. Start meta service:
   bash Copy

   cp ~/3fs/deploy/systemd/meta_main.service /usr/lib/systemd/system
   systemctl start meta_main

5. Run list-nodes command to check if meta service has joined the cluster:
   bash Copy

   /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "list-nodes"

If multiple instances of meta services deployed, meta requests will be evenly distributed to all instances.

Step 6: Storage service Copy Chapter

Install storage service on storage node.

1. Format the attached 16 SSDs as XFS and mount at /storage/data{1..16}, then create data directories /storage/data{1..16}/3fs and log directory /var/log/3fs.
   bash Copy

   mkdir -p /storage/data{1..16}
   mkdir -p /var/log/3fs
   for i in {1..16};do mkfs.xfs -L data${i} -s size=4096 /dev/nvme${i}n1;mount -o noatime,nodiratime -L data${i} /storage/data${i};done
   mkdir -p /storage/data{1..16}/3fs

2. Increase the max number of asynchronous aio requests:
   bash Copy

   sysctl -w fs.aio-max-nr=67108864

3. Copy storage_main to /opt/3fs/bin and config files to /opt/3fs/etc.
   bash Copy

   rsync -avz meta:~/3fs/build/bin/storage_main /opt/3fs/bin
   rsync -avz meta:~/3fs/configs/{storage_main_launcher.toml,storage_main.toml,storage_main_app.toml} /opt/3fs/etc

4. Update config files:
   • Set node_id in storage_main_app.toml. Each storage service is assigned a unique id between 10001 and 10005.
   • Set cluster_id and mgmtd address in storage_main_launcher.toml.
   toml Copy

   cluster_id = "stage"
   
   [mgmtd_client]
   mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]

   • Add target paths in storage_main.toml:
   toml Copy

   [server.mgmtd]
   mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]
   
   [common.monitor.reporters.monitor_collector]
   remote_ip = "192.168.1.1:10000"
   
   [server.targets]
   target_paths = ["/storage/data1/3fs","/storage/data2/3fs","/storage/data3/3fs","/storage/data4/3fs","/storage/data5/3fs","/storage/data6/3fs","/storage/data7/3fs","/storage/data8/3fs","/storage/data9/3fs","/storage/data10/3fs","/storage/data11/3fs","/storage/data12/3fs","/storage/data13/3fs","/storage/data14/3fs","/storage/data15/3fs","/storage/data16/3fs",]

5. Config file of storage service is managed by mgmtd service. Use admin_cli to upload the config file to mgmtd:
   bash Copy

   /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "set-config --type STORAGE --file /opt/3fs/etc/storage_main.toml"

6. Start storage service:
   bash Copy

   rsync -avz meta:~/3fs/deploy/systemd/storage_main.service /usr/lib/systemd/system
   systemctl start storage_main

7. Run list-nodes command to check if storage service has joined the cluster:
   text Copy

   /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "list-nodes"

Step 7: Create admin user, storage targets and chain table Copy Chapter

1. Create an admin user:
   bash Copy

   /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "user-add --root --admin 0 root"

   The admin token is printed to the console, save it to /opt/3fs/etc/token.txt.
2. Generate admin_cli commands to create storage targets on 5 storage nodes (16 SSD per node, 6 targets per SSD).
   • Follow instructions at here to install Python packages.
   bash Copy

   pip install -r ~/3fs/deploy/data_placement/requirements.txt
   python ~/3fs/deploy/data_placement/src/model/data_placement.py \
      -ql -relax -type CR --num_nodes 5 --replication_factor 3 --min_targets_per_disk 6
   python ~/3fs/deploy/data_placement/src/setup/gen_chain_table.py \
      --chain_table_type CR --node_id_begin 10001 --node_id_end 10005 \
      --num_disks_per_node 16 --num_targets_per_disk 6 \
      --target_id_prefix 1 --chain_id_prefix 9 \
      --incidence_matrix_path output/DataPlacementModel-v_5-b_10-r_6-k_3-λ_2-lb_1-ub_1/incidence_matrix.pickle

   The following 3 files will be generated in output directory: create_target_cmd.txt, generated_chains.csv, and generated_chain_table.csv.

3. Create storage targets:
   bash Copy

   /opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' --config.user_info.token $(<"/opt/3fs/etc/token.txt") < output/create_target_cmd.txt

4. Upload chains to mgmtd service:
   bash Copy

   /opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' --config.user_info.token $(<"/opt/3fs/etc/token.txt") "upload-chains output/generated_chains.csv"

5. Upload chain table to mgmtd service:
   bash Copy

   /opt/3fs/bin/admin_cli --cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' --config.user_info.token $(<"/opt/3fs/etc/token.txt") "upload-chain-table --desc stage 1 output/generated_chain_table.csv"

6. List chains and chain tables to check if they have been correctly uploaded:
   bash Copy

   /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "list-chains"
   /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "list-chain-tables"

Step 8: FUSE client Copy Chapter

For simplicity FUSE client is deployed on the meta node in this guide. However, we strongly advise against deploying clients on service nodes in production environment.

1. Copy hf3fs_fuse_main to /opt/3fs/bin and config files to /opt/3fs/etc.
   bash Copy

   cp ~/3fs/build/bin/hf3fs_fuse_main /opt/3fs/bin
   cp ~/3fs/configs/{hf3fs_fuse_main_launcher.toml,hf3fs_fuse_main.toml,hf3fs_fuse_main_app.toml} /opt/3fs/etc

2. Create the mount point:
   bash Copy

   mkdir -p /3fs/stage

3. Set cluster ID, mountpoint, token file and mgmtd address in hf3fs_fuse_main_launcher.toml
   toml Copy

   cluster_id = "stage"
   mountpoint = '/3fs/stage'
   token_file = '/opt/3fs/etc/token.txt'
   
   [mgmtd_client]
   mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]

4. Set mgmtd and monitor address in hf3fs_fuse_main.toml.
   toml Copy

   [mgmtd]
   mgmtd_server_addresses = ["RDMA://192.168.1.1:8000"]
   
   [common.monitor.reporters.monitor_collector]
   remote_ip = "192.168.1.1:10000"

5. Config file of FUSE client is also managed by mgmtd service. Use admin_cli to upload the config file to mgmtd:
   bash Copy

   /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://192.168.1.1:8000"]' "set-config --type FUSE --file /opt/3fs/etc/hf3fs_fuse_main.toml"

6. Start FUSE client:
   bash Copy

   cp ~/3fs/deploy/systemd/hf3fs_fuse_main.service /usr/lib/systemd/system
   systemctl start hf3fs_fuse_main

7. Check if 3FS has been mounted at /3fs/stage:
   bash Copy

   mount | grep '/3fs/stage'

FAQ Copy Chapter

How to troubleshoot admin_cli init-cluster error?

If mgmtd fails to start after running init-cluster, the most likely cause is an error in mgmtd_main.toml. Any changes to this file require clearing all FoundationDB data and re-running init-cluster

How to build a single-node cluster?

A minimum of two storage services is required for data replication. If set --num-nodes=1, the gen_chain_table.py script will fail. In a test environment, this limitation can be bypassed by deploying multiple storage services on a single machine.

How to update config files?

All config files are managed by mgmtd. If any *_main.toml is updated, such as storage_main.toml, the modified file should be uploaded using admin_cli set-config.

How to troubleshoot common deployment issues?

When encountering any error during deployment,

• Check the log messages in stdout/stderr using journalctl, especially during service startup.
• Check log files stored in /var/log/3fs/ on service and client nodes.
• Ensure that the directory /var/log/3fs/ exists before starting any service.

Overview (`docs/README.md`) Copy Section

Documentation

• Design Notes
• Setup Guide
• USRBIO API Reference
• P Specifications

Overview (`hf3fs_utils/README.md`) Copy Section

hf3fs_cli

build:

python3 setup_hf3fs_utils.py bdist_wheel

usage:

$ hf3fs_cli rmtree --help
Usage: hf3fs_cli rmtree [OPTIONS] [DIR_PATHS]...

  Move a directory tree to the trash and set an expiration time, it will be automatically deleted after expiration

  Example:
  hf3fs_cli rmtree <path/to/remove> --expire <expire_time>

  - Use --expire [1h|3h|8h|1d|3d|7d] to specify the expiration time, the directory will be deleted after expiration.
  - Before expiration, you can restore the directory from the trash using `hf3fs_cli mv <trash_path> <target_path>`.
  - If you need to free up space immediately, you can use `hf3fs_cli rmtree <trash_path>` to delete the data in the trash immediately, this operation cannot be undone!
  - Use `ls /path/to/hf3fs/trash` to view the trash.

Options:
  --expire [1h|3h|8h|1d|3d|7d]  Expiration time, contents in the trash will be automatically deleted after expiration
  -y, --yes                     Skip confirmation prompt and delete immediately
  --help                        Show this message and exit.

$ hf3fs_cli mv --help
Usage: hf3fs_cli mv [OPTIONS] OLD_PATH NEW_PATH

  Move files, supports moving files between different mount points within the same 3FS

Options:
  --help  Show this message and exit.

If you want to use rmtree command, the administrator needs to create a trash directory for each user at /{3fs_mountpoint}/trash/{user_name}. The cleanup of the trash directory is handled by the trash_cleaner. For instructions on how to use it, please refer to src/client/trash_cleaner/.

Overview (`specs/README.md`) Copy Section

P specifications

Build prerequisites Copy Chapter

Follow the official guide to install the P framework.

Or if dotnet has been installed, run the following command to store the p command.

dotnet tool restore

Run tests Copy Chapter

A helper script RunTests.ps1, implemented in PowerShell, is used to run tests and summarize the results.

DataStorage specifies the CRAQ implementation in 3FS.

PS > cd DataStorage
PS > ..\RunTests.ps1

...

-----------------------
Summary of test results
-----------------------
[02/26/2025 10:57:58] Elapsed time: 372.4s

test                                       status       seed schedules seconds min avg max
----                                       ------       ---- --------- ------- --- --- ---
tcOneClientWriteNoFailure[0]               pass   1402445568        10 15.8     -1  -1  -1
tcTwoClientsWriteNoFailure[0]              pass    189933208        10 23.6     -1  -1  -1
tcThreeClientsWriteNoFailure[0]            pass   3060254145        10 40.7     -1  -1  -1
tcOneClientWriteUnreliableDetector[0]      pass   2016460916        10 17.7     -1  -1  -1
tcTwoClientsWriteUnreliableDetector[0]     pass     18777396        10 24.7     -1  -1  -1
tcOneClientWriteWithFailure[0]             pass   2559323541        10 15.7     -1  -1  -1
tcTwoClientsWriteWithFailure[0]            pass   1199246267        10 29.9     -1  -1  -1
tcOneClientWriteWithFailures[0]            pass    672618818        10 15.4     -1  -1  -1
tcTwoClientsWriteWithFailures[0]           pass   1908913074        10 32.3     -1  -1  -1
tcOneClientWriteShortChainWithFailure[0]   pass   3031701162        10 6.3      -1  -1  -1
tcTwoClientsWriteShortChainWithFailures[0] pass   2907349611        10 16.6     -1  -1  -1
tcTwoClientsWriteLongChainWithFailures[0]  pass    260515276        10 67.0     -1  -1  -1

[02/26/2025 10:57:58] All tests passed

RDMASocket verifies the RDMA socket implementation in 3FS.

PS > cd RDMASocket
PS > ..\RunTests.ps1

...


-----------------------
Summary of test results
-----------------------
[02/26/2025 11:19:22] Elapsed time: 40.6s

test          status       seed schedules seconds min avg max
----          ------       ---- --------- ------- --- --- ---
tcPingPong[0] pass   3776118231        10 9.8      -1  -1  -1
tcOneWay[0]   pass    200216558        10 3.6      -1  -1  -1
tcTwoWay[0]   pass   1923093627        10 7.1      -1  -1  -1

[02/26/2025 11:19:22] All tests passed

Timer includes modified portions of the following open-source project:

• The original implementation of Timer is part of P tutorials licensed under MIT License.

Overview (`benchmarks/fio_usrbio/README.md`) Copy Section

FIO engine for 3FS USRBIO

This repository contains the fio external plugin used for benchmarking 3FS USRBIO.

Build Copy Chapter

First, build 3FS and fio.

Configure the following variables:

• HF3FS_LIB_DIR: directory contains libhf3fs_api_shared.so, the default path in 3FS repo is 3FS/build/src/lib/api.
• HF3FS_INCLUDE_DIR: directory contains hf3fs_usrbio.h, the default path in 3FS repo is 3FS/src/lib/api.
• FIO_SRC_DIR: directory contains config-host.h. After building fio, this header will be in the root of the fio repo.

Then run:

make HF3FS_LIB_DIR=${HF3FS_LIB_DIR} HF3FS_INCLUDE_DIR=${HF3FS_INCLUDE_DIR} FIO_SRC_DIR=${FIO_SRC_DIR}

You will get the external plugin as hf3fs_usrbio.so.

Usage Copy Chapter

To use this plugin, set the ioengine args in fio as external:hf3fs_usrbio.so. Please refer to fio documentation for further explanation.

Since multiple 3FS instances might be mounted, you need to specify the 3FS instance during benchmarking. This can be done by adding the mountpoint arguments (in the .fio):

mountpoint=/hf3fs/mount/point

To benchmark batched small I/Os, please set these four parameters to batch_size simultaneously:

iodepth=1024
iodepth_batch_submit=1024
iodepth_batch_complete_min=1024
iodepth_batch_complete_max=1024

Overview (`deploy/data_placement/README.md`) Copy Section

How to generate chain tables

Suppose we are going to setup a small 3FS cluster:

• 3 replicas for each chunk
• 5 storage nodes: 10001 ... 10005
• 16 SSDs attached to each node
• 6 storage targets on each SSD

Install dependencies Copy Chapter

The data placement problem is formulated using Pyomo and solved with the HiGHS solver. Install them and other dependencies:

$ cd deploy/data_placement
$ pip install -r requirements.txt

Generate chain table Copy Chapter

First generate a solution of the data placement problem.

/* Detailed source-code truncated for AI context efficiency. */

Note that some combinations of --num_nodes and --replication_factor may have no solution.

Then generate commands to create/remove storage targets and the chain table.

$ python src/setup/gen_chain_table.py --chain_table_type CR --node_id_begin 10001 --node_id_end 10005 --num_disks_per_node 16 --num_targets_per_disk 6 --incidence_matrix_path output/DataPlacementModel-v_5-b_10-r_6-k_3-λ_2-lb_1-ub_1/incidence_matrix.pickle

$ ls -1 output/
DataPlacementModel-v_5-b_10-r_6-k_3-λ_2-lb_1-ub_1
appsi_highs.log
create_target_cmd.txt
generated_chain_table.csv
generated_chains.csv
remove_target_cmd.txt

Overview (`specs/Timer/README.md`) Copy Section

Timer

This project includes modified portions of the following open-source project:

• P Licensed under the MIT License.
  • Source: the original implementation of Timer.

Overview (`src/core/README.md`) Copy Section

core is intended as a base component of mgmtd, meta, storage, and other concrete server components.

Overview (`src/simple_example/README.md`) Copy Section

How to create a new service

1. Copy src/simple_example to a new subfolder in src/.
2. Copy src/fbs/simple_example to a new subfolder in src/fbs/.
3. Rename simple_example to your_service_name.
4. Rename SimpleExample to YourSimpleExample.
5. Add the new cmake projects to CMakeLists.txt files in src/ and src/fbs/.

svr_name='migration'
SrvName='Migration'
mkdir -p "src/$svr_name" && pushd src/simple_example && cp -rf --parents . "../$svr_name" && popd
mkdir -p "src/fbs/$svr_name" && pushd src/fbs/simple_example && cp -rf --parents . "../$svr_name" && popd
find "src/$svr_name" "src/fbs/$svr_name" -type f | xargs sed -i "s/simple_example/$svr_name/g"
find "src/$svr_name" "src/fbs/$svr_name" -type f | xargs sed -i "s/SimpleExample/$SrvName/g"

Overview (`src/client/trash_cleaner/README.md`) Copy Section

trash_cleaner

usage:

trash_cleaner 0.1.0

USAGE:
    trash_cleaner [FLAGS] [OPTIONS] --interval <interval>

FLAGS:
        --abort-on-error    
    -h, --help              Prints help information
    -V, --version           Prints version information

OPTIONS:
    -i, --interval <interval>            Scan interval (in seconds), exit after one scan if set to 0
        --log <log>                      Log path, default is current directory [default: ./]
        --log-level <log-level>          Log level, default is info [default: info]
    -p, --paths <paths>...               path to trash directory
        --stdout-level <stdout-level>    stdout log level, default is warn [default: warn]

Overview (`src/storage/chunk_engine/README.md`) Copy Section

chunk-engine

Design Copy Section

1. The entire Chunk Engine can be divided into two components:
   1. Allocator: Responsible for allocating/reclaiming chunks and modifying memory states.
   2. MetaStore: Responsible for persisting allocation/reclamation events.
2. Workflow for writing a new chunk:
   1. The Allocator assigns a new chunk position, pointing to a disk space (purely in-memory operation).
   2. Write data to this chunk position. If a power failure or write failure occurs at this stage, no existing data is affected.
   3. Generate corresponding chunk metadata and persist it alongside the allocation event to the MetaStore. Using RocksDB's WriteBatch ensures atomic updates—the entire write operation either succeeds or fails, with no intermediate states.
3. Maintaining the Allocator's in-memory state:
   1. At startup, the Allocator quickly loads all allocation information from RocksDB.
   2. Allocation is performed in-memory first, followed by persistence. If a failure occurs before persistence, the allocation event is lost.
   3. Reclamation first persists the event to disk, then modifies the memory state. Even if a chunk deletion event is persisted, the chunk remains readable as long as memory holds its reference.
   4. This ensures conflict-free read/write operations: a read operation acquires a chunk reference, guaranteeing the chunk's validity until the read completes.
4. Use Arc to manage ownership of chunk position:
   1. For allocation, returns an Arc<ChunkPos>. If persistence fails, the position is automatically released when the Arc is dropped.
   2. Read operations also return an Arc<ChunkPos>, ensuring safe data access even during concurrent writes or deletions.

Allocator Copy Section

Storage hierarchy:

1. Chunk: Basic data unit, currently proposed as 64KB, 512KB, and 4MB.
2. Group: Each group contains 256 chunks (16MB, 128MB, or 1GB depending on chunk size).
3. File: For 512KB chunks, a single file (~120GB) contains ~960 groups.
4. Disk: Single disk capacity of 30TB, divided into 256 files per chunk size.
5. Node: A single node contains 10–20 disks.

This configuration supports up to ~1.2 billion chunks and ~5 million groups per machine.

Implementation details:

1. Each group uses a 256-bit bitset (4 uint64_t) to track allocation status.
2. Maintain three in-memory structures:
   • allocated_groups: Groups with allocated space but no chunks assigned.
   • unallocated_groups: Groups without allocated space.
   • active_groups: Map of <group_id, group_state> tracking allocation status.
3. Chunk allocation workflow:
   1. Prioritize finding free slots in active_groups using __builtin_ctz for fast bitwise operations.
   2. If active_groups is empty, acquire a new group from allocated_groups.
   3. If allocated_groups is empty, fetch a group from unallocated_groups and allocate disk space synchronously.
4. Background threads:
   • allocate_thread: Maintains active_groups within a target size range to ensure in-memory allocation efficiency.
   • compact_thread: Periodically scans active_groups, migrates all chunks from selected groups, releases space, and returns groups to allocated_groups.

MetaStore Copy Section

Persists three mappings:

1. chunk_id -> chunk_meta: Metadata includes chunk location, length, hash, version, etc., serialized using derse.
2. group_id -> group_state: Tracks chunk allocation status within groups, leveraging RocksDB's MergeOp for atomic updates.
3. chunk_pos -> chunk_id: Maps physical positions to chunk IDs, used by compact_thread during chunk migration.

Chunk Engine Copy Section

1. MetaCache: Maintains an in-memory chunk_id -> chunk_info mapping, where chunk_info includes chunk_meta and Arc<ChunkPos>.
2. Read operation: Returns chunk_info. The Arc<ChunkPos> ensures safe data access until the read completes.
3. Write operation workflow:
   1. Query MetaCache to retrieve the current chunk_info.
   2. Invoke Allocator::allocate() to obtain a new chunk position.
   3. Read existing chunk data, write it to the new chunk position, append the new write request, and generate new_chunk_info.
   4. Persist new_chunk_info to the MetaStore along with a release record for the original chunk position.

Design Notes (`docs/design_notes.md`) Copy Section

Design Notes

Design and implementation Copy Chapter

The 3FS system has four components: cluster manager, metadata service, storage service and client. All components are connected in an RDMA network (InfiniBand or RoCE).

Metadata and storage services send heartbeats to cluster manager. Cluster manager handles membership changes and distributes cluster configuration to other services and clients. Multiple cluster managers are deployed and one of them is elected as the primary. Another manager is promoted as primary when the primary fails. Cluster configuration is typically stored in a reliable distributed coordination service, such as ZooKeeper or etcd. In our production environment, we use the same key-value store as file metadata to reduce dependencies.

File metadata operations (e.g. open or create files/directories) are sent to metadata services, which implement the file system semantics. Metadata services are stateless, since file metadata are stored in a transactional key-value store (e.g. FoundationDB). Clients can connect to any metadata service.

Each storage service manages a few local SSDs and provides a chunk store interface. The storage service implements Chain Replication with Apportioned Queries (CRAQ) to ensure strong consistency. CRAQ’s write-all-read-any approach helps to unleash the throughput of SSDs and RDMA network. A 3FS file is split into equally sized chunks, which are replicated over multiple SSDs.

Two clients are developed for applications: FUSE client and native client. Most applications use FUSE client, which has a low adoption barrier. Performance-critical applications are integrated with the native client.

File system interfaces Copy Chapter

Object store is becoming a popular option for data analytics and machine learning. However, file system semantics and a unified namespace where files are organized in directories provide greater flexibility for applications.

• Atomic directory manipulation An object store can approximate hierarchical directory structures by using slashes (/) in object keys. However, it doesn’t natively support operations like atomically moving files/directories, or recursively deleting entire directories. Actually a common pattern in our internal applications involves creating a temporary directory, writing files to it, and then moving the directory to its final location. When handling a large number of small files, the recursive delete for directories is crucial. Without it, applications have to traverse each directory and remove files one by one.

• Symbolic and hard links Our applications utilize symbolic and hard links to create lightweight snapshots of dynamically updated datasets, where new data is appended as individual files.

• Familiar interface The file interface is well known and used everywhere. There is no need to learn a new storage API. Many datasets are stored as CSV/Parquet files. Adapting file-based data loaders to use the 3FS FUSE client or native client is straightforward.

Limitations of FUSE Copy Section

FUSE (Filesystem in Userspace) simplifies file system client development by redirecting I/O operations to user-space processes through the FUSE kernel module. It creates the illusion that applications are accessing the remote file system as if it were a local file system. However, it has performance limitations:

• Memory copy overhead The user-space file system daemon cannot access application memory. Data transfer between kernel and user spaces consumes memory bandwidth and increases end-to-end latency.

• Primitive multi-threading support When an application initiates I/O requests, FUSE places these requests into a multi-threaded shared queue, protected by a spin lock. The user-space file system daemon then retrieves and processes requests from this queue. Due to lock contention, FUSE’s I/O processing capability fails to scale with the number of threads. Our benchmark results indicate that FUSE only handles approximately 400K 4KiB reads per second. Further increasing concurrency does not improve performance as lock contention intensifies. perf profiling reveals that the kernel-space spin lock consumes a significant amount of CPU time.

Most applications, e.g. data analytics, perform large block writes on 3FS or they can buffer data in memory and flush it to 3FS when write buffer is full. However, FUSE on Linux 5.x does not support concurrent writes to the same file^1. Applications overcome this limitation by writing to multiple files concurrently, maximizing the total throughput.

Read operations exhibit more complex patterns. Some training jobs require random access to dataset samples, with read sizes varying from a few kilobytes to several megabytes per sample. And samples are typically not 4K-aligned in files. Data loaders are specifically designed to fetch batches of samples. But they perform poorly when handling small random reads on FUSE-mounted 3FS. Bandwidth of SSDs and RDMA network are not fully utilized.

Asynchronous zero-copy API Copy Section

Implementing the file system client as a VFS kernel module avoids performance issues mentioned above. But kernel module development is significantly more challenging than user-space system programming. Bugs are difficult to diagnose and can lead to catastrophic failures in production environments. For example, machines may crash and leave no log message for debugging. When upgrading a kernel module, all processes using the file system must be stopped cleanly; otherwise, a machine restart is required.

For these reasons, we have chosen to implement a native client within the FUSE daemon. This client offers an interface that supports asynchronous zero-copy I/O operations. File meta operations are still handled by FUSE daemon (e.g. open/close/stat files). Applications call open() to obtain a file descriptor (fd) and register it via native API. They can then perform I/O operations on the file with native client. This approach ensures consistency in metadata operations with the POSIX API, making it easier to migrate existing code.

The asynchronous, zero-copy API is inspired by Linux io_uring. Below are the key data structures in the API:

• Iov A large memory region for zero-copy read/write operations, shared between the user process and the native client. InfiniBand memory registration is managed by the client. In native API, all read data will be read into Iov, and all write data should be written to Iov before calling the API.

• Ior A small shared ring buffer for communication between user process and native client. The usage of Ior is similar to Linux io_uring, where the user process enqueues read/write requests, and the native client dequeues these requests for completion. The requests are executed in batches, with their sizes controlled by the io_depth parameter. Multiple batches are processed in parallel, whether from different rings or the same ring. However, multiple rings are still recommended for multi-threaded applications, as sharing a ring requires synchronization, which can impact performance.

Within the native client, multiple threads are spawned to fetch I/O requests from the Iors. These requests are batched and dispatched to storage services, reducing RPC overhead caused by small read requests.

File metadata store Copy Chapter

Location of file chunks Copy Section

3FS divides file data into equally sized chunks and stripes them across multiple replication chains (replication chains and chain tables are defined in Section Data placement). Users can specify the chain table, chunk size, and stripe size for files on a per-directory basis. Each chunk is independently stored on multiple storage services, with its chunk ID generated by concatenating the file’s inode id and chunk index.

When creating a new file, the metadata service employs a round-robin strategy to select consecutive replication chains from the designated chain table, based on the stripe size. Next, a random seed is generated to shuffle the selected chains. This allocation strategy ensures balanced data distribution across chains and SSDs.

When an application opens a file, the client contacts the meta service to obtain the file’s data layout information. Then the client can independently compute chunk IDs and chains for data operations, minimizing the involvement of the meta service in the critical path.

File metadata on transactional key-value store Copy Section

3FS uses FoundationDB as its distributed storage system for metadata. FoundationDB provides a key-value store interface and supports transactions with Serializable Snapshot Isolation (SSI). 3FS stores all metadata as key-value pairs in FoundationDB. Meta services follow a stateless architecture, greatly enhancing maintainability by allowing administrators to seamlessly upgrade or restart services without disruption. When clients experience request failures or timeouts, they can automatically fail over to other available services.

The file system metadata primarily consists of two core structures: inodes and directory entries. Inodes store attribute information for files, directories, and symbolic links, each identified by a globally unique 64-bit identifier that increments monotonically. Inode keys are constructed by concatenating the "INOD" prefix with the inode id, which is encoded in little-endian byte order to spread inodes over multiple FoundationDB nodes. The inode values vary by its type:

• All inode types contain basic attributes: ownership, permissions, access/modification/change times.

• Additional attributes for file inodes: file length, chunk size, selected range in chain table, shuffle seed.

• Additional attributes for directory inodes: the parent directory’s inode id, default layout configurations for subdirectories/files (chain table, chunk size, stripe size). The parent’s inode id is required to detect loops when moving directories. When moving dir_a/dir_b to dir_c/, we need to ensure that dir_c is not a descendant of dir_b, which can be achieved by checking all ancestors of dir_c upward.

• Additional attributes for symbolic link inodes: target path string.

Directory entry keys are composed of a "DENT" prefix, the parent inode ID, and the entry name. Directory entry values store the target inode id and inode type. All entries within a directory naturally form a contiguous key range, allowing efficient directory listing via range queries.

The meta operations leverage FoundationDB’s transactions:

• Read-only transactions used for metadata queries: fstat, lookup, listdir etc.

• Read-write transactions used for metadata updates: create, link, unlink, rename etc.

For write transactions, FoundationDB tracks the read/write key sets to form conflict detection sets. When concurrent transaction conflicts are detected, the meta service automatically retries the transaction. This design enables multiple meta services to process requests in parallel while maintaining file system metadata consistency.

Dynamic file attributes Copy Section

On most local file systems, deleting an opened file is deferred until all associated file descriptors are closed. Consequently, it is necessary to track all file descriptors of the file. Training jobs open a large number of files during startup. Storing all file descriptors would impose heavy load on meta service and FoundationDB. Since training jobs do not depend on this feature, 3FS does not track file descriptors opened in read-only mode.

3FS maintains a file session for each file descriptor (fd) opened in write mode since deleting write opened files may lead to unreclaimable garbage chunks from concurrent writes. When a file with active write sessions is deleted, meta service delays the deletion until all its fds are closed. To prevent lingering sessions from offline clients, the 3FS meta service periodically checks client liveness and cleans up sessions of offline clients.

The file length is stored in the inode. For files being actively updated, the length stored in inode may diverge from the actual length. Clients periodically (5 seconds by default) report to meta service maximum write position of each file opened in write mode. If this position exceeds the length in inode and there is no concurrent truncate operation, this position is adopted as the new file length.

Due to the possibility of concurrent writes from multiple clients, the method described above ensures only eventual consistency for file lengths. When processing close/fsync operations, the meta service obtains the precise file length by querying the ID and length of the last chunk from the storage service. Since file data is striped across multiple chains, this operation incurs non-negligible overhead.

Concurrent updates to the same file’s length by multiple meta services may cause transaction conflicts and lead to repeated file length computation. To mitigate this, meta service distributes file length update tasks across multiple meta services using inode IDs and the rendezvous hash algorithm.

Our production environments use a large stripe size: 200. For small files, the number of chains containing file chunks is well below this number. The number of potentially used chains is stored in file inode and used as a hint when updating the length. It starts with an initial value of 16 and is doubled each time additional file chunks are written to more chains. This allows us to avoid querying all 200 chains when updating lengths of small files. This optimization can also be extended to the deletion of small files.

Chunk storage system Copy Chapter

The design goal of chunk storage system is to achieve the highest bandwidth possible even when there are storage medium failures. The read/write throughput of 3FS should scale linearly with the number of SSDs and bisection network bandwidth between clients and storage services. Applications access storage services in a locality-oblivious manner.

Data placement Copy Section

Each file chunk is replicated over a chain of storage targets using chain replication with apportioned queries (CRAQ). In CRAQ write requests are sent to the head target and propagated along a chain. Read requests can be sent to any of the storage target. Usually the read traffic is evenly distributed among all targets in a chain for better load balance. Multiple storage targets are created on each SSD and the targets join different chains.

Suppose there are 6 nodes: A, B, C, D, E, F. Each node has 1 SSD. Create 5 storage targets on each SSD: 1, 2, ... 5. Then there are 30 targets in total: A1, A2, A3, ..., F5. If each chunk has 3 replicas, a chain table is constructed as follows.

Each chain has a version number. The version number is incremented if the chain is changed (e.g. a storage target is offline). Only the primary cluster manager makes changes to chain tables.

A few chain tables can be constructed to support different data placement requirements. For example, two chain tables can be created, one for batch/offline jobs and another for online services. The two tables consist of storage targets on mutually exclusive nodes and SSDs.

Logically, the state of each chain changes independently. Each chain can be included in multiple chain tables. The concept of chain table is created to let metadata service pick a table for each file and stripe file chunks across chains in the table.

Balanced traffic during recovery Copy Section

Suppose read traffic is evenly distributed among all storage targets in the above chain table. When A fails its read requests would be redirected to B and C. Under heavy load the read bandwidth of B, C is immediately saturated and B, C become the bottleneck of the entire system. Replacing a failed SSD and syncing data to the new SSD can take several hours. The read throughput is impaired during this period.

To reduce the performance impact, we can have more SSDs share the redirected traffic. In the following chain table, A is paired with every other SSDs. When A fails, each of the other SSDs receives 1/5 of A’s read traffic.

To achieve maximum read throughput during recovery, the load balance problem can be formulated as a balanced incomplete block design. The optimal solution is obtained by using integer programming solver.

Data replication Copy Section

CRAQ is a write-all-read-any replication protocol optimized for read-heavy workloads. Utilizing read bandwidth of all replicas is critical to achieve highest read throughput in an all-flash storage system.

When a write request is received by a storage service, it goes through the following steps:

1. The service checks if the chain version in write request matches with the latest known version; reject the request if it’s not. The write request could be sent by a client or a predecessor in the chain.

2. The service issues RDMA Read operations to pull write data. If the client/predecessor fails, the RDMA Read operations may time out and the write is aborted.

3. Once the write data is fetched into local memory buffer, a lock for the chunk to be updated is acquired from a lock manager. Concurrent writes to the same chunk are blocked. All writes are serialized at the head target.

4. The service reads the committed version of the chunk into memory, applies the update, and stores the updated chunk as a pending version. A storage target may store two versions of a chunk: a committed version and a pending version. Each version has a monotonically-increasing version number. The version numbers of committed version and pending versions are v and u respectively, and satisfy u = v + 1.

5. If the service is the tail, the committed version is atomically replaced by the pending version and an acknowledgment message is sent to the predecessor. Otherwise, the write request is forwarded to the successor. When the committed version is updated, the current chain version is stored as a field in the chunk metadata.

6. When an acknowledgment message arrives at a storage service, the service replaces the committed version with the pending version and continues to propagate the message to its predecessor. The local chunk lock is then released.

Suppose there are 3 targets in the chain: A, B, C. A write request has just entered step 5 at A. A forwards the request to successor B. Then B instantly fails and the forwarded write request is lost. When cluster manager detects B’s failure, it marks B as offline and moves it to the end of chain and broadcasts the updated chain table. Once A receives the latest chain table, it forwards the write request to the new successor C. C may not receive the latest chain table yet and rejects the request. But A can keep forwarding the request to C. Eventually C gets the latest chain table and accepts the request.

When a read request arrives at a storage service:

1. When the service only has a committed version of the chunk, this version is returned to the client.

2. Unlike CRAQ, our implementation does not issue version query to the tail target. When there are both committed and pending versions, the service replies a special status code to notify the client. The client may wait for a short interval and retry. Or the client can issue a relaxed read request to get the pending version.

Failure detection Copy Section

The cluster manager relies on heartbeats to detect fail-stop failures. Cluster manager declares a service failed if it does not receive heartbeats from it for a configurable interval (e.g. T seconds). A service stops processing requests and exits if it cannot communicate with cluster manager for T/2 seconds. The heartbeat can be seen as a request to *renew a lease* granted by the manager.

The metadata services are stateless. The list of online meta services provided by cluster manager is a simple service discovery mechanism that helps clients create connections to metadata services. If one meta service is down, the clients may switch to any other metadata service.

Cluster manager plays a more critical role in membership changes of storage services. It maintains a global view of chain tables and storage targets’ states. Each storage target has a public state and a local state.

Public state indicates if it’s ready to serve read requests and if write requests would be propagated to it. Public states are stored with chain tables and distributed to services and clients.

Local state is only known by storage services and cluster manager, and it’s stored in the memory of cluster manager. If a storage target has medium failure, the related service sets the target’s local state to offline in heartbeat. If a storage service is down, storage targets managed by the service are marked offline.

A storage target can change from one public state to another in response to the latest local state. The local state plays the role of a triggering event. The cluster manager periodically scans every chain and updates the public states of targets on the chain according to a state-transition table.

• The chain version is incremented if the chain is updated.

• If a storage target is marked offline, it’s moved to the end of chain.

• If a storage service finds public state of any local storage target is lastsrv or offline, it exits immediately. The service may be isolated from the cluster manager by network partition error.

• Once the data recovery of a storage target in syncing state is completed, the storage service set the target’s local state to up-to-date in subsequent heartbeat messages sent to cluster manager.

Data recovery Copy Section

When a storage service exits (e.g. process crashes or restarts during upgrade), or a storage medium failure occurs, all related storage targets will be marked as offline and moved to the end of chains by cluster manager. Once the service restarts, each target on the service enters into the recovery process independently. The entire recovery process overlaps with normal activity and minimizes any interruption.

When a previously offline storage service starts:

1. The service periodically pulls latest chain tables from cluster manager. But it does not send heartbeats until all its storage targets have been marked offline in the latest chain tables. This ensures all its targets would go through the data recovery process.

2. When a write request arrives during recovery, the request is always a full-chunk-replace write. The local committed version is updated and any existing pending version is abandoned. Since current service is the tail, an acknowledgment message is sent to the predecessor. The full state of the predecessor is copied to the returning service through a continuous stream of full-chunk-replace writes.

3. Before the data recovery of a storage target starts, the predecessor sends a dump-chunkmeta request to the returning service. Then the service iterates the local chunk metadata store to collect the ids, chain versions and committed/pending version numbers of all chunks on the target, and replies the collected metadata to the predecessor.

4. When a sync-done message arrives, the service knows that the storage target is up-to-date. It sets local state of the target to up-to-date in heartbeat messages sent to cluster manager.

When a storage service finds a previously offline successor is online:

1. The service starts to forward normal write requests to the successor. Clients may only update a portion of the chunk, but the forwarded write requests should contain the whole chunk, i.e. a full-chunk-replace write.

2. The service sends a dump-chunkmeta request to the successor. Once the metadata of all chunks on the successor target are received, it collects the chunk metadata on its local target. Then it compares the two copies of chunk metadata to decide which chunks should be transferred.

3. The selected chunks are transferred to the successor by issuing full-chunk-replace write requests.

   • The chunk lock is first acquired for each chunk.

   • The chain version, committed version number and chunk content are read and transferred to successor by sending a full-chunk-replace request.

   • The chunk lock is released.

4. When all required chunks have been transferred, a sync-done message is sent to the successor.

The rules used to decide which chunks should be transferred are:

• If a chunk only exists on the local target, it should be transferred.

• If a chunk only exists on the remote target, it should be removed.

• If the chain version of local chunk replica is greater than that of the remote chunk replica, it should be transferred.

• If the chain versions of local/remote chunk replicas are the same but local committed version number does not equal to the remote pending version number, it should be transferred.

• Otherwise, two chunk replicas are either the same or being updated by in-progress write requests.

Chunks and the metadata Copy Section

File chunks are stored in the chunk engine. On each SSD, the persistent storage of the chunk engine consists of a fixed number of data files for storing chunk data, and a RocksDB instance for maintaining chunk metadata and other system information. Additionally, the chunk engine maintains an in-memory cache of chunk metadata to enhance query performance. A chunk allocator is implemented for fast allocation of new chunks. The chunk engine interface provides thread-safe access through the following operations:

1. open/close Initializes the engine by loading metadata from RocksDB and reconstructing chunk allocator states.

2. get Retrieves chunk metadata and reference-counted handle through a hashmap cache, enabling concurrent access with O(1) average complexity.

3. update Implements copy-on-write (COW) semantics by allocating new chunks before modifying data. Old chunks remain readable until all handles are released.

4. commit Commit the updated chunk metadata to RocksDB via write batches to ensure atomic updates; synchronously refresh the chunk metadata cache.

The chunk data will ultimately be stored on physical blocks. Physical block sizes range from 64KiB to 64MiB in increments of powers of two, totaling 11 distinct sizes. The allocator will assign physical blocks whose sizes most closely match the actual chunk size. A resource pool is constructed for each physical block size, with each pool containing 256 physical files. The usage status of physical blocks is maintained in memory using bitmaps. When a physical block is reclaimed, its bitmap flag is set to 0. The actual storage space of the block remains preserved and will be prioritized for subsequent allocations. When no available physical blocks remain, fallocate() will be used to allocate a contiguous large space in physical files, creating 256 new physical blocks - this approach helps reduce disk fragmentation.

When performing write operations on a chunk, the allocator first assigns a new physical block. The system then reads existing chunk data into a buffer, applies the update, and writes the updated buffer to the newly allocated block. An optimized process is implemented for appends, where data is directly added in-place at the end of the existing block. A new copy of metadata is constructed from the new block's location and existing chunk metadata. Subsequently, both the new chunk metadata and statuses of new and old physical blocks are atomically updated in RocksDB.

Metrics (`docs/metrics.md`) Copy Section

Metric System of 3FS

Type of Metrics Copy Chapter

3FS calculates metrics in each service and store metrics to ClickHouse through monitor service. There are some types of metrics in 3FS, and the metric may or may not be reset after each report.

There are many types of metrics:

List of Metrics Copy Chapter

Following is a partial list of the metrics and their meanings:

--- METRICS ---

• Files Extracted: 14
• Estimated Token Budget: ~20305 tokens
• Canonical Reference: https://codewiki.google/github.com/deepseek-ai/3FS