## File: README.md This work presents **Depth Anything 3 (DA3)**, a model that predicts spatially consistent geometry from arbitrary visual inputs, with or without known camera poses. In pursuit of minimal modeling, DA3 yields two key insights: - ๐Ÿ’Ž A **single plain transformer** (e.g., vanilla DINO encoder) is sufficient as a backbone without architectural specialization, - โœจ A singular **depth-ray representation** obviates the need for complex multi-task learning. ๐Ÿ† DA3 significantly outperforms [DA2](https://github.com/DepthAnything/Depth-Anything-V2) for monocular depth estimation, and [VGGT](https://github.com/facebookresearch/vggt) for multi-view depth estimation and pose estimation. All models are trained exclusively on **public academic datasets**. ## ๐Ÿ“ฐ News - **11-12-2025:** ๐Ÿš€ New models and [**DA3-Streaming**](da3_streaming/README.md) released! Handle ultra-long video sequence inference with less than 12GB GPU memory via sliding-window streaming inference. Special thanks to [Kai Deng](https://github.com/DengKaiCQ) for his contribution to DA3-Streaming! - **08-12-2025:** ๐Ÿ“Š [Benchmark evaluation pipeline](docs/BENCHMARK.md) released! Evaluate pose estimation & 3D reconstruction on 5 datasets. - **30-11-2025:** Add [`use_ray_pose`](#use-ray-pose) and [`ref_view_strategy`](docs/funcs/ref_view_strategy.md) (reference view selection for multi-view inputs). - **25-11-2025:** Add [Awesome DA3 Projects](#-awesome-da3-projects), a community-driven section featuring DA3-based applications. - **14-11-2025:** Paper, project page, code and models are all released. ## โœจ Highlights ### ๐Ÿ† Model Zoo We release three series of models, each tailored for specific use cases in visual geometry. - ๐ŸŒŸ **DA3 Main Series** (`DA3-Giant`, `DA3-Large`, `DA3-Base`, `DA3-Small`) These are our flagship foundation models, trained with a unified depth-ray representation. By varying the input configuration, a single model can perform a wide range of tasks: + ๐ŸŒŠ **Monocular Depth Estimation**: Predicts a depth map from a single RGB image. + ๐ŸŒŠ **Multi-View Depth Estimation**: Generates consistent depth maps from multiple images for high-quality fusion. + ๐ŸŽฏ **Pose-Conditioned Depth Estimation**: Achieves superior depth consistency when camera poses are provided as input. + ๐Ÿ“ท **Camera Pose Estimation**: Estimates camera extrinsics and intrinsics from one or more images. + ๐ŸŸก **3D Gaussian Estimation**: Directly predicts 3D Gaussians, enabling high-fidelity novel view synthesis. - ๐Ÿ“ **DA3 Metric Series** (`DA3Metric-Large`) A specialized model fine-tuned for metric depth estimation in monocular settings, ideal for applications requiring real-world scale. - ๐Ÿ” **DA3 Monocular Series** (`DA3Mono-Large`). A dedicated model for high-quality relative monocular depth estimation. Unlike disparity-based models (e.g., [Depth Anything 2](https://github.com/DepthAnything/Depth-Anything-V2)), it directly predicts depth, resulting in superior geometric accuracy. ๐Ÿ”— Leveraging these available models, we developed a **nested series** (`DA3Nested-Giant-Large`). This series combines a any-view giant model with a metric model to reconstruct visual geometry at a real-world metric scale. ### ๐Ÿ› ๏ธ Codebase Features Our repository is designed to be a powerful and user-friendly toolkit for both practical application and future research. - ๐ŸŽจ **Interactive Web UI & Gallery**: Visualize model outputs and compare results with an easy-to-use Gradio-based web interface. - โšก **Flexible Command-Line Interface (CLI)**: Powerful and scriptable CLI for batch processing and integration into custom workflows. - ๐Ÿ’พ **Multiple Export Formats**: Save your results in various formats, including `glb`, `npz`, depth images, `ply`, 3DGS videos, etc, to seamlessly connect with other tools. - ๐Ÿ”ง **Extensible and Modular Design**: The codebase is structured to facilitate future research and the integration of new models or functionalities. ## ๐Ÿš€ Quick Start ### ๐Ÿ“ฆ Installation ```bash pip install xformers torch\>=2 torchvision pip install -e . # Basic pip install --no-build-isolation git+https://github.com/nerfstudio-project/gsplat.git@0b4dddf04cb687367602c01196913cde6a743d70 # for gaussian head pip install -e ".[app]" # Gradio, python>=3.10 pip install -e ".[all]" # ALL ``` For detailed model information, please refer to the [Model Cards](#-model-cards) section below. ### ๐Ÿ’ป Basic Usage ```python import glob, os, torch from depth_anything_3.api import DepthAnything3 device = torch.device("cuda") model = DepthAnything3.from_pretrained("depth-anything/DA3NESTED-GIANT-LARGE") model = model.to(device=device) example_path = "assets/examples/SOH" images = sorted(glob.glob(os.path.join(example_path, "*.png"))) prediction = model.inference( images, ) # prediction.processed_images : [N, H, W, 3] uint8 array print(prediction.processed_images.shape) # prediction.depth : [N, H, W] float32 array print(prediction.depth.shape) # prediction.conf : [N, H, W] float32 array print(prediction.conf.shape) # prediction.extrinsics : [N, 3, 4] float32 array # opencv w2c or colmap format print(prediction.extrinsics.shape) # prediction.intrinsics : [N, 3, 3] float32 array print(prediction.intrinsics.shape) ``` ```bash export MODEL_DIR=depth-anything/DA3NESTED-GIANT-LARGE # This can be a Hugging Face repository or a local directory # If you encounter network issues, consider using the following mirror: export HF_ENDPOINT=https://hf-mirror.com # Alternatively, you can download the model directly from Hugging Face export GALLERY_DIR=workspace/gallery mkdir -p $GALLERY_DIR # CLI auto mode with backend reuse da3 backend --model-dir ${MODEL_DIR} --gallery-dir ${GALLERY_DIR} # Cache model to gpu da3 auto assets/examples/SOH \ --export-format glb \ --export-dir ${GALLERY_DIR}/TEST_BACKEND/SOH \ --use-backend # CLI video processing with feature visualization da3 video assets/examples/robot_unitree.mp4 \ --fps 15 \ --use-backend \ --export-dir ${GALLERY_DIR}/TEST_BACKEND/robo \ --export-format glb-feat_vis \ --feat-vis-fps 15 \ --process-res-method lower_bound_resize \ --export-feat "11,21,31" # CLI auto mode without backend reuse da3 auto assets/examples/SOH \ --export-format glb \ --export-dir ${GALLERY_DIR}/TEST_CLI/SOH \ --model-dir ${MODEL_DIR} ``` The model architecture is defined in [`DepthAnything3Net`](src/depth_anything_3/model/da3.py), and specified with a Yaml config file located at [`src/depth_anything_3/configs`](src/depth_anything_3/configs). The input and output processing are handled by [`DepthAnything3`](src/depth_anything_3/api.py). To customize the model architecture, simply create a new config file (*e.g.*, `path/to/new/config`) as: ```yaml __object__: path: depth_anything_3.model.da3 name: DepthAnything3Net args: as_params net: __object__: path: depth_anything_3.model.dinov2.dinov2 name: DinoV2 args: as_params name: vitb out_layers: [5, 7, 9, 11] alt_start: 4 qknorm_start: 4 rope_start: 4 cat_token: True head: __object__: path: depth_anything_3.model.dualdpt name: DualDPT args: as_params dim_in: &head_dim_in 1536 output_dim: 2 features: &head_features 128 out_channels: &head_out_channels [96, 192, 384, 768] ``` Then, the model can be created with the following code snippet. ```python from depth_anything_3.cfg import create_object, load_config Model = create_object(load_config("path/to/new/config")) ``` ## ๐Ÿ“š Useful Documentation - ๐Ÿ–ฅ๏ธ [Command Line Interface](docs/CLI.md) - ๐Ÿ“‘ [Python API](docs/API.md) - ๐Ÿ“Š [Benchmark Evaluation](docs/BENCHMARK.md) ## ๐Ÿ—‚๏ธ Model Cards Generally, you should observe that DA3-LARGE achieves comparable results to VGGT. The Nested series uses an Any-view model to estimate pose and depth, and a monocular metric depth estimator for scaling. โš ๏ธ Models with the `-1.1` suffix are retrained after fixing a training bug; prefer these refreshed checkpoints. The original `DA3NESTED-GIANT-LARGE`, `DA3-GIANT`, and `DA3-LARGE` remain available but are deprecated. You could expect much better performance for street scenes with the `-1.1` models. | ๐Ÿ—ƒ๏ธ Model Name | ๐Ÿ“ Params | ๐Ÿ“Š Rel. Depth | ๐Ÿ“ท Pose Est. | ๐Ÿงญ Pose Cond. | ๐ŸŽจ GS | ๐Ÿ“ Met. Depth | โ˜๏ธ Sky Seg | ๐Ÿ“„ License | |-------------------------------|-----------|---------------|--------------|---------------|-------|---------------|-----------|----------------| | **Nested** | | | | | | | | | | [DA3NESTED-GIANT-LARGE-1.1](https://huggingface.co/depth-anything/DA3NESTED-GIANT-LARGE-1.1) | 1.40B | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | CC BY-NC 4.0 | | [DA3NESTED-GIANT-LARGE](https://huggingface.co/depth-anything/DA3NESTED-GIANT-LARGE) | 1.40B | โœ… | โœ… | โœ… | โœ… | โœ… | โœ… | CC BY-NC 4.0 | | **Any-view Model** | | | | | | | | | | [DA3-GIANT-1.1](https://huggingface.co/depth-anything/DA3-GIANT-1.1) | 1.15B | โœ… | โœ… | โœ… | โœ… | | | CC BY-NC 4.0 | | [DA3-GIANT](https://huggingface.co/depth-anything/DA3-GIANT) | 1.15B | โœ… | โœ… | โœ… | โœ… | | | CC BY-NC 4.0 | | [DA3-LARGE-1.1](https://huggingface.co/depth-anything/DA3-LARGE-1.1) | 0.35B | โœ… | โœ… | โœ… | | | | CC BY-NC 4.0 | | [DA3-LARGE](https://huggingface.co/depth-anything/DA3-LARGE) | 0.35B | โœ… | โœ… | โœ… | | | | CC BY-NC 4.0 | | [DA3-BASE](https://huggingface.co/depth-anything/DA3-BASE) | 0.12B | โœ… | โœ… | โœ… | | | | Apache 2.0 | | [DA3-SMALL](https://huggingface.co/depth-anything/DA3-SMALL) | 0.08B | โœ… | โœ… | โœ… | | | | Apache 2.0 | | | | | | | | | | | | **Monocular Metric Depth** | | | | | | | | | | [DA3METRIC-LARGE](https://huggingface.co/depth-anything/DA3METRIC-LARGE) | 0.35B | โœ… | | | | โœ… | โœ… | Apache 2.0 | | | | | | | | | | | | **Monocular Depth** | | | | | | | | | | [DA3MONO-LARGE](https://huggingface.co/depth-anything/DA3MONO-LARGE) | 0.35B | โœ… | | | | | โœ… | Apache 2.0 | ## โ“ FAQ - **Monocular Metric Depth**: To obtain metric depth in meters from `DA3METRIC-LARGE`, use `metric_depth = focal * net_output / 300.`, where `focal` is the focal length in pixels (typically the average of fx and fy from the camera intrinsic matrix K). Note that the output from `DA3NESTED-GIANT-LARGE` is already in meters. - **Ray Head (`use_ray_pose`)**: Our API and CLI support `use_ray_pose` arg, which means that the model will derive camera pose from ray head, which is generally slightly slower, but more accurate. Note that the default is `False` for faster inference speed. AUC3 Results for DA3NESTED-GIANT-LARGE | Model | HiRoom | ETH3D | DTU | 7Scenes | ScanNet++ | |-------|------|-------|-----|---------|-----------| | `ray_head` | 84.4 | 52.6 | 93.9 | 29.5 | 89.4 | | `cam_head` | 80.3 | 48.4 | 94.1 | 28.5 | 85.0 | - **Older GPUs without XFormers support**: See [Issue #11](https://github.com/ByteDance-Seed/Depth-Anything-3/issues/11). Thanks to [@S-Mahoney](https://github.com/S-Mahoney) for the solution! ## ๐Ÿข Awesome DA3 Projects A community-curated list of Depth Anything 3 integrations across 3D tools, creative pipelines, robotics, and web/VR viewers, including but not limited to these. You are welcome to submit your DA3-based project via PR, and we will review and feature it if applicable. - [DA3-blender](https://github.com/xy-gao/DA3-blender): Blender addon for DA3-based 3D reconstruction from a set of images. - [ComfyUI-DepthAnythingV3](https://github.com/PozzettiAndrea/ComfyUI-DepthAnythingV3): ComfyUI nodes for Depth Anything 3, supporting single/multi-view and video-consistent depth with optional pointโ€‘cloud export. - [DA3-ROS2-Wrapper](https://github.com/GerdsenAI/GerdsenAI-Depth-Anything-3-ROS2-Wrapper): Real-time DA3 depth in ROS2 with multi-camera support. - [DA3-ROS2-CPP-TensorRT](https://github.com/ika-rwth-aachen/ros2-depth-anything-v3-trt): DA3 ROS2 C++ TensorRT Inference Node: a ROS2 node for DA3 depth estimation using TensorRT for real-time inference. - [VideoDepthViewer3D](https://github.com/amariichi/VideoDepthViewer3D): Streaming videos with DA3 metric depth to a Three.js/WebXR 3D viewer for VR/stereo playback. ## ๐Ÿง‘โ€๐Ÿ’ป Official Codebase Core Contributors and Maintainers | **Bingyi Kang** | Haotong Lin | Sili Chen | Jun Hao Liew | Donny Y. Chen | Kai Deng | | --- | --- | --- | --- | --- | --- | ## ๐Ÿ“ Citations If you find Depth Anything 3 useful in your research or projects, please cite our work: ``` @article{depthanything3, title={Depth Anything 3: Recovering the visual space from any views}, author={Haotong Lin and Sili Chen and Jun Hao Liew and Donny Y. Chen and Zhenyu Li and Guang Shi and Jiashi Feng and Bingyi Kang}, journal={arXiv preprint arXiv:2511.10647}, year={2025} } ``` --- ## File: da3_streaming/README.md This repo introduces a streaming pipeline that enables `Depth Anything 3` to process โญ **`long video sequences`** and **`super-large scale scenes`** โญ under tight CPU/GPU memory budgets by chunking frames and managing state across chunks. Built on the ideas of `VGGT-Long`, it focuses on memory efficiency and stable online inference for near-real-time video processing. `DA3-Streaming` is built on the [VGGT-Long](https://github.com/DengKaiCQ/VGGT-Long) and [Depth Anything 3](https://github.com/ByteDance-Seed/Depth-Anything-3). ### **Updates** `[11 Nov 2025]` Code of `DA3-Streaming` release. ## Setup, Installation & Running ### ๐Ÿ“ฎ 1 - Clone this project Clone the repo using the `--recursive` flag ```cmd git clone --recursive https://github.com/ByteDance-Seed/Depth-Anything-3.git ``` If you forgot `--recursive` ```cmd cd /Depth-Anything-3/ git submodule update --init --recursive . ``` ### ๐Ÿ“ฆ 2 - Environment Setup #### Step 1: Dependency Installation Install `Depth-Anything-3` first. ```cmd pip install -r requirements.txt ``` #### Step 2: Weights Download Download all the pre-trained weights needed: ```cmd bash ./scripts/download_weights.sh ``` #### System dependencies you may encounter. If you encounter an error about `libGL.so.1` (the error comes from `opencv-python`), please run the following cmd to install the system dependencies. ```cmd sudo apt-get install -y libgl1-mesa-glx ``` ### ๐Ÿš€ 3 - Running the code ```cmd python da3_streaming.py --image_dir ./path_of_images ``` or ```cmd python da3_streaming.py --image_dir ./path_of_images --config ./configs/base_config.yaml --output_dir ${OUTPUT_DIR} ``` You may run the following cmd if you got videos before `python da3_streaming.py`. ``` mkdir ./extract_images ffmpeg -i your_video.mp4 -vf "fps=5,scale=640:-1" ./extract_images/frame_%06d.png ``` ### 4 - Outputs #### Basic Outputs After running the code, you will get the following outputs: - `${OUTPUT_DIR}/camera_poses.txt`: The camera poses file. Each line contains the extrinsic matrix parameters of a frame. - `${OUTPUT_DIR}/intrinsic.txt`: The intrinsic parameters of the camera. Each line contains fx, fy, cx, cy of a frame. - `${OUTPUT_DIR}/pcd/combined_pcd.ply`: The combined point cloud file. It contains the 3D points from all frames. #### Additional Outputs If setting `save_depth_conf_result` in config to `True`, you will get outputs for each frame: - `${OUTPUT_DIR}/results_output`: The folder that contains the rgb, depth, confidence and intrinsic results for each frame. Note that the minimum value of confidence is 0. To verify the results, you can use the following cmd to fuse point cloud from `./results_output`. The fused point cloud file will be saved as `${OUTPUT_DIR}/output.ply`. ```cmd python npz_output_process.py --npz_folder ${OUTPUT_DIR}/results_output --pose_file ${OUTPUT_DIR}/camera_poses.txt --output_file ${OUTPUT_DIR}/output.ply ``` **Note on Space Requirements**: Please ensure your machine has sufficient disk space before running the code for `DA3-Streaming`. When finishing, the code will delete these intermediate results to prevent excessive disk usage. ## Experiment Results We conducted some additional experiments to compare the performance differences among different architectures. Below is the comparison of `ATE RMSE [m]` on KITTI Odometry between `DA3-Streaming`, `VGGT-Long` and `Pi-Long`. All methods are evaluated with overlap equal to half chunk size, comparable resolution (~500px-width), and loop closure with similarity threshold 0.85. | **Method** | **chunk size**| **AVG** | **AVG (w/o 01)** | **00** | **01** | **02** | **03** | **04** | **05** | **06** | **07** | **08** | **09** | **10** | |:-------------------------------:|:--------:|:--------:|:----------------:|:--------:|:--------:|:--------:|:------:|:------:|:-------:|:-------:|:-------:|:-------:|:--------:|:-------:| | **Num. of Frames** | | 2109 | 2210 | 4542 | 1101 | 4661 | 801 | 271 | 2761 | 1101 | 1101 | 4071 | 1591 | 1201 | | **VGGT-Long** |120| 25.60 | 22.81 | 16.13 | 53.43 | 51.98 | 4.37 | 2.15 | 12.69 | 11.33 | 3.60 | 70.29 | 34.55 | 21.05 | | **Pi-Long** |120| 21.17 | 11.81 | 5.55 | 114.83 | 50.29 | 1.63 | 1.11 | 3.48 | 2.88 | 3.92 | 24.25 | 7.38 | 17.61 | | **DA3-Streaming** |120| 18.63 | **10.42** | 4.48 | 100.77 | 33.41 | 3.58 | 2.39 | 3.95 | 7.59 | 2.09 | 31.20 | 8.06 | 7.44 | | **VGGT-Long** |60| 26.36 | 19.30 | 8.06 | 96.96 | 34.16 | 6.83 | 4.16 | 9.15 | 4.68 | 2.68 | 63.15 | 32.24 | 27.87 | | **Pi-Long** |60| 30.63 | 17.10 | 7.82 | 165.92 | 73.59 | 3.67 | 0.91 | 5.16 | 3.89 | 3.57 | 33.97 | 17.01 | 21.41 | | **DA3-Streaming** |60| **16.83** | 10.64 | 5.13 | 78.76 | 35.64 | 5.38 | 3.18 | 3.04 | 2.83 | 2.32 | 26.55 | 8.86 | 13.42 | In `DA3-Streaming`, we restructured the code and accelerated it using GPU technology. Currently, our method achieves a running speed of nearly `10 FPS` without the Keyframe strategy (the test time has excluded warm-up, model loading and ply result saving). Following results are evaluated on KITTI sequences 00, 05, and 08, totaling 11,373 frames, using an NVIDIA A100 GPU. | **Method** | **Time** | **FPS** | |:------------------:|:------------------------:|:------------------------:| | **VGGT-Long** | 65min 08sec | 2.91 | | **Pi-long** | 60min 09sec | 3.15 | | **DA3-Streaming** | 22min 17sec |**8.51** | Although the current pipeline is not an SLAM system, `DA3-Streaming` still has a certain degree of accuracy compared to the uncalibrated SLAM method on TUM RGB-D. `DA3-Streaming`, `VGGT-Long` and `Pi-Long` are evaluated with chunk size 120, overlap 60, comparable resolution (~500px-width) and with loop closure. | **Methods** | **AVG** | **360** | **desk** | **desk2** | **floor** | **plant** | **room** | **rpy** | **teddy** | **xyz** | |:----------------------------:|:-------:|:--------:|:---------:|:---------:|:---------:|:--------:|:-------:|:---------:|:-------:|:-------:| | **Droid-SLAM Uncalibrated** | 0.163 | 0.202 | 0.032 | 0.091 | 0.064 | 0.045 | 0.918 | 0.056 | 0.045 | 0.012 | | **Mast3r-SLAM Uncalibrated** | **0.060** | 0.070 | 0.035 | 0.055 | 0.056 | 0.035 | 0.118 | 0.041 | 0.114 | 0.020 | | **VGGT-Long** | 0.110 | 0.118 | 0.058 | 0.111 | 0.118 | 0.071 | 0.155 | 0.140 | 0.120 | 0.099 | | **Pi-long** | 0.094 | 0.115 | 0.047 | 0.052 | 0.160 | 0.085 | 0.114 | 0.143 | 0.081 | 0.052 | | **DA3-Streaming** | 0.087 | 0.059 | 0.034 | 0.042 | 0.107 | 0.060 | 0.105 | 0.206 | 0.126 | 0.044 | We also evaluate `DA3-Streaming` with different chunk sizes on KITTI (w/o 01) with resolution 504x154 and TUM RGB-D with resolution 504x378. Overlap is set to half of chunk size. | | **Chunk size** | **120** | **90** | **60** | **30** | |:-------:|:---------:|:--------:|:-----:|:------:|:------:| | **KITTI (504x154)** | **Peak VRAM [GB]** | 15.9 | 14.3 | 12.7 | 11.5 | || **ATE RMSE [m]** | 10.42 | 9.38 | 10.64 | 19.39 | | **TUM RGB-D (504x378)** | **Peak VRAM [GB]** | 28.3 | 25.1 | 21.2 | 18.7 | || **ATE RMSE [m]** | 0.087 | 0.091 | 0.127 | 0.227 | ## Acknowledgements Our project is based on [VGGT-Long](https://github.com/DengKaiCQ/VGGT-Long) and [Depth Anything 3](https://github.com/ByteDance-Seed/Depth-Anything-3). --- ## File: docs/funcs/ref_view_strategy.md # ๐Ÿ“ Reference View Selection Strategy ## ๐Ÿ“– Overview Reference view selection is a component in multi-view depth estimation. When processing multiple input views, the model needs to determine which view should serve as the primary reference frame for depth prediction, defining the world coordinate system. Different reference view will leads to different reconstruction results. This is a known consideration in multi-view geometry and was analyzed in [PI3](https://arxiv.org/abs/2507.13347). The choice of reference view can affect the quality and consistency of depth predictions across the scene. ## ๐Ÿš€ Our Simple Solution: Automatic Reference View Selection DA3 provides a simple approach to address this through **automatic reference view selection** based on **class tokens**. Instead of relying on heuristics or manual selection, the model analyzes the class token features from all input views and intelligently selects the most suitable reference frame. --- ## ๐ŸŽจ Available Strategies ### 1. โš–๏ธ `saddle_balanced` (Recommended, Default) **Philosophy:** Select a view that achieves balance across multiple feature metrics. This strategy looks for a "middle ground" view that is neither too similar nor too different from other views, making it a stable reference point. **How it works:** 1. Extracts and normalizes class tokens from all views 2. Computes three complementary metrics for each view: - **Similarity score**: Average cosine similarity with other views - **Feature norm**: L2 norm of the original features - **Feature variance**: Variance across feature dimensions 3. Normalizes each metric to [0, 1] range 4. Selects the view closest to 0.5 (median) across all three metrics ### 2. ๐ŸŽข `saddle_sim_range` **Philosophy:** Select a view with the largest similarity range to other views. This identifies "saddle point" views that are highly similar to some views but dissimilar to others, making them information-rich anchor points. **How it works:** 1. Computes pairwise cosine similarity between all views 2. For each view, calculates the range (max - min) of similarities to other views 3. Selects the view with the maximum similarity range --- ### 3. 1๏ธโƒฃ `first` (Not Recommended) **Philosophy:** Always use the first view in the input sequence as the reference. **How it works:** Simply returns index 0. **When to use:** - โ›” **Not recommended** in general - ๐Ÿ”ง Only use when you have manually pre-sorted your views and know the first view is optimal - ๐Ÿ› Debugging or baseline comparisons --- ### 4. โธ๏ธ `middle` **Philosophy:** Select the view in the middle of the input sequence. **How it works:** Returns the view at index `S // 2` where S is the number of views. **When to use:** - โฑ๏ธ **Only recommended when input images are temporally ordered** - ๐ŸŽฌ Video sequences (e.g., **DA3-LONG** setting) - ๐Ÿ“น Sequential captures where the middle frame likely has the most stable viewpoint **Specific use case: DA3-LONG** ๐ŸŽฌ In video-based depth estimation scenarios (like DA3-LONG), where inputs are consecutive frames, `middle` is often the **optimal choice** because that it has maximum overlap with all other frames. ## ๐Ÿ’ป Usage ### ๐Ÿ Python API ```python from depth_anything_3 import DepthAnything3 model = DepthAnything3.from_pretrained("depth-anything/DA3NESTED-GIANT-LARGE") # Use default (saddle_balanced) prediction = model.inference( images, ref_view_strategy="saddle_balanced" ) # For video sequences, consider using middle prediction = model.inference( video_frames, ref_view_strategy="middle" # Good for temporal sequences ) # For complex scenes with wide baselines prediction = model.inference( images, ref_view_strategy="saddle_sim_range" ) ``` ### ๐Ÿ–ฅ๏ธ Command Line Interface ```bash # Default (saddle_balanced) da3 auto input/ --export-dir output/ # Explicitly specify strategy da3 auto input/ --ref-view-strategy saddle_balanced # For video processing da3 video input.mp4 --ref-view-strategy middle # For wide-baseline multi-view da3 images captures/ --ref-view-strategy saddle_sim_range ``` --- ### ๐ŸŽฏ When Selection Is Applied Reference view selection is applied when: - 3๏ธโƒฃ Number of views S โ‰ฅ 3 --- ## ๐Ÿ’ก Recommendations ### ๐Ÿ“‹ Quick Guide | Scenario | Recommended Strategy | Rationale | |----------|---------------------|-----------| | **Default / Unknown** | `saddle_balanced` | Robust, balanced, works well across diverse scenarios | | **Video frames** | `middle` | Temporal coherence, stable middle frame | | **Wide-baseline multi-view** | `saddle_sim_range` | Maximizes information coverage | | **Pre-sorted inputs** | `first` | Use only if you've manually optimized ordering | | **Single image** | `first` | Automatically used (no reordering needed for S โ‰ค 2) | ### โœจ Best Practices 1. ๐ŸŽฏ **Start with defaults**: `saddle_balanced` works well in most cases 2. ๐ŸŽฌ **Consider your input type**: Use `middle` for videos, `saddle_balanced` for photos 3. ๐Ÿ”ฌ **Experiment if needed**: Try different strategies if results are suboptimal 4. ๐Ÿ“Š **Monitor performance**: Check `glb` quality and consistency across views. --- ## ๐Ÿ”ง Technical Details ### ๐ŸŽš๏ธ Selection Threshold The reference view selection is only triggered when: ```python num_views >= 3 # At least 3 views required ``` For 1-2 views, no reordering is performed (equivalent to using `first`). ### โš™๏ธ Implementation The selection happens at layer `alt_start - 1` in the vision transformer, before the first global attention layer. This ensures the selected reference view influences the entire depth prediction pipeline. --- ## โ“ FAQ **Q: ๐Ÿค” Why is this feature provided?** A: The model can handle any view order, but this feature provides automatic optimization for reference view selection, which can help improve depth prediction quality in multi-view scenarios. **Q: โฑ๏ธ Does this add computational cost?** A: The overhead is totally negligible. **Q: ๐ŸŽฎ Can I manually specify which view to use as reference?** A: Not directly through this parameter. You can pre-sort your input images to place your preferred reference view first and use `ref_view_strategy="first"`. **Q: โš™๏ธ What happens if I don't specify this parameter?** A: The default `saddle_balanced` strategy is used automatically. **Q: ๐Ÿ“Š Is this feature used in the DA3 paper benchmarks?** A: No, the paper used `first` as the default strategy for all multi-view experiments. The current default has been updated to `saddle_balanced` for better robustness. --- ## File: docs/API.md # ๐Ÿ“š DepthAnything3 API Documentation ## ๐Ÿ“‘ Table of Contents 1. [๐Ÿ“– Overview](#overview) 2. [๐Ÿ’ก Usage Examples](#usage-examples) 3. [๐Ÿ”ง Core API](#core-api) - [DepthAnything3 Class](#depthanything3-class) - [inference() Method](#inference-method) 4. [โš™๏ธ Parameters](#parameters) - [Input Parameters](#input-parameters) - [Pose Alignment Parameters](#pose-alignment-parameters) - [Feature Export Parameters](#feature-export-parameters) - [Rendering Parameters](#rendering-parameters) - [Processing Parameters](#processing-parameters) - [Export Parameters](#export-parameters) 5. [๐Ÿ“ค Export Formats](#export-formats) 6. [โ†ฉ๏ธ Return Value](#return-value) ## ๐Ÿ“– Overview This documentation provides comprehensive API reference for DepthAnything3, including usage examples, parameter specifications, export formats, and advanced features. It covers both basic pose and depth estimation workflows and advanced pose-conditioned processing with multiple export capabilities. ## ๐Ÿ’ก Usage Examples Here are quick examples to get you started: ### ๐Ÿš€ Basic Depth Estimation ```python from depth_anything_3.api import DepthAnything3 # Initialize and run inference model = DepthAnything3.from_pretrained("depth-anything/DA3NESTED-GIANT-LARGE").to("cuda") prediction = model.inference(["image1.jpg", "image2.jpg"]) ``` ### ๐Ÿ“ท Pose-Conditioned Depth Estimation ```python import numpy as np # With camera parameters for better consistency prediction = model.inference( image=["image1.jpg", "image2.jpg"], extrinsics=extrinsics_array, # (N, 4, 4) intrinsics=intrinsics_array # (N, 3, 3) ) ``` ### ๐Ÿ“ค Export Results ```python # Export depth data and 3D visualization prediction = model.inference( image=image_paths, export_dir="./output", export_format="mini_npz-glb" ) ``` ### ๐Ÿ” Feature Extraction ```python # Export intermediate features from specific layers prediction = model.inference( image=image_paths, export_dir="./output", export_format="feat_vis", export_feat_layers=[0, 1, 2] # Export features from layers 0, 1, 2 ) ``` ### โœจ Advanced Export with Gaussian Splatting ```python # Export multiple formats including Gaussian Splatting # Note: infer_gs=True requires da3-giant or da3nested-giant-large model model = DepthAnything3(model_name="da3-giant").to("cuda") prediction = model.inference( image=image_paths, extrinsics=extrinsics_array, intrinsics=intrinsics_array, export_dir="./output", export_format="npz-glb-gs_ply-gs_video", align_to_input_ext_scale=True, infer_gs=True, # Required for gs_ply and gs_video exports ) ``` ### ๐ŸŽจ Advanced Export with Feature Visualization ```python # Export with intermediate feature visualization prediction = model.inference( image=image_paths, export_dir="./output", export_format="mini_npz-glb-depth_vis-feat_vis", export_feat_layers=[0, 5, 10, 15, 20], feat_vis_fps=30, ) ``` ### ๐Ÿ“ Using Ray-Based Pose Estimation ```python # Use ray-based pose estimation instead of camera decoder prediction = model.inference( image=image_paths, export_dir="./output", export_format="glb", use_ray_pose=True, # Enable ray-based pose estimation ) ``` ### ๐ŸŽฏ Reference View Selection ```python # For multi-view inputs, automatically select the best reference view prediction = model.inference( image=image_paths, ref_view_strategy="saddle_balanced", # Default: balanced selection ) # For video sequences, use middle frame as reference prediction = model.inference( image=video_frames, ref_view_strategy="middle", # Good for temporally ordered inputs ) ``` ## ๐Ÿ”ง Core API ### ๐Ÿ”จ DepthAnything3 Class The main API class that provides depth estimation capabilities with optional pose conditioning. #### ๐ŸŽฏ Initialization ```python from depth_anything_3 import DepthAnything3 # Initialize the model with a model name model = DepthAnything3(model_name="da3-large") model = model.to("cuda") # Move to GPU ``` **Parameters:** - `model_name` (str, default: "da3-large"): The name of the model preset to use. - **Available models:** - ๐Ÿฆพ `"da3-giant"` - 1.15B params, any-view model with GS support - โญ `"da3-large"` - 0.35B params, any-view model (recommended for most use cases) - ๐Ÿ“ฆ `"da3-base"` - 0.12B params, any-view model - ๐Ÿชถ `"da3-small"` - 0.08B params, any-view model - ๐Ÿ‘๏ธ `"da3mono-large"` - 0.35B params, monocular depth only - ๐Ÿ“ `"da3metric-large"` - 0.35B params, metric depth with sky segmentation - ๐ŸŽฏ `"da3nested-giant-large"` - 1.40B params, nested model with all features ### ๐Ÿš€ inference() Method The primary inference method that processes images and returns depth predictions. ```python prediction = model.inference( image=image_list, extrinsics=extrinsics_array, # Optional intrinsics=intrinsics_array, # Optional align_to_input_ext_scale=True, # Whether to align predicted poses to input scale infer_gs=True, # Enable Gaussian branch for gs exports use_ray_pose=False, # Use ray-based pose estimation instead of camera decoder ref_view_strategy="saddle_balanced", # Reference view selection strategy render_exts=render_extrinsics, # Optional renders for gs_video render_ixts=render_intrinsics, # Optional renders for gs_video render_hw=(height, width), # Optional renders for gs_video process_res=504, process_res_method="upper_bound_resize", export_dir="output_directory", # Optional export_format="mini_npz", export_feat_layers=[], # List of layer indices to export features from conf_thresh_percentile=40.0, # Confidence threshold percentile for depth map in GLB export num_max_points=1_000_000, # Maximum number of points to export in GLB export show_cameras=True, # Whether to show cameras in GLB export feat_vis_fps=15, # Frames per second for feature visualization in feat_vis export export_kwargs={} # Optional, additional arguments to export functions. export_format:key:val, see 'Parameters/Export Parameters' for details ) ``` ## โš™๏ธ Parameters ### ๐Ÿ“ธ Input Parameters #### `image` (required) - **Type**: `List[Union[np.ndarray, Image.Image, str]]` - **Description**: List of input images. Can be numpy arrays, PIL Images, or file paths. - **Example**: ```python # From file paths image = ["image1.jpg", "image2.jpg", "image3.jpg"] # From numpy arrays image = [np.array(img1), np.array(img2)] # From PIL Images image = [Image.open("image1.jpg"), Image.open("image2.jpg")] ``` #### `extrinsics` (optional) - **Type**: `Optional[np.ndarray]` - **Shape**: `(N, 4, 4)` where N is the number of input images - **Description**: Camera extrinsic matrices (world-to-camera transformation). When provided, enables pose-conditioned depth estimation mode. - **Note**: If not provided, the model operates in standard depth estimation mode. #### `intrinsics` (optional) - **Type**: `Optional[np.ndarray]` - **Shape**: `(N, 3, 3)` where N is the number of input images - **Description**: Camera intrinsic matrices containing focal length and principal point information. When provided, enables pose-conditioned depth estimation mode. ### ๐ŸŽฏ Pose Alignment Parameters #### `align_to_input_ext_scale` (default: True) - **Type**: `bool` - **Description**: When True the predicted extrinsics are replaced with the input ones and the depth maps are rescaled to match their metric scale. When False the function returns the internally aligned poses computed via Umeyama alignment. #### `infer_gs` (default: False) - **Type**: `bool` - **Description**: Enable Gaussian Splatting branch for gaussian splatting exports. Required when using `gs_ply` or `gs_video` export formats. #### `use_ray_pose` (default: False) - **Type**: `bool` - **Description**: Use ray-based pose estimation instead of camera decoder for pose prediction. When True, the model uses ray prediction heads to estimate camera poses; when False, it uses the camera decoder approach. #### `ref_view_strategy` (default: "saddle_balanced") - **Type**: `str` - **Description**: Strategy for selecting the reference view from multiple input views. Options: `"first"`, `"middle"`, `"saddle_balanced"`, `"saddle_sim_range"`. Only applied when number of views โ‰ฅ 3. See [detailed documentation](funcs/ref_view_strategy.md) for strategy comparisons. - **Available strategies**: - `"saddle_balanced"`: Selects view with balanced features across multiple metrics (recommended default) - `"saddle_sim_range"`: Selects view with largest similarity range - `"first"`: Always uses first view (not recommended, equivalent to no reordering for views < 3) - `"middle"`: Uses middle view (recommended for video sequences) ### ๐Ÿ” Feature Export Parameters #### `export_feat_layers` (default: []) - **Type**: `List[int]` - **Description**: List of layer indices to export intermediate features from. Features are stored in the `aux` dictionary of the Prediction object with keys like `feat_layer_0`, `feat_layer_1`, etc. ### ๐ŸŽฅ Rendering Parameters These arguments are only used when exporting Gaussian-splatting videos (include `"gs_video"` in `export_format`). They describe an auxiliary camera trajectory with ``M`` views. #### `render_exts` (optional) - **Type**: `Optional[np.ndarray]` - **Shape**: `(M, 4, 4)` - **Description**: Camera extrinsics for the synthesized trajectory. If omitted, the exporter falls back to the predicted poses. #### `render_ixts` (optional) - **Type**: `Optional[np.ndarray]` - **Shape**: `(M, 3, 3)` - **Description**: Camera intrinsics for each rendered frame. Leave `None` to reuse the input intrinsics. #### `render_hw` (optional) - **Type**: `Optional[Tuple[int, int]]` - **Description**: Explicit output resolution `(height, width)` for the rendered frames. Defaults to the input resolution when not provided. ### โšก Processing Parameters #### `process_res` (default: 504) - **Type**: `int` - **Description**: Base resolution for processing. The model will resize images to this resolution for inference. #### `process_res_method` (default: "upper_bound_resize") - **Type**: `str` - **Description**: Method for resizing images to the target resolution. - **Options**: - `"upper_bound_resize"`: Resize so that the specified dimension (504) becomes the longer side - `"lower_bound_resize"`: Resize so that the specified dimension (504) becomes the shorter side - **Example**: - Input: 1200ร—1600 โ†’ Output: 378ร—504 (with `process_res=504`, `process_res_method="upper_bound_resize"`) - Input: 504ร—672 โ†’ Output: 504ร—672 (no change needed) ### ๐Ÿ“ฆ Export Parameters #### `export_dir` (optional) - **Type**: `Optional[str]` - **Description**: Directory path where exported files will be saved. If not provided, no files will be exported. #### `export_format` (default: "mini_npz") - **Type**: `str` - **Description**: Format for exporting results. Supports multiple formats separated by `-`. - **Example**: `"mini_npz-glb"` exports both mini_npz and glb formats. #### ๐ŸŒ GLB Export Parameters These parameters are passed directly to the `inference()` method and only apply when `export_format` includes `"glb"`. ##### `conf_thresh_percentile` (default: 40.0) - **Type**: `float` - **Description**: Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out from the point cloud. ##### `num_max_points` (default: 1,000,000) - **Type**: `int` - **Description**: Maximum number of points in the exported point cloud. If the point cloud exceeds this limit, it will be downsampled. ##### `show_cameras` (default: True) - **Type**: `bool` - **Description**: Whether to include camera wireframes in the exported GLB file for visualization. #### ๐ŸŽจ Feature Visualization Parameters These parameters are passed directly to the `inference()` method and only apply when `export_format` includes `"feat_vis"`. ##### `feat_vis_fps` (default: 15) - **Type**: `int` - **Description**: Frame rate for the output video when visualizing features across multiple images. #### โœจ๐ŸŽฅ 3DGS and 3DGS Video Parameters These parameters are passed directly to the `inference()` method and only apply when `export_format` includes `"gs_ply"` or `"gs_video"`. ##### `export_kwargs` (default: `{}`) - Type: `dict[str, dict[str, Any]]` - Description: Per-format extra arguments passed to export functions, mainly for `"gs_ply"` and `"gs_video"`. - Access pattern: `export_kwargs[export_format][key] = value` - Example: ```python { "gs_ply": { "gs_views_interval": 1, }, "gs_video": { "trj_mode": "interpolate_smooth", "chunk_size": 1, "vis_depth": None, }, } ``` ## ๐Ÿ“ค Export Formats The API supports multiple export formats for different use cases: ### ๐Ÿ“Š `mini_npz` - **Description**: Minimal NPZ format containing essential data - **Contents**: `depth`, `conf`, `exts`, `ixts` - **Use case**: Lightweight storage for depth data with camera parameters ### ๐Ÿ“ฆ `npz` - **Description**: Full NPZ format with comprehensive data - **Contents**: `depth`, `conf`, `exts`, `ixts`, `image`, etc. - **Use case**: Complete data export for advanced processing ### ๐ŸŒ `glb` - **Description**: 3D visualization format with point cloud and camera poses - **Contents**: - Point cloud with colors from original images - Camera wireframes for visualization - Confidence-based filtering and downsampling - **Use case**: 3D visualization, inspection, and analysis - **Features**: - Automatic sky depth handling - Confidence threshold filtering - Background filtering (black/white) - Scene scale normalization - **Parameters** (passed via `inference()` method directly): - `conf_thresh_percentile` (float, default: 40.0): Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out. - `num_max_points` (int, default: 1,000,000): Maximum number of points in the exported point cloud. If exceeded, points will be downsampled. - `show_cameras` (bool, default: True): Whether to include camera wireframes in the exported GLB file for visualization. ### โœจ `gs_ply` - **Description**: Gaussian Splatting point cloud format - **Contents**: 3DGS data in PLY format. Compatible with standard 3DGS viewers such as [SuperSplat](https://superspl.at/editor) (recommended), [SPARK](https://sparkjs.dev/viewer/). - **Use case**: Gaussian Splatting reconstruction - **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models. - **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)): - `gs_views_interval`: Export to 3DGS every N views, default: `1`. ### ๐ŸŽฅ `gs_video` - **Description**: Rasterized 3DGS to obtain videos - **Contents**: A video of 3DGS-rasterized views using either provided viewpoints or a predefined camera trajectory. - **Use case**: Video rendering for Gaussian Splatting - **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models. - **Note**: Can optionally use `render_exts`, `render_ixts`, and `render_hw` parameters in `inference()` method to specify novel viewpoints. - **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)): - `extrinsics`: Optional world-to-camera poses for novel views. Falls back to the predicted poses of input views if not provided. (Alternatively, use `render_exts` parameter in `inference()`) - `intrinsics`: Optional camera intrinsics for novel views. Falls back to the predicted intrinsics of input views if not provided. (Alternatively, use `render_ixts` parameter in `inference()`) - `out_image_hw`: Optional output resolution `H x W`. Falls back to input resolution if not provided. (Alternatively, use `render_hw` parameter in `inference()`) - `chunk_size`: Number of views rasterized per batch. Default: `8`. - `trj_mode`: Predefined camera trajectory for novel-view rendering. - `color_mode`: Same as `render_mode` in [gsplat](https://docs.gsplat.studio/main/apis/rasterization.html#gsplat.rasterization). - `vis_depth`: How depth is combined with RGB. Default: `hcat` (horizontal concatenation). - `enable_tqdm`: Whether to display a tqdm progress bar during rendering. - `output_name`: File name of the rendered video. - `video_quality`: Video quality to save. Default: `high`. - `high`: High quality video (default) - `medium`: Medium quality video (balance of storage space and quality) - `low`: Low quality video (fewer storage space) ### ๐Ÿ” `feat_vis` - **Description**: Feature visualization format - **Contents**: PCA-visualized intermediate features from specified layers - **Use case**: Model interpretability and feature analysis - **Note**: Requires `export_feat_layers` to be specified - **Parameters** (passed via `inference()` method directly): - `feat_vis_fps` (int, default: 15): Frame rate for the output video when visualizing features across multiple images. ### ๐ŸŽจ `depth_vis` - **Description**: Depth visualization format - **Contents**: Color-coded depth maps alongside original images - **Use case**: Visual inspection of depth estimation quality ### ๐Ÿ”— Multiple Format Export You can export multiple formats simultaneously by separating them with `-`: ```python # Export both mini_npz and glb formats export_format = "mini_npz-glb" # Export multiple formats export_format = "npz-glb-gs_ply" ``` ## โ†ฉ๏ธ Return Value The `inference()` method returns a `Prediction` object with the following attributes: ### ๐Ÿ“Š Core Outputs - **depth**: `np.ndarray` - Estimated depth maps with shape `(N, H, W)` where N is the number of images, H is height, and W is width. - **conf**: `np.ndarray` - Confidence maps with shape `(N, H, W)` indicating prediction reliability (optional, depends on model). ### ๐Ÿ“ท Camera Parameters - **extrinsics**: `np.ndarray` - Camera extrinsic matrices with shape `(N, 3, 4)` representing world-to-camera transformations. Only present if camera poses were estimated or provided as input. - **intrinsics**: `np.ndarray` - Camera intrinsic matrices with shape `(N, 3, 3)` containing focal length and principal point information. Only present if poses were estimated or provided as input. ### ๐ŸŽ Additional Outputs - **processed_images**: `np.ndarray` - Preprocessed input images with shape `(N, H, W, 3)` in RGB format (0-255 uint8). - **aux**: `dict` - Auxiliary outputs including: - `feat_layer_X`: Intermediate features from layer X (if `export_feat_layers` was specified) - `gaussians`: 3D Gaussian Splats data (if `infer_gs=True`) ### ๐Ÿ’ป Usage Example ```python prediction = model.inference(image=["img1.jpg", "img2.jpg"]) # Access depth maps depth_maps = prediction.depth # shape: (2, H, W) # Access confidence if hasattr(prediction, 'conf'): confidence = prediction.conf # Access camera parameters (if available) if hasattr(prediction, 'extrinsics'): camera_poses = prediction.extrinsics # shape: (2, 4, 4) if hasattr(prediction, 'intrinsics'): camera_intrinsics = prediction.intrinsics # shape: (2, 3, 3) # Access intermediate features (if export_feat_layers was set) if hasattr(prediction, 'aux') and 'feat_layer_0' in prediction.aux: features = prediction.aux['feat_layer_0'] ``` --- ## File: docs/BENCHMARK.md # ๐Ÿ“ Visual Geometry Benchmark This document provides comprehensive instructions for running benchmark evaluation on Depth Anything 3. ## โœจ Highlights - ๐Ÿ—‚๏ธ **Diverse and Challenging Datasets**: 5 datasets (ETH3D, 7Scenes, ScanNet++, HiRoom, DTU) covering from objects to indoor and outdoor scenes. Part of datasets are recalibrated for high accuracy (see [ScanNet++](#scannet) details). All preprocessed datasets are uploaded to [depth-anything/DA3-BENCH](https://huggingface.co/datasets/depth-anything/DA3-BENCH). - ๐Ÿ”ง **Robust Evaluation Pipeline**: Standardized pipeline featuring RANSAC-based pose alignment for better coordinate system alignment, TSDF fusion for directly reflecting depth 3D consistency. - ๐Ÿ“Š **Standardized Metrics**: Performance measured using established metrics: AUC for pose accuracy, F1-score and Chamfer Distance for reconstruction. --- ## ๐Ÿ“‘ Table of Contents - [๐Ÿš€ Quick Start](#quick-start) - [๐Ÿ“ฅ Dataset Download](#dataset-download) - [โš™๏ธ Evaluation Pipeline](#evaluation-pipeline) - [๐Ÿ”ง Configuration](#configuration) - [๐Ÿ“Š Metrics](#metrics) - [๐Ÿ—‚๏ธ Dataset Details](#dataset-details) - [๐Ÿ’ป Command Reference](#command-reference) - [๐Ÿ” Troubleshooting](#troubleshooting) --- ## ๐Ÿš€ Quick Start ### 1. Download Benchmark Data > ๐Ÿ’ก **Note:** Install HuggingFace CLI first: `pip install -U huggingface_hub[cli]` > > ๐ŸŒ **Mirror:** If download is slow, try: `export HF_ENDPOINT=https://hf-mirror.com` ```bash cd da3_release # Create directory and download from HuggingFace mkdir -p workspace/benchmark_dataset hf download depth-anything/DA3-BENCH \ --local-dir workspace/benchmark_dataset \ --repo-type dataset # Extract all datasets cd workspace/benchmark_dataset for f in *.zip; do unzip -q "$f"; done ``` ### 2. Run Evaluation ```bash # Set model (default: depth-anything/DA3-GIANT) MODEL=depth-anything/DA3-GIANT # Full evaluation (all datasets, all modes) python -m depth_anything_3.bench.evaluator model.path=$MODEL # View results python -m depth_anything_3.bench.evaluator eval.print_only=true ``` --- ## ๐Ÿ“ฅ Dataset Download All benchmark datasets are hosted on HuggingFace: **[depth-anything/DA3-BENCH](https://huggingface.co/datasets/depth-anything/DA3-BENCH)** | Dataset | File | Size | Description | |---------|------|------|-------------| | ETH3D | `eth3d.zip` | ~14.1 GB | High-resolution multi-view stereo (indoor/outdoor) | | ScanNet++ | `scannetpp.zip` | ~10.1 GB | High-quality RGB-D indoor scenes | | DTU-49 | `dtu.zip` | ~8.3 GB | Multi-view stereo benchmark (22 scenes ร— 49 views) | | 7Scenes | `7scenes.zip` | ~3.3 GB | RGB-D indoor localization | | DTU-64 | `dtu64.zip` | ~1.7 GB | DTU subset for pose evaluation (13 scenes ร— 64 views) | | HiRoom | `hiroom.zip` | ~0.7 GB | High-resolution indoor rooms | ### Download Options **Option 1: Download All (Recommended)** ```bash hf download depth-anything/DA3-BENCH \ --local-dir workspace/benchmark_dataset \ --repo-type dataset ``` **Option 2: Download Specific Dataset** ```bash # Download only HiRoom hf download depth-anything/DA3-BENCH hiroom.zip \ --local-dir workspace/benchmark_dataset \ --repo-type dataset ``` **Option 3: Manual Download** Visit [https://huggingface.co/datasets/depth-anything/DA3-BENCH](https://huggingface.co/datasets/depth-anything/DA3-BENCH) and download the zip files manually. ### Extract Datasets ```bash cd workspace/benchmark_dataset # Extract all for f in *.zip; do unzip -q "$f"; done # Or extract specific dataset unzip hiroom.zip ``` ### Expected Directory Structure After extraction, your directory should look like: ``` workspace/benchmark_dataset/ โ”œโ”€โ”€ eth3d/ โ”‚ โ”œโ”€โ”€ courtyard/ โ”‚ โ”œโ”€โ”€ electro/ โ”‚ โ””โ”€โ”€ ... โ”œโ”€โ”€ 7scenes/ โ”‚ โ””โ”€โ”€ 7Scenes/ โ”‚ โ”œโ”€โ”€ chess/ โ”‚ โ””โ”€โ”€ ... โ”œโ”€โ”€ scannetpp/ โ”‚ โ”œโ”€โ”€ 09c1414f1b/ โ”‚ โ””โ”€โ”€ ... โ”œโ”€โ”€ hiroom/ โ”‚ โ”œโ”€โ”€ data/ โ”‚ โ”œโ”€โ”€ fused_pcd/ โ”‚ โ””โ”€โ”€ selected_scene_list_val.txt โ”œโ”€โ”€ dtu/ โ”‚ โ”œโ”€โ”€ Rectified/ โ”‚ โ”œโ”€โ”€ Cameras/ โ”‚ โ”œโ”€โ”€ Points/ โ”‚ โ”œโ”€โ”€ SampleSet/ โ”‚ โ””โ”€โ”€ depth_raw/ โ””โ”€โ”€ dtu64/ โ”œโ”€โ”€ Cameras/ โ”œโ”€โ”€ scan105/ โ””โ”€โ”€ ... ``` --- ## โš™๏ธ Evaluation Pipeline ### Evaluation Modes | Mode | Description | Metrics | |------|-------------|---------| | `pose` | Camera pose estimation | AUC@3ยฐ, AUC@30ยฐ | | `recon_unposed` | 3D reconstruction with **predicted** poses | F-score, Overall | | `recon_posed` | 3D reconstruction with **GT** poses | F-score, Overall | ### Basic Usage ```bash cd da3_release MODEL=depth-anything/DA3-GIANT # Full evaluation (inference + evaluation + print results) python -m depth_anything_3.bench.evaluator model.path=$MODEL # Skip inference, only evaluate existing predictions python -m depth_anything_3.bench.evaluator eval.eval_only=true # Only print saved metrics python -m depth_anything_3.bench.evaluator eval.print_only=true ``` ### Selective Evaluation ```bash # Evaluate specific datasets python -m depth_anything_3.bench.evaluator model.path=$MODEL eval.datasets=[hiroom] # Evaluate specific modes python -m depth_anything_3.bench.evaluator model.path=$MODEL eval.modes=[pose,recon_unposed] # Combine dataset and mode selection python -m depth_anything_3.bench.evaluator model.path=$MODEL \ eval.datasets=[hiroom] \ eval.modes=[pose] ``` ### ๐Ÿ–ฅ๏ธ Multi-GPU Inference The evaluator automatically distributes inference across available GPUs: ```bash # Use 4 GPUs CUDA_VISIBLE_DEVICES=0,1,2,3 python -m depth_anything_3.bench.evaluator model.path=$MODEL # Use all available GPUs (default) python -m depth_anything_3.bench.evaluator model.path=$MODEL # Single GPU CUDA_VISIBLE_DEVICES=0 python -m depth_anything_3.bench.evaluator model.path=$MODEL ``` --- ## ๐Ÿ”ง Configuration ### Config File Default config: `src/depth_anything_3/bench/configs/eval_bench.yaml` ```yaml # Model path model: path: depth-anything/DA3-GIANT # Workspace directory workspace: work_dir: ./workspace/evaluation # Evaluation settings eval: datasets: [eth3d, 7scenes, scannetpp, hiroom, dtu, dtu64] modes: [pose, recon_unposed, recon_posed] max_frames: 100 # Max frames per scene (-1 = no limit) scenes: null # Specific scenes (null = all) # Inference settings inference: num_fusion_workers: 4 debug: false ``` ### Output Structure ``` workspace/evaluation/ โ”œโ”€โ”€ model_results/ # Inference outputs โ”‚ โ”œโ”€โ”€ eth3d/ โ”‚ โ”‚ โ””โ”€โ”€ {scene}/ โ”‚ โ”‚ โ”œโ”€โ”€ unposed/ # Predictions for recon_unposed โ”‚ โ”‚ โ””โ”€โ”€ posed/ # Predictions for recon_posed โ”‚ โ”œโ”€โ”€ 7scenes/ โ”‚ โ”œโ”€โ”€ scannetpp/ โ”‚ โ”œโ”€โ”€ hiroom/ โ”‚ โ”œโ”€โ”€ dtu/ โ”‚ โ””โ”€โ”€ dtu64/ โ””โ”€โ”€ metric_results/ # Evaluation metrics (JSON) โ”œโ”€โ”€ eth3d_pose.json โ”œโ”€โ”€ eth3d_recon_unposed.json โ”œโ”€โ”€ eth3d_recon_posed.json โ””โ”€โ”€ ... ``` --- ## ๐Ÿ“Š Metrics ### ๐ŸŽฏ Pose Estimation | Metric | Description | |--------|-------------| | **Auc3** | Area Under Curve at 3ยฐ angular error threshold | | **Auc30** | Area Under Curve at 30ยฐ angular error threshold | ### ๐Ÿ—๏ธ 3D Reconstruction | Metric | Description | Note | |--------|-------------|------| | **F-score** | Harmonic mean of Precision and Recall | Higher is better | | **Overall** | (Accuracy + Completeness) / 2 | Lower is better (error in meters/mm) | > **Note:** DTU reports Overall in millimeters; other datasets report in meters. ### Expected Results for DA3-GIANT If your setup is correct, you should get the following results when evaluating the **DA3-GIANT** model: ``` ======================================================== ๐Ÿ“Š SUMMARY ======================================================== ๐ŸŽฏ POSE ESTIMATION --------------------------------------------------------------------------------------- Metric Avg HiRoom ETH3D DTU-64 7Scenes ScanNet++ --------------------------------------------------------------------------------------- Auc3 0.6705 0.8030 0.4872 0.9408 0.2744 0.8470 Auc30 0.9436 0.9592 0.9153 0.9939 0.8668 0.9827 ๐Ÿ—๏ธ RECON_UNPOSED (Pred Pose) --------------------------------------------------------------------------------------- Metric Avg* HiRoom ETH3D DTU 7Scenes ScanNet++ --------------------------------------------------------------------------------------- F-score 0.7345 0.8629 0.7876 N/A 0.5043 0.7831 Overall 0.1682 0.0457 0.4366 1.7927 0.1230 0.0676 ๐Ÿ—๏ธ RECON_POSED (GT Pose) --------------------------------------------------------------------------------------- Metric Avg* HiRoom ETH3D DTU 7Scenes ScanNet++ --------------------------------------------------------------------------------------- F-score 0.7978 0.9546 0.8685 N/A 0.5635 0.8045 Overall 0.1408 0.0213 0.3679 1.7488 0.1092 0.0649 * Avg F-score / Overall = average over HiRoom, ETH3D, 7Scenes, ScanNet++ (4 datasets) ``` --- ## ๐Ÿ—‚๏ธ Dataset Details ### ETH3D High-resolution multi-view stereo benchmark with laser-scanned ground truth. - **Scenes:** 11 (courtyard, electro, kicker, pipes, relief, delivery_area, facade, office, playground, relief_2, terrains) - **Resolution:** Variable (high-res DSLR images) - **GT:** Laser-scanned meshes + depth maps > **โš ๏ธ Image Filtering:** Some images with unusual camera rotations are filtered out for stable evaluation. See `ETH3D_FILTER_KEYS` in `constants.py`. ### 7Scenes RGB-D dataset for camera relocalization. - **Scenes:** 7 (chess, fire, heads, office, pumpkin, redkitchen, stairs) - **Resolution:** 640ร—480 - **GT:** Poses from KinectFusion, meshes from TSDF fusion ### ScanNet++ High-quality indoor RGB-D dataset with dense annotations. - **Scenes:** 20 validation scenes - **Resolution:** 768ร—1024 (after undistortion) - **GT:** High-quality meshes from FARO scanner > **โš ๏ธ Camera Pose Re-calibration:** The default ScanNet++ poses are often inaccurate due to motion blur and textureless frames from iPhone captures. We re-ran COLMAP with the following improvements: > - **Frame filtering:** Removed blurry images during frame extraction > - **Fisheye calibration:** Jointly calibrated fisheye camera for wider FOV and better accuracy > - **Exhaustive matching:** Used COLMAP's exhaustive matcher and mapper for reliable poses (takes several days per scene but necessary for quality) > - All processed scenes are available at [haotongl/scannetpp_zipnerf](https://huggingface.co/datasets/haotongl/scannetpp_zipnerf) ### HiRoom Indoor room scenes with high-resolution RGB-D data. - **Scenes:** 24 validation scenes - **GT:** Fused point clouds ### DTU-49 (Reconstruction Only) Multi-view stereo benchmark following MVSNet evaluation protocol. - **Scenes:** 22 evaluation scenes - **Views:** 49 images per scene - **GT:** Laser-scanned point clouds with observation masks - **Metrics:** Overall only (accuracy + completeness in mm) ### DTU-64 (Pose Only) DTU subset for pose estimation evaluation. - **Scenes:** 13 scenes - **Views:** 64 images per scene - **Metrics:** AUC@3ยฐ, AUC@30ยฐ > **Why two DTU settings?** > - **DTU-64** (pose): More views = more challenging pose estimation > - **DTU-49** (recon): Standard MVSNet protocol for fair comparison with MVS methods --- ## ๐Ÿ’ป Command Reference ``` python -m depth_anything_3.bench.evaluator [OPTIONS] [KEY=VALUE ...] Configuration: --config PATH Config YAML file (default: bench/configs/eval_bench.yaml) Config Overrides (using dotlist notation): model.path=VALUE Model path or HuggingFace ID workspace.work_dir=VALUE Working directory for outputs eval.datasets=[dataset1,dataset2] Datasets to evaluate (eth3d,7scenes,scannetpp,hiroom,dtu,dtu64) eval.modes=[mode1,mode2] Evaluation modes (pose,recon_unposed,recon_posed) eval.scenes=[scene1,scene2] Specific scenes to evaluate (null=all) eval.max_frames=VALUE Max frames per scene (-1=no limit, default: 100) eval.ref_view_strategy=VALUE Reference view strategy (default: first) eval.eval_only=VALUE Only run evaluation (skip inference) (true/false) eval.print_only=VALUE Only print saved metrics (true/false) inference.num_fusion_workers=VALUE Number of parallel workers (default: 4) inference.debug=VALUE Enable debug mode (true/false) Special Flags: --help, -h Show this help message Multi-GPU: Use CUDA_VISIBLE_DEVICES to specify GPUs (auto-detected and distributed) ``` ### Examples ```bash MODEL=depth-anything/DA3-GIANT # Full evaluation python -m depth_anything_3.bench.evaluator model.path=$MODEL # Quick test on HiRoom only python -m depth_anything_3.bench.evaluator \ model.path=$MODEL \ eval.datasets=[hiroom] \ eval.modes=[pose] # Pose-only evaluation (all 5 pose datasets) python -m depth_anything_3.bench.evaluator \ model.path=$MODEL \ eval.datasets=[eth3d,7scenes,scannetpp,hiroom,dtu64] \ eval.modes=[pose] # Recon-only evaluation (all 5 recon datasets) python -m depth_anything_3.bench.evaluator \ model.path=$MODEL \ eval.datasets=[eth3d,7scenes,scannetpp,hiroom,dtu] \ eval.modes=[recon_unposed,recon_posed] # Debug specific scenes python -m depth_anything_3.bench.evaluator \ model.path=$MODEL \ eval.datasets=[eth3d] \ eval.scenes=[courtyard] \ inference.debug=true # Re-evaluate without re-running inference python -m depth_anything_3.bench.evaluator eval.eval_only=true # Just view results python -m depth_anything_3.bench.evaluator eval.print_only=true ``` --- ## ๐Ÿ” Troubleshooting ### Data Path Issues Ensure dataset paths in `src/depth_anything_3/utils/constants.py` are correct: ```python # Default paths (relative to project root) ETH3D_EVAL_DATA_ROOT = "workspace/benchmark_dataset/eth3d" SEVENSCENES_EVAL_DATA_ROOT = "workspace/benchmark_dataset/7scenes" SCANNETPP_EVAL_DATA_ROOT = "workspace/benchmark_dataset/scannetpp" HIROOM_EVAL_DATA_ROOT = "workspace/benchmark_dataset/hiroom/data" DTU_EVAL_DATA_ROOT = "workspace/benchmark_dataset/dtu" DTU64_EVAL_DATA_ROOT = "workspace/benchmark_dataset/dtu64" ``` --- ## ๐Ÿ“ Citation If you find this benchmark useful, please cite: ``` @article{depthanything3, title={Depth Anything 3: Recovering the visual space from any views}, author={Haotong Lin and Sili Chen and Jun Hao Liew and Donny Y. Chen and Zhenyu Li and Guang Shi and Jiashi Feng and Bingyi Kang}, journal={arXiv preprint arXiv:2511.10647}, year={2025} } ``` Please also cite the original dataset papers for each benchmark you use. --- ## ๐Ÿ“„ License The benchmark datasets are provided for research purposes only. Users must follow the original licenses of each dataset: - **ETH3D:** [https://www.eth3d.net/](https://www.eth3d.net/) - **7Scenes:** [Microsoft Research](https://www.microsoft.com/en-us/research/project/rgb-d-dataset-7-scenes/) - **ScanNet++:** [http://www.scan-net.org/](http://www.scan-net.org/) - **DTU:** [https://roboimagedata.compute.dtu.dk/](https://roboimagedata.compute.dtu.dk/) - **HiRoom:** [SVLightVerse](https://jerrypiglet.github.io/SVLightVerse/) --- ## File: docs/CLI.md # ๐Ÿš€ Depth Anything 3 Command Line Interface ## ๐Ÿ“‹ Table of Contents - [๐Ÿ“– Overview](#overview) - [โšก Quick Start](#quick-start) - [๐Ÿ“š Command Reference](#command-reference) - [๐Ÿค– auto - Auto Mode](#auto---auto-mode) - [๐Ÿ–ผ๏ธ image - Single Image Processing](#image---single-image-processing) - [๐Ÿ—‚๏ธ images - Image Directory Processing](#images---image-directory-processing) - [๐ŸŽฌ video - Video Processing](#video---video-processing) - [๐Ÿ“ colmap - COLMAP Dataset Processing](#colmap---colmap-dataset-processing) - [๐Ÿ”ง backend - Backend Service](#backend---backend-service) - [๐ŸŽจ gradio - Gradio Application](#gradio---gradio-application) - [๐Ÿ–ผ๏ธ gallery - Gallery Server](#gallery---gallery-server) - [โš™๏ธ Parameter Details](#parameter-details) - [๐Ÿ’ก Usage Examples](#usage-examples) ## ๐Ÿ“– Overview The Depth Anything 3 CLI provides a comprehensive command-line toolkit supporting image depth estimation, video processing, COLMAP dataset handling, and web applications. The backend service enables cache model to GPU so that we do not need to reload model for each command. ## โšก Quick Start The CLI can run fully offline or connect to the backend for cached weights and task scheduling: ```bash # ๐Ÿ”ง Start backend service (optional, keeps model resident in GPU memory) da3 backend --model-dir depth-anything/DA3NESTED-GIANT-LARGE # ๐Ÿš€ Use auto mode to process input (local inference, no backend involved) da3 auto path/to/input --export-dir ./workspace/scene001 # โ™ป๏ธ Reuse backend for next job da3 auto path/to/video.mp4 \ --export-dir workspace/gallery/scene002 \ --use-backend \ --backend-url http://localhost:8008 ``` Each export directory contains `scene.glb`, `scene.jpg`, and optional extras such as `depth_vis/` or `gs_video/` depending on the requested format. > **Note on `--export-dir` with `--use-backend`:** when a job is submitted to a running `da3 backend` over HTTP, `--export-dir` should be a relative path under that backend's `--gallery-dir` (`workspace/gallery` by default). This doesn't apply to local (non-backend) runs, where `--export-dir` can be anywhere you have write access. The `--use-backend` snippets elsewhere in this doc use short placeholders like `./output` purely to illustrate flag syntax โ€” when actually running with `--use-backend`, use a path under the backend's gallery directory instead (e.g. `workspace/gallery/output`). ## ๐Ÿ“š Command Reference ### ๐Ÿค– auto - Auto Mode Automatically detect input type and dispatch to the appropriate handler. **Usage:** ```bash da3 auto INPUT_PATH [OPTIONS] ``` **Input Type Detection:** - ๐Ÿ–ผ๏ธ Single image file (.jpg, .png, .jpeg, .webp, .bmp, .tiff, .tif) - ๐Ÿ“ Image directory - ๐ŸŽฌ Video file (.mp4, .avi, .mov, .mkv, .flv, .wmv, .webm, .m4v) - ๐Ÿ“ COLMAP directory (containing `images/` and `sparse/` subdirectories) **Parameters:** | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `INPUT_PATH` | str | Required | Input path (image, directory, video, or COLMAP) | | `--model-dir` | str | Default model | Model directory path | | `--export-dir` | str | `debug` | Export directory | | `--export-format` | str | `glb` | Export format (supports `mini_npz`, `glb`, `feat_vis`, etc., can be combined with hyphens) | | `--device` | str | `cuda` | Device to use | | `--use-backend` | bool | `False` | Use backend service for inference | | `--backend-url` | str | `http://localhost:8008` | Backend service URL | | `--process-res` | int | `504` | Processing resolution | | `--process-res-method` | str | `upper_bound_resize` | Processing resolution method | | `--export-feat` | str | `""` | Export features from specified layers, comma-separated (e.g., `"0,1,2"`) | | `--auto-cleanup` | bool | `False` | Automatically clean export directory without confirmation | | `--fps` | float | `1.0` | [Video] Frame sampling FPS | | `--sparse-subdir` | str | `""` | [COLMAP] Sparse reconstruction subdirectory (e.g., `"0"` for `sparse/0/`) | | `--align-to-input-ext-scale` | bool | `True` | [COLMAP] Align prediction to input extrinsics scale | | `--use-ray-pose` | bool | `False` | Use ray-based pose estimation instead of camera decoder | | `--ref-view-strategy` | str | `saddle_balanced` | Reference view selection strategy: `first`, `middle`, `saddle_balanced`, `saddle_sim_range`. See [docs](funcs/ref_view_strategy.md) | | `--conf-thresh-percentile` | float | `40.0` | [GLB] Lower percentile for adaptive confidence threshold | | `--num-max-points` | int | `1000000` | [GLB] Maximum number of points in the point cloud | | `--show-cameras` | bool | `True` | [GLB] Show camera wireframes in the exported scene | | `--feat-vis-fps` | int | `15` | [FEAT_VIS] Frame rate for output video | **Examples:** ```bash # ๐Ÿ–ผ๏ธ Auto-process an image da3 auto path/to/image.jpg --export-dir ./output # ๐ŸŽฌ Auto-process a video da3 auto path/to/video.mp4 --fps 2.0 --export-dir ./output # ๐Ÿ”ง Use backend service da3 auto path/to/input \ --export-format mini_npz-glb \ --use-backend \ --backend-url http://localhost:8008 \ --export-dir ./output ``` --- ### ๐Ÿ–ผ๏ธ image - Single Image Processing Process a single image for camera pose and depth estimation. **Usage:** ```bash da3 image IMAGE_PATH [OPTIONS] ``` **Parameters:** | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `IMAGE_PATH` | str | Required | Input image file path | | `--model-dir` | str | Default model | Model directory path | | `--export-dir` | str | `debug` | Export directory | | `--export-format` | str | `glb` | Export format | | `--device` | str | `cuda` | Device to use | | `--use-backend` | bool | `False` | Use backend service for inference | | `--backend-url` | str | `http://localhost:8008` | Backend service URL | | `--process-res` | int | `504` | Processing resolution | | `--process-res-method` | str | `upper_bound_resize` | Processing resolution method | | `--export-feat` | str | `""` | Export feature layer indices (comma-separated) | | `--auto-cleanup` | bool | `False` | Automatically clean export directory | | `--use-ray-pose` | bool | `False` | Use ray-based pose estimation instead of camera decoder | | `--ref-view-strategy` | str | `saddle_balanced` | Reference view selection strategy. See [docs](funcs/ref_view_strategy.md) | | `--conf-thresh-percentile` | float | `40.0` | [GLB] Confidence threshold percentile | | `--num-max-points` | int | `1000000` | [GLB] Maximum number of points | | `--show-cameras` | bool | `True` | [GLB] Show cameras | | `--feat-vis-fps` | int | `15` | [FEAT_VIS] Video frame rate | **Examples:** ```bash # โœจ Basic usage da3 image path/to/image.png --export-dir ./output # โšก With backend acceleration da3 image path/to/image.png \ --use-backend \ --backend-url http://localhost:8008 \ --export-dir ./output # ๐Ÿ” Export feature visualization da3 image image.jpg \ --export-format feat_vis \ --export-feat "9,19,29,39" \ --export-dir ./results ``` --- ### ๐Ÿ—‚๏ธ images - Image Directory Processing Process a directory of images for batch depth estimation. **Usage:** ```bash da3 images IMAGES_DIR [OPTIONS] ``` **Parameters:** | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `IMAGES_DIR` | str | Required | Directory path containing images | | `--image-extensions` | str | `png,jpg,jpeg` | Image file extensions to process (comma-separated) | | `--model-dir` | str | Default model | Model directory path | | `--export-dir` | str | `debug` | Export directory | | `--export-format` | str | `glb` | Export format | | `--device` | str | `cuda` | Device to use | | `--use-backend` | bool | `False` | Use backend service for inference | | `--backend-url` | str | `http://localhost:8008` | Backend service URL | | `--process-res` | int | `504` | Processing resolution | | `--process-res-method` | str | `upper_bound_resize` | Processing resolution method | | `--export-feat` | str | `""` | Export feature layer indices | | `--auto-cleanup` | bool | `False` | Automatically clean export directory | | `--use-ray-pose` | bool | `False` | Use ray-based pose estimation instead of camera decoder | | `--ref-view-strategy` | str | `saddle_balanced` | Reference view selection strategy. See [docs](funcs/ref_view_strategy.md) | | `--conf-thresh-percentile` | float | `40.0` | [GLB] Confidence threshold percentile | | `--num-max-points` | int | `1000000` | [GLB] Maximum number of points | | `--show-cameras` | bool | `True` | [GLB] Show cameras | | `--feat-vis-fps` | int | `15` | [FEAT_VIS] Video frame rate | **Examples:** ```bash # ๐Ÿ“ Process directory (defaults to png/jpg/jpeg) da3 images ./image_folder --export-dir ./output # ๐ŸŽฏ Custom extensions da3 images ./dataset --image-extensions "png,jpg,webp" --export-dir ./output # ๐Ÿ”ง Use backend service da3 images ./dataset \ --use-backend \ --backend-url http://localhost:8008 \ --export-dir ./output ``` --- ### ๐ŸŽฌ video - Video Processing Process video by extracting frames for depth estimation. **Usage:** ```bash da3 video VIDEO_PATH [OPTIONS] ``` **Parameters:** | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `VIDEO_PATH` | str | Required | Input video file path | | `--fps` | float | `1.0` | Frame extraction sampling FPS | | `--model-dir` | str | Default model | Model directory path | | `--export-dir` | str | `debug` | Export directory | | `--export-format` | str | `glb` | Export format | | `--device` | str | `cuda` | Device to use | | `--use-backend` | bool | `False` | Use backend service for inference | | `--backend-url` | str | `http://localhost:8008` | Backend service URL | | `--process-res` | int | `504` | Processing resolution | | `--process-res-method` | str | `upper_bound_resize` | Processing resolution method | | `--export-feat` | str | `""` | Export feature layer indices | | `--auto-cleanup` | bool | `False` | Automatically clean export directory | | `--use-ray-pose` | bool | `False` | Use ray-based pose estimation instead of camera decoder | | `--ref-view-strategy` | str | `saddle_balanced` | Reference view selection strategy. See [docs](funcs/ref_view_strategy.md) | | `--conf-thresh-percentile` | float | `40.0` | [GLB] Confidence threshold percentile | | `--num-max-points` | int | `1000000` | [GLB] Maximum number of points | | `--show-cameras` | bool | `True` | [GLB] Show cameras | | `--feat-vis-fps` | int | `15` | [FEAT_VIS] Video frame rate | **Examples:** ```bash # โœจ Basic video processing da3 video path/to/video.mp4 --export-dir ./output # โš™๏ธ Control frame sampling and resolution da3 video path/to/video.mp4 \ --fps 2.0 \ --process-res 1024 \ --export-dir ./output # ๐Ÿ”ง Use backend service da3 video path/to/video.mp4 \ --use-backend \ --backend-url http://localhost:8008 \ --export-dir ./output ``` --- ### ๐Ÿ“ colmap - COLMAP Dataset Processing Run pose-conditioned depth estimation on COLMAP data. **Usage:** ```bash da3 colmap COLMAP_DIR [OPTIONS] ``` **Parameters:** | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `COLMAP_DIR` | str | Required | COLMAP directory containing `images/` and `sparse/` subdirectories | | `--sparse-subdir` | str | `""` | Sparse reconstruction subdirectory (e.g., `"0"` for `sparse/0/`) | | `--align-to-input-ext-scale` | bool | `True` | Align prediction to input extrinsics scale | | `--model-dir` | str | Default model | Model directory path | | `--export-dir` | str | `debug` | Export directory | | `--export-format` | str | `glb` | Export format | | `--device` | str | `cuda` | Device to use | | `--use-backend` | bool | `False` | Use backend service for inference | | `--backend-url` | str | `http://localhost:8008` | Backend service URL | | `--process-res` | int | `504` | Processing resolution | | `--process-res-method` | str | `upper_bound_resize` | Processing resolution method | | `--export-feat` | str | `""` | Export feature layer indices | | `--auto-cleanup` | bool | `False` | Automatically clean export directory | | `--use-ray-pose` | bool | `False` | Use ray-based pose estimation instead of camera decoder | | `--ref-view-strategy` | str | `saddle_balanced` | Reference view selection strategy. See [docs](funcs/ref_view_strategy.md) | | `--conf-thresh-percentile` | float | `40.0` | [GLB] Confidence threshold percentile | | `--num-max-points` | int | `1000000` | [GLB] Maximum number of points | | `--show-cameras` | bool | `True` | [GLB] Show cameras | | `--feat-vis-fps` | int | `15` | [FEAT_VIS] Video frame rate | **Examples:** ```bash # ๐Ÿ“ Process COLMAP dataset da3 colmap ./colmap_dataset --export-dir ./output # ๐ŸŽฏ Use specific sparse subdirectory and align scale da3 colmap ./colmap_dataset \ --sparse-subdir 0 \ --align-to-input-ext-scale \ --export-dir ./output # ๐Ÿ”ง Use backend service da3 colmap ./colmap_dataset \ --use-backend \ --backend-url http://localhost:8008 \ --export-dir ./output ``` --- ### ๐Ÿ”ง backend - Backend Service Start model backend service with integrated gallery. **Usage:** ```bash da3 backend [OPTIONS] ``` **Parameters:** | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `--model-dir` | str | Default model | Model directory path | | `--device` | str | `cuda` | Device to use | | `--host` | str | `127.0.0.1` | Host address to bind to | | `--port` | int | `8008` | Port number to bind to | | `--gallery-dir` | str | Default gallery dir | Gallery directory path (optional) | | `--api-key` | str | `None` | Require this key (as an `X-API-Key` header) on `/inference` requests. Falls back to the `DA3_BACKEND_API_KEY` env var, then to an auto-generated key (see below). | | `--allow-unauthenticated` | bool | `False` | Skip authentication entirely on a non-loopback `--host`, instead of using an auto-generated key. | **Features:** - ๐ŸŽฏ Keeps model resident in GPU memory - ๐Ÿ”Œ Provides REST inference API - ๐Ÿ“Š Integrated dashboard and status monitoring - ๐Ÿ–ผ๏ธ Optional gallery browser (if `--gallery-dir` is provided) - ๐Ÿ“ Exports for a given request are written within the configured `--gallery-dir` - ๐Ÿ”‘ Optional `X-API-Key` authentication for `/inference` (see below) **Authentication:** - By default the server binds to `127.0.0.1` (localhost-only) and requires no authentication, matching the examples below โ€” nothing changes if you're not exposing the backend beyond your own machine. - If you bind `--host` to anything other than localhost and don't set `--api-key`, the backend generates a random key for that run and prints it on startup, e.g.: ``` No --api-key was set, so one was generated for this run: X-API-Key: FdGcurw5z45kuGBjd5wO5S1agAf-iknYnfHhpWA3RmA ``` Copy that value into `DA3_BACKEND_API_KEY` in the environment of anything that submits jobs to it (see below) โ€” no need to come up with a key yourself. - For a key that stays the same across restarts (e.g. scripted/production setups), set it yourself before starting the backend: ```bash export DA3_BACKEND_API_KEY=$(openssl rand -hex 32) # or any secret string you prefer da3 backend --host 0.0.0.0 --model-dir depth-anything/DA3NESTED-GIANT-LARGE ``` `--api-key "$DA3_BACKEND_API_KEY"` works the same way if you'd rather pass it explicitly. - On the client side, `da3 ... --use-backend` automatically reads `DA3_BACKEND_API_KEY` from its own environment and sends it โ€” set the same env var wherever you run those commands from (same machine or not). - Pass `--allow-unauthenticated` if you'd rather skip authentication altogether (e.g. a network you already fully trust). **Available Endpoints:** - ๐Ÿ  `/` - Home page - ๐Ÿ“Š `/dashboard` - Dashboard - โœ… `/status` - API status - ๐Ÿ–ผ๏ธ `/gallery/` - Gallery browser (if enabled) **Examples:** ```bash # ๐Ÿš€ Basic backend service da3 backend --model-dir depth-anything/DA3NESTED-GIANT-LARGE # ๐Ÿ–ผ๏ธ Backend with gallery, reachable beyond localhost, with a fixed API key da3 backend \ --model-dir depth-anything/DA3NESTED-GIANT-LARGE \ --device cuda \ --host 0.0.0.0 \ --port 8008 \ --gallery-dir ./workspace \ --api-key "$DA3_BACKEND_API_KEY" # ๐Ÿ’ป Use CPU da3 backend --model-dir depth-anything/DA3NESTED-GIANT-LARGE --device cpu ``` --- ### ๐ŸŽจ gradio - Gradio Application Launch Depth Anything 3 Gradio interactive web application. **Usage:** ```bash da3 gradio [OPTIONS] ``` **Parameters:** | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `--model-dir` | str | Required | Model directory path | | `--workspace-dir` | str | Required | Workspace directory path | | `--gallery-dir` | str | Required | Gallery directory path | | `--host` | str | `127.0.0.1` | Host address to bind to | | `--port` | int | `7860` | Port number to bind to | | `--share` | bool | `False` | Create a public link | | `--debug` | bool | `False` | Enable debug mode | | `--cache-examples` | bool | `False` | Pre-cache all example scenes at startup | | `--cache-gs-tag` | str | `""` | Tag to match scene names for high-res+3DGS caching | **Examples:** ```bash # ๐ŸŽจ Basic Gradio application da3 gradio \ --model-dir depth-anything/DA3NESTED-GIANT-LARGE \ --workspace-dir ./workspace \ --gallery-dir ./gallery # ๐ŸŒ Enable sharing and debug da3 gradio \ --model-dir depth-anything/DA3NESTED-GIANT-LARGE \ --workspace-dir ./workspace \ --gallery-dir ./gallery \ --share \ --debug # โšก Pre-cache examples da3 gradio \ --model-dir depth-anything/DA3NESTED-GIANT-LARGE \ --workspace-dir ./workspace \ --gallery-dir ./gallery \ --cache-examples \ --cache-gs-tag "dl3dv" ``` --- ### ๐Ÿ–ผ๏ธ gallery - Gallery Server Launch standalone Depth Anything 3 Gallery server. **Usage:** ```bash da3 gallery [OPTIONS] ``` **Parameters:** | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `--gallery-dir` | str | Default gallery dir | Gallery root directory | | `--host` | str | `127.0.0.1` | Host address to bind to | | `--port` | int | `8007` | Port number to bind to | | `--open-browser` | bool | `False` | Open browser after launch | **Note:** The gallery expects each scene folder to contain at least `scene.glb` and `scene.jpg`, with optional subfolders such as `depth_vis/` or `gs_video/`. **Examples:** ```bash # ๐Ÿ–ผ๏ธ Basic gallery server da3 gallery --gallery-dir ./workspace # ๐ŸŒ Custom host and port da3 gallery \ --gallery-dir ./workspace \ --host 0.0.0.0 \ --port 8007 # ๐Ÿš€ Auto-open browser da3 gallery --gallery-dir ./workspace --open-browser ``` --- ## โš™๏ธ Parameter Details ### ๐Ÿ”ง Common Parameters - **`--export-dir`**: Output directory, defaults to `debug` - **`--export-format`**: Export format, supports combining multiple formats with hyphens: - ๐Ÿ“ฆ `mini_npz`: Compressed NumPy format - ๐ŸŽจ `glb`: glTF binary format (3D scene) - ๐Ÿ” `feat_vis`: Feature visualization - Example: `mini_npz-glb` exports both formats - **`--process-res`** / **`--process-res-method`**: Control preprocessing resolution strategy - `process-res`: Target resolution (default 504) - `process-res-method`: Resize method (default `upper_bound_resize`) - **`--auto-cleanup`**: Remove existing export directory without confirmation - **`--use-backend`** / **`--backend-url`**: Reuse running backend service - โšก Reduces model loading time - ๐ŸŒ Supports distributed processing - **`--export-feat`**: Layer indices for exporting intermediate features (comma-separated) - Example: `"9,19,29,39"` ### ๐ŸŽจ GLB Export Parameters - **`--conf-thresh-percentile`**: Lower percentile for adaptive confidence threshold (default 40.0) - Used to filter low-confidence points - **`--num-max-points`**: Maximum number of points in point cloud (default 1,000,000) - Controls output file size and performance - **`--show-cameras`**: Show camera wireframes in exported scene (default True) ### ๐Ÿ” Feature Visualization Parameters - **`--feat-vis-fps`**: Frame rate for feature visualization output video (default 15) ### ๐ŸŽฌ Video-Specific Parameters - **`--fps`**: Video frame extraction sampling rate (default 1.0 FPS) - Higher values extract more frames ### ๐Ÿ“ COLMAP-Specific Parameters - **`--sparse-subdir`**: Sparse reconstruction subdirectory - Empty string uses `sparse/` directory - `"0"` uses `sparse/0/` directory - **`--align-to-input-ext-scale`**: Align prediction to input extrinsics scale (default True) - Ensures depth estimation is consistent with COLMAP scale --- ## ๐Ÿ’ก Usage Examples ### 1๏ธโƒฃ Basic Workflow ```bash # ๐Ÿ”ง Start backend service da3 backend --model-dir depth-anything/DA3NESTED-GIANT-LARGE --host 0.0.0.0 --port 8008 # ๐Ÿ–ผ๏ธ Process single image da3 image image.jpg --export-dir ./output1 --use-backend # ๐ŸŽฌ Process video da3 video video.mp4 --fps 2.0 --export-dir ./output2 --use-backend # ๐Ÿ“ Process COLMAP dataset da3 colmap ./colmap_data --export-dir ./output3 --use-backend ``` ### 2๏ธโƒฃ Using Auto Mode ```bash # ๐Ÿค– Auto-detect and process da3 auto ./unknown_input --export-dir ./output # โšก With backend acceleration da3 auto ./unknown_input \ --use-backend \ --backend-url http://localhost:8008 \ --export-dir ./output ``` ### 3๏ธโƒฃ Multi-Format Export ```bash # ๐Ÿ“ฆ Export both NPZ and GLB formats da3 auto assets/examples/SOH \ --export-format mini_npz-glb \ --export-dir ./workspace/soh # ๐Ÿ” Export feature visualization da3 image image.jpg \ --export-format feat_vis \ --export-feat "9,19,29,39" \ --export-dir ./results ``` ### 4๏ธโƒฃ Advanced Configuration ```bash # โš™๏ธ Custom resolution and point cloud density da3 image image.jpg \ --process-res 1024 \ --num-max-points 2000000 \ --conf-thresh-percentile 30.0 \ --export-dir ./output # ๐Ÿ“ COLMAP advanced options da3 colmap ./colmap_data \ --sparse-subdir 0 \ --align-to-input-ext-scale \ --process-res 756 \ --export-dir ./output ``` ### 5๏ธโƒฃ Batch Processing Workflow ```bash # ๐Ÿ”ง Start backend da3 backend \ --model-dir depth-anything/DA3NESTED-GIANT-LARGE \ --device cuda \ --host 0.0.0.0 \ --port 8008 \ --gallery-dir ./workspace # ๐Ÿ”„ Batch process multiple scenes for scene in scene1 scene2 scene3; do da3 auto ./data/$scene \ --export-dir ./workspace/$scene \ --use-backend \ --auto-cleanup done # ๐Ÿ–ผ๏ธ Launch gallery to view results da3 gallery --gallery-dir ./workspace --open-browser ``` ### 6๏ธโƒฃ Web Applications ```bash # ๐ŸŽจ Launch Gradio application da3 gradio \ --model-dir depth-anything/DA3NESTED-GIANT-LARGE \ --workspace-dir workspace/gradio \ --gallery-dir ./gallery \ --host 0.0.0.0 \ --port 7860 \ --share ``` ### 7๏ธโƒฃ Transformer Feature Visualization ```bash # ๐Ÿ” Export Transformer features # ๐Ÿ“ฆ Combined with numerical output da3 auto video.mp4 \ --export-format glb-feat_vis \ --export-feat "11,21,31" \ --export-dir ./debug \ --use-backend ``` --- ## ๐Ÿ“ Notes 1. **๐Ÿ”ง Backend Service**: Recommended for processing multiple tasks to improve efficiency 2. **๐Ÿ’พ GPU Memory**: Be mindful of GPU memory usage when processing high-resolution inputs 3. **๐Ÿ“ Export Directory**: Use `--auto-cleanup` to avoid manual confirmation for deletion 4. **๐Ÿ”€ Format Combination**: Multiple export formats can be combined with hyphens (e.g., `mini_npz-glb-feat_vis`) 5. **๐Ÿ“ COLMAP Data**: Ensure COLMAP directory structure is correct (contains `images/` and `sparse/` subdirectories) --- ## โ“ Getting Help View detailed help for any command: ```bash # ๐Ÿ“– View main help da3 --help # ๐Ÿ” View specific command help da3 auto --help da3 image --help da3 backend --help ```