{"owner":"ByteDance-Seed","repo":"Depth-Anything-3","hasSkills":true,"totalSkillsCount":1,"totalTokensCount":4896,"categories":["project-spec"],"hasMcp":false,"mcpConfig":null,"found":["docs/API.md"],"skills":{"docs/API.md":"# 📚 DepthAnything3 API Documentation\n\n## 📑 Table of Contents\n\n1. [📖 Overview](#overview)\n2. [💡 Usage Examples](#usage-examples)\n3. [🔧 Core API](#core-api)\n   - [DepthAnything3 Class](#depthanything3-class)\n   - [inference() Method](#inference-method)\n4. [⚙️ Parameters](#parameters)\n   - [Input Parameters](#input-parameters)\n   - [Pose Alignment Parameters](#pose-alignment-parameters)\n   - [Feature Export Parameters](#feature-export-parameters)\n   - [Rendering Parameters](#rendering-parameters)\n   - [Processing Parameters](#processing-parameters)\n   - [Export Parameters](#export-parameters)\n5. [📤 Export Formats](#export-formats)\n6. [↩️ Return Value](#return-value)\n\n## 📖 Overview\n\nThis documentation provides comprehensive API reference for DepthAnything3, including usage examples, parameter specifications, export formats, and advanced features. It covers both basic pose and depth estimation workflows and advanced pose-conditioned processing with multiple export capabilities.\n\n## 💡 Usage Examples\n\nHere are quick examples to get you started:\n\n### 🚀 Basic Depth Estimation\n```python\nfrom depth_anything_3.api import DepthAnything3\n\n# Initialize and run inference\nmodel = DepthAnything3.from_pretrained(\"depth-anything/DA3NESTED-GIANT-LARGE\").to(\"cuda\")\nprediction = model.inference([\"image1.jpg\", \"image2.jpg\"])\n```\n\n### 📷 Pose-Conditioned Depth Estimation\n```python\nimport numpy as np\n\n# With camera parameters for better consistency\nprediction = model.inference(\n    image=[\"image1.jpg\", \"image2.jpg\"],\n    extrinsics=extrinsics_array,  # (N, 4, 4)\n    intrinsics=intrinsics_array   # (N, 3, 3)\n)\n```\n\n### 📤 Export Results\n```python\n# Export depth data and 3D visualization\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"mini_npz-glb\"\n)\n```\n\n### 🔍 Feature Extraction\n```python\n# Export intermediate features from specific layers\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"feat_vis\",\n    export_feat_layers=[0, 1, 2]  # Export features from layers 0, 1, 2\n)\n```\n\n### ✨ Advanced Export with Gaussian Splatting\n```python\n# Export multiple formats including Gaussian Splatting\n# Note: infer_gs=True requires da3-giant or da3nested-giant-large model\nmodel = DepthAnything3(model_name=\"da3-giant\").to(\"cuda\")\n\nprediction = model.inference(\n    image=image_paths,\n    extrinsics=extrinsics_array,\n    intrinsics=intrinsics_array,\n    export_dir=\"./output\",\n    export_format=\"npz-glb-gs_ply-gs_video\",\n    align_to_input_ext_scale=True,\n    infer_gs=True,  # Required for gs_ply and gs_video exports\n)\n```\n\n### 🎨 Advanced Export with Feature Visualization\n```python\n# Export with intermediate feature visualization\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"mini_npz-glb-depth_vis-feat_vis\",\n    export_feat_layers=[0, 5, 10, 15, 20],\n    feat_vis_fps=30,\n)\n```\n\n### 📐 Using Ray-Based Pose Estimation\n```python\n# Use ray-based pose estimation instead of camera decoder\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"glb\",\n    use_ray_pose=True,  # Enable ray-based pose estimation\n)\n```\n\n### 🎯 Reference View Selection\n```python\n# For multi-view inputs, automatically select the best reference view\nprediction = model.inference(\n    image=image_paths,\n    ref_view_strategy=\"saddle_balanced\",  # Default: balanced selection\n)\n\n# For video sequences, use middle frame as reference\nprediction = model.inference(\n    image=video_frames,\n    ref_view_strategy=\"middle\",  # Good for temporally ordered inputs\n)\n```\n\n## 🔧 Core API\n\n### 🔨 DepthAnything3 Class\n\nThe main API class that provides depth estimation capabilities with optional pose conditioning.\n\n#### 🎯 Initialization\n\n```python\nfrom depth_anything_3 import DepthAnything3\n\n# Initialize the model with a model name\nmodel = DepthAnything3(model_name=\"da3-large\")\nmodel = model.to(\"cuda\")  # Move to GPU\n```\n\n**Parameters:**\n- `model_name` (str, default: \"da3-large\"): The name of the model preset to use.\n  - **Available models:**\n    - 🦾 `\"da3-giant\"` - 1.15B params, any-view model with GS support\n    - ⭐ `\"da3-large\"` - 0.35B params, any-view model (recommended for most use cases)\n    - 📦 `\"da3-base\"` - 0.12B params, any-view model\n    - 🪶 `\"da3-small\"` - 0.08B params, any-view model\n    - 👁️ `\"da3mono-large\"` - 0.35B params, monocular depth only\n    - 📏 `\"da3metric-large\"` - 0.35B params, metric depth with sky segmentation\n    - 🎯 `\"da3nested-giant-large\"` - 1.40B params, nested model with all features\n\n### 🚀 inference() Method\n\nThe primary inference method that processes images and returns depth predictions.\n\n```python\nprediction = model.inference(\n    image=image_list,\n    extrinsics=extrinsics_array,      # Optional\n    intrinsics=intrinsics_array,      # Optional\n    align_to_input_ext_scale=True,   # Whether to align predicted poses to input scale\n    infer_gs=True,                   # Enable Gaussian branch for gs exports\n    use_ray_pose=False,              # Use ray-based pose estimation instead of camera decoder\n    ref_view_strategy=\"saddle_balanced\",  # Reference view selection strategy\n    render_exts=render_extrinsics,    # Optional renders for gs_video\n    render_ixts=render_intrinsics,    # Optional renders for gs_video\n    render_hw=(height, width),        # Optional renders for gs_video\n    process_res=504,\n    process_res_method=\"upper_bound_resize\",\n    export_dir=\"output_directory\",    # Optional\n    export_format=\"mini_npz\",\n    export_feat_layers=[],            # List of layer indices to export features from\n    conf_thresh_percentile=40.0,      # Confidence threshold percentile for depth map in GLB export\n    num_max_points=1_000_000,         # Maximum number of points to export in GLB export\n    show_cameras=True,                # Whether to show cameras in GLB export\n    feat_vis_fps=15,                  # Frames per second for feature visualization in feat_vis export\n    export_kwargs={}                  # Optional, additional arguments to export functions. export_format:key:val, see 'Parameters/Export Parameters' for details\n)\n```\n\n## ⚙️ Parameters\n\n### 📸 Input Parameters\n\n#### `image` (required)\n- **Type**: `List[Union[np.ndarray, Image.Image, str]]`\n- **Description**: List of input images. Can be numpy arrays, PIL Images, or file paths.\n- **Example**:\n  ```python\n  # From file paths\n  image = [\"image1.jpg\", \"image2.jpg\", \"image3.jpg\"]\n\n  # From numpy arrays\n  image = [np.array(img1), np.array(img2)]\n\n  # From PIL Images\n  image = [Image.open(\"image1.jpg\"), Image.open(\"image2.jpg\")]\n  ```\n\n#### `extrinsics` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(N, 4, 4)` where N is the number of input images\n- **Description**: Camera extrinsic matrices (world-to-camera transformation). When provided, enables pose-conditioned depth estimation mode.\n- **Note**: If not provided, the model operates in standard depth estimation mode.\n\n#### `intrinsics` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(N, 3, 3)` where N is the number of input images\n- **Description**: Camera intrinsic matrices containing focal length and principal point information. When provided, enables pose-conditioned depth estimation mode.\n\n### 🎯 Pose Alignment Parameters\n\n#### `align_to_input_ext_scale` (default: True)\n- **Type**: `bool`\n- **Description**: When True the predicted extrinsics are replaced with the input\n  ones and the depth maps are rescaled to match their metric scale. When False the\n  function returns the internally aligned poses computed via Umeyama alignment.\n\n#### `infer_gs` (default: False)\n- **Type**: `bool`\n- **Description**: Enable Gaussian Splatting branch for gaussian splatting exports. Required when using `gs_ply` or `gs_video` export formats.\n\n#### `use_ray_pose` (default: False)\n- **Type**: `bool`\n- **Description**: Use ray-based pose estimation instead of camera decoder for pose prediction. When True, the model uses ray prediction heads to estimate camera poses; when False, it uses the camera decoder approach.\n\n#### `ref_view_strategy` (default: \"saddle_balanced\")\n- **Type**: `str`\n- **Description**: Strategy for selecting the reference view from multiple input views. Options: `\"first\"`, `\"middle\"`, `\"saddle_balanced\"`, `\"saddle_sim_range\"`. Only applied when number of views ≥ 3. See [detailed documentation](funcs/ref_view_strategy.md) for strategy comparisons.\n- **Available strategies**:\n  - `\"saddle_balanced\"`: Selects view with balanced features across multiple metrics (recommended default)\n  - `\"saddle_sim_range\"`: Selects view with largest similarity range\n  - `\"first\"`: Always uses first view (not recommended, equivalent to no reordering for views < 3)\n  - `\"middle\"`: Uses middle view (recommended for video sequences)\n\n### 🔍 Feature Export Parameters\n\n#### `export_feat_layers` (default: [])\n- **Type**: `List[int]`\n- **Description**: List of layer indices to export intermediate features from. Features are stored in the `aux` dictionary of the Prediction object with keys like `feat_layer_0`, `feat_layer_1`, etc.\n\n### 🎥 Rendering Parameters\n\nThese arguments are only used when exporting Gaussian-splatting videos (include\n`\"gs_video\"` in `export_format`). They describe an auxiliary camera trajectory\nwith ``M`` views.\n\n#### `render_exts` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(M, 4, 4)`\n- **Description**: Camera extrinsics for the synthesized trajectory. If omitted,\n  the exporter falls back to the predicted poses.\n\n#### `render_ixts` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(M, 3, 3)`\n- **Description**: Camera intrinsics for each rendered frame. Leave `None` to\n  reuse the input intrinsics.\n\n#### `render_hw` (optional)\n- **Type**: `Optional[Tuple[int, int]]`\n- **Description**: Explicit output resolution `(height, width)` for the rendered\n  frames. Defaults to the input resolution when not provided.\n\n### ⚡ Processing Parameters\n\n#### `process_res` (default: 504)\n- **Type**: `int`\n- **Description**: Base resolution for processing. The model will resize images to this resolution for inference.\n\n#### `process_res_method` (default: \"upper_bound_resize\")\n- **Type**: `str`\n- **Description**: Method for resizing images to the target resolution.\n- **Options**:\n  - `\"upper_bound_resize\"`: Resize so that the specified dimension (504) becomes the longer side\n  - `\"lower_bound_resize\"`: Resize so that the specified dimension (504) becomes the shorter side\n- **Example**:\n  - Input: 1200×1600 → Output: 378×504 (with `process_res=504`, `process_res_method=\"upper_bound_resize\"`)\n  - Input: 504×672 → Output: 504×672 (no change needed)\n\n### 📦 Export Parameters\n\n#### `export_dir` (optional)\n- **Type**: `Optional[str]`\n- **Description**: Directory path where exported files will be saved. If not provided, no files will be exported.\n\n#### `export_format` (default: \"mini_npz\")\n- **Type**: `str`\n- **Description**: Format for exporting results. Supports multiple formats separated by `-`.\n- **Example**: `\"mini_npz-glb\"` exports both mini_npz and glb formats.\n\n#### 🌐 GLB Export Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"glb\"`.\n\n##### `conf_thresh_percentile` (default: 40.0)\n- **Type**: `float`\n- **Description**: Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out from the point cloud.\n\n##### `num_max_points` (default: 1,000,000)\n- **Type**: `int`\n- **Description**: Maximum number of points in the exported point cloud. If the point cloud exceeds this limit, it will be downsampled.\n\n##### `show_cameras` (default: True)\n- **Type**: `bool`\n- **Description**: Whether to include camera wireframes in the exported GLB file for visualization.\n\n#### 🎨 Feature Visualization Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"feat_vis\"`.\n\n##### `feat_vis_fps` (default: 15)\n- **Type**: `int`\n- **Description**: Frame rate for the output video when visualizing features across multiple images.\n\n#### ✨🎥 3DGS and 3DGS Video Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"gs_ply\"` or `\"gs_video\"`.\n\n##### `export_kwargs` (default: `{}`)\n- Type: `dict[str, dict[str, Any]]`\n- Description: Per-format extra arguments passed to export functions, mainly for `\"gs_ply\"` and `\"gs_video\"`.\n  - Access pattern: `export_kwargs[export_format][key] = value`\n  - Example:\n    ```python\n    {\n        \"gs_ply\": {\n            \"gs_views_interval\": 1,\n        },\n        \"gs_video\": {\n            \"trj_mode\": \"interpolate_smooth\",\n            \"chunk_size\": 1,\n            \"vis_depth\": None,\n        },\n    }\n    ```\n\n## 📤 Export Formats\n\nThe API supports multiple export formats for different use cases:\n\n### 📊 `mini_npz`\n- **Description**: Minimal NPZ format containing essential data\n- **Contents**: `depth`, `conf`, `exts`, `ixts`\n- **Use case**: Lightweight storage for depth data with camera parameters\n\n### 📦 `npz`\n- **Description**: Full NPZ format with comprehensive data\n- **Contents**: `depth`, `conf`, `exts`, `ixts`, `image`, etc.\n- **Use case**: Complete data export for advanced processing\n\n### 🌐 `glb`\n- **Description**: 3D visualization format with point cloud and camera poses\n- **Contents**:\n  - Point cloud with colors from original images\n  - Camera wireframes for visualization\n  - Confidence-based filtering and downsampling\n- **Use case**: 3D visualization, inspection, and analysis\n- **Features**:\n  - Automatic sky depth handling\n  - Confidence threshold filtering\n  - Background filtering (black/white)\n  - Scene scale normalization\n- **Parameters** (passed via `inference()` method directly):\n  - `conf_thresh_percentile` (float, default: 40.0): Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out.\n  - `num_max_points` (int, default: 1,000,000): Maximum number of points in the exported point cloud. If exceeded, points will be downsampled.\n  - `show_cameras` (bool, default: True): Whether to include camera wireframes in the exported GLB file for visualization.\n\n### ✨ `gs_ply`\n- **Description**: Gaussian Splatting point cloud format\n- **Contents**: 3DGS data in PLY format. Compatible with standard 3DGS viewers such as [SuperSplat](https://superspl.at/editor) (recommended), [SPARK](https://sparkjs.dev/viewer/).\n- **Use case**: Gaussian Splatting reconstruction\n- **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models.\n- **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)):\n  - `gs_views_interval`: Export to 3DGS every N views, default: `1`.\n\n### 🎥 `gs_video`\n- **Description**: Rasterized 3DGS to obtain videos\n- **Contents**: A video of 3DGS-rasterized views using either provided viewpoints or a predefined camera trajectory.\n- **Use case**: Video rendering for Gaussian Splatting\n- **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models.\n- **Note**: Can optionally use `render_exts`, `render_ixts`, and `render_hw` parameters in `inference()` method to specify novel viewpoints.\n- **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)):\n  - `extrinsics`: Optional world-to-camera poses for novel views. Falls back to the predicted poses of input views if not provided. (Alternatively, use `render_exts` parameter in `inference()`)\n  - `intrinsics`: Optional camera intrinsics for novel views. Falls back to the predicted intrinsics of input views if not provided. (Alternatively, use `render_ixts` parameter in `inference()`)\n  - `out_image_hw`: Optional output resolution `H x W`. Falls back to input resolution if not provided. (Alternatively, use `render_hw` parameter in `inference()`)\n  - `chunk_size`: Number of views rasterized per batch. Default: `8`.\n  - `trj_mode`: Predefined camera trajectory for novel-view rendering.\n  - `color_mode`: Same as `render_mode` in [gsplat](https://docs.gsplat.studio/main/apis/rasterization.html#gsplat.rasterization).\n  - `vis_depth`: How depth is combined with RGB. Default: `hcat` (horizontal concatenation).\n  - `enable_tqdm`: Whether to display a tqdm progress bar during rendering.\n  - `output_name`: File name of the rendered video.\n  - `video_quality`: Video quality to save. Default: `high`.\n    - `high`: High quality video (default)\n    - `medium`: Medium quality video (balance of storage space and quality)\n    - `low`: Low quality video (fewer storage space)\n\n### 🔍 `feat_vis`\n- **Description**: Feature visualization format\n- **Contents**: PCA-visualized intermediate features from specified layers\n- **Use case**: Model interpretability and feature analysis\n- **Note**: Requires `export_feat_layers` to be specified\n- **Parameters** (passed via `inference()` method directly):\n  - `feat_vis_fps` (int, default: 15): Frame rate for the output video when visualizing features across multiple images.\n\n### 🎨 `depth_vis`\n- **Description**: Depth visualization format\n- **Contents**: Color-coded depth maps alongside original images\n- **Use case**: Visual inspection of depth estimation quality\n\n### 🔗 Multiple Format Export\nYou can export multiple formats simultaneously by separating them with `-`:\n\n```python\n# Export both mini_npz and glb formats\nexport_format = \"mini_npz-glb\"\n\n# Export multiple formats\nexport_format = \"npz-glb-gs_ply\"\n```\n\n## ↩️ Return Value\n\nThe `inference()` method returns a `Prediction` object with the following attributes:\n\n### 📊 Core Outputs\n\n- **depth**: `np.ndarray` - Estimated depth maps with shape `(N, H, W)` where N is the number of images, H is height, and W is width.\n- **conf**: `np.ndarray` - Confidence maps with shape `(N, H, W)` indicating prediction reliability (optional, depends on model).\n\n### 📷 Camera Parameters\n\n- **extrinsics**: `np.ndarray` - Camera extrinsic matrices with shape `(N, 3, 4)` representing world-to-camera transformations. Only present if camera poses were estimated or provided as input.\n- **intrinsics**: `np.ndarray` - Camera intrinsic matrices with shape `(N, 3, 3)` containing focal length and principal point information. Only present if poses were estimated or provided as input.\n\n### 🎁 Additional Outputs\n\n- **processed_images**: `np.ndarray` - Preprocessed input images with shape `(N, H, W, 3)` in RGB format (0-255 uint8).\n- **aux**: `dict` - Auxiliary outputs including:\n  - `feat_layer_X`: Intermediate features from layer X (if `export_feat_layers` was specified)\n  - `gaussians`: 3D Gaussian Splats data (if `infer_gs=True`)\n\n### 💻 Usage Example\n\n```python\nprediction = model.inference(image=[\"img1.jpg\", \"img2.jpg\"])\n\n# Access depth maps\ndepth_maps = prediction.depth  # shape: (2, H, W)\n\n# Access confidence\nif hasattr(prediction, 'conf'):\n    confidence = prediction.conf\n\n# Access camera parameters (if available)\nif hasattr(prediction, 'extrinsics'):\n    camera_poses = prediction.extrinsics  # shape: (2, 4, 4)\n\nif hasattr(prediction, 'intrinsics'):\n    camera_intrinsics = prediction.intrinsics  # shape: (2, 3, 3)\n\n# Access intermediate features (if export_feat_layers was set)\nif hasattr(prediction, 'aux') and 'feat_layer_0' in prediction.aux:\n    features = prediction.aux['feat_layer_0']\n```\n"},"files":{"docs/API.md":"# 📚 DepthAnything3 API Documentation\n\n## 📑 Table of Contents\n\n1. [📖 Overview](#overview)\n2. [💡 Usage Examples](#usage-examples)\n3. [🔧 Core API](#core-api)\n   - [DepthAnything3 Class](#depthanything3-class)\n   - [inference() Method](#inference-method)\n4. [⚙️ Parameters](#parameters)\n   - [Input Parameters](#input-parameters)\n   - [Pose Alignment Parameters](#pose-alignment-parameters)\n   - [Feature Export Parameters](#feature-export-parameters)\n   - [Rendering Parameters](#rendering-parameters)\n   - [Processing Parameters](#processing-parameters)\n   - [Export Parameters](#export-parameters)\n5. [📤 Export Formats](#export-formats)\n6. [↩️ Return Value](#return-value)\n\n## 📖 Overview\n\nThis documentation provides comprehensive API reference for DepthAnything3, including usage examples, parameter specifications, export formats, and advanced features. It covers both basic pose and depth estimation workflows and advanced pose-conditioned processing with multiple export capabilities.\n\n## 💡 Usage Examples\n\nHere are quick examples to get you started:\n\n### 🚀 Basic Depth Estimation\n```python\nfrom depth_anything_3.api import DepthAnything3\n\n# Initialize and run inference\nmodel = DepthAnything3.from_pretrained(\"depth-anything/DA3NESTED-GIANT-LARGE\").to(\"cuda\")\nprediction = model.inference([\"image1.jpg\", \"image2.jpg\"])\n```\n\n### 📷 Pose-Conditioned Depth Estimation\n```python\nimport numpy as np\n\n# With camera parameters for better consistency\nprediction = model.inference(\n    image=[\"image1.jpg\", \"image2.jpg\"],\n    extrinsics=extrinsics_array,  # (N, 4, 4)\n    intrinsics=intrinsics_array   # (N, 3, 3)\n)\n```\n\n### 📤 Export Results\n```python\n# Export depth data and 3D visualization\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"mini_npz-glb\"\n)\n```\n\n### 🔍 Feature Extraction\n```python\n# Export intermediate features from specific layers\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"feat_vis\",\n    export_feat_layers=[0, 1, 2]  # Export features from layers 0, 1, 2\n)\n```\n\n### ✨ Advanced Export with Gaussian Splatting\n```python\n# Export multiple formats including Gaussian Splatting\n# Note: infer_gs=True requires da3-giant or da3nested-giant-large model\nmodel = DepthAnything3(model_name=\"da3-giant\").to(\"cuda\")\n\nprediction = model.inference(\n    image=image_paths,\n    extrinsics=extrinsics_array,\n    intrinsics=intrinsics_array,\n    export_dir=\"./output\",\n    export_format=\"npz-glb-gs_ply-gs_video\",\n    align_to_input_ext_scale=True,\n    infer_gs=True,  # Required for gs_ply and gs_video exports\n)\n```\n\n### 🎨 Advanced Export with Feature Visualization\n```python\n# Export with intermediate feature visualization\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"mini_npz-glb-depth_vis-feat_vis\",\n    export_feat_layers=[0, 5, 10, 15, 20],\n    feat_vis_fps=30,\n)\n```\n\n### 📐 Using Ray-Based Pose Estimation\n```python\n# Use ray-based pose estimation instead of camera decoder\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"glb\",\n    use_ray_pose=True,  # Enable ray-based pose estimation\n)\n```\n\n### 🎯 Reference View Selection\n```python\n# For multi-view inputs, automatically select the best reference view\nprediction = model.inference(\n    image=image_paths,\n    ref_view_strategy=\"saddle_balanced\",  # Default: balanced selection\n)\n\n# For video sequences, use middle frame as reference\nprediction = model.inference(\n    image=video_frames,\n    ref_view_strategy=\"middle\",  # Good for temporally ordered inputs\n)\n```\n\n## 🔧 Core API\n\n### 🔨 DepthAnything3 Class\n\nThe main API class that provides depth estimation capabilities with optional pose conditioning.\n\n#### 🎯 Initialization\n\n```python\nfrom depth_anything_3 import DepthAnything3\n\n# Initialize the model with a model name\nmodel = DepthAnything3(model_name=\"da3-large\")\nmodel = model.to(\"cuda\")  # Move to GPU\n```\n\n**Parameters:**\n- `model_name` (str, default: \"da3-large\"): The name of the model preset to use.\n  - **Available models:**\n    - 🦾 `\"da3-giant\"` - 1.15B params, any-view model with GS support\n    - ⭐ `\"da3-large\"` - 0.35B params, any-view model (recommended for most use cases)\n    - 📦 `\"da3-base\"` - 0.12B params, any-view model\n    - 🪶 `\"da3-small\"` - 0.08B params, any-view model\n    - 👁️ `\"da3mono-large\"` - 0.35B params, monocular depth only\n    - 📏 `\"da3metric-large\"` - 0.35B params, metric depth with sky segmentation\n    - 🎯 `\"da3nested-giant-large\"` - 1.40B params, nested model with all features\n\n### 🚀 inference() Method\n\nThe primary inference method that processes images and returns depth predictions.\n\n```python\nprediction = model.inference(\n    image=image_list,\n    extrinsics=extrinsics_array,      # Optional\n    intrinsics=intrinsics_array,      # Optional\n    align_to_input_ext_scale=True,   # Whether to align predicted poses to input scale\n    infer_gs=True,                   # Enable Gaussian branch for gs exports\n    use_ray_pose=False,              # Use ray-based pose estimation instead of camera decoder\n    ref_view_strategy=\"saddle_balanced\",  # Reference view selection strategy\n    render_exts=render_extrinsics,    # Optional renders for gs_video\n    render_ixts=render_intrinsics,    # Optional renders for gs_video\n    render_hw=(height, width),        # Optional renders for gs_video\n    process_res=504,\n    process_res_method=\"upper_bound_resize\",\n    export_dir=\"output_directory\",    # Optional\n    export_format=\"mini_npz\",\n    export_feat_layers=[],            # List of layer indices to export features from\n    conf_thresh_percentile=40.0,      # Confidence threshold percentile for depth map in GLB export\n    num_max_points=1_000_000,         # Maximum number of points to export in GLB export\n    show_cameras=True,                # Whether to show cameras in GLB export\n    feat_vis_fps=15,                  # Frames per second for feature visualization in feat_vis export\n    export_kwargs={}                  # Optional, additional arguments to export functions. export_format:key:val, see 'Parameters/Export Parameters' for details\n)\n```\n\n## ⚙️ Parameters\n\n### 📸 Input Parameters\n\n#### `image` (required)\n- **Type**: `List[Union[np.ndarray, Image.Image, str]]`\n- **Description**: List of input images. Can be numpy arrays, PIL Images, or file paths.\n- **Example**:\n  ```python\n  # From file paths\n  image = [\"image1.jpg\", \"image2.jpg\", \"image3.jpg\"]\n\n  # From numpy arrays\n  image = [np.array(img1), np.array(img2)]\n\n  # From PIL Images\n  image = [Image.open(\"image1.jpg\"), Image.open(\"image2.jpg\")]\n  ```\n\n#### `extrinsics` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(N, 4, 4)` where N is the number of input images\n- **Description**: Camera extrinsic matrices (world-to-camera transformation). When provided, enables pose-conditioned depth estimation mode.\n- **Note**: If not provided, the model operates in standard depth estimation mode.\n\n#### `intrinsics` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(N, 3, 3)` where N is the number of input images\n- **Description**: Camera intrinsic matrices containing focal length and principal point information. When provided, enables pose-conditioned depth estimation mode.\n\n### 🎯 Pose Alignment Parameters\n\n#### `align_to_input_ext_scale` (default: True)\n- **Type**: `bool`\n- **Description**: When True the predicted extrinsics are replaced with the input\n  ones and the depth maps are rescaled to match their metric scale. When False the\n  function returns the internally aligned poses computed via Umeyama alignment.\n\n#### `infer_gs` (default: False)\n- **Type**: `bool`\n- **Description**: Enable Gaussian Splatting branch for gaussian splatting exports. Required when using `gs_ply` or `gs_video` export formats.\n\n#### `use_ray_pose` (default: False)\n- **Type**: `bool`\n- **Description**: Use ray-based pose estimation instead of camera decoder for pose prediction. When True, the model uses ray prediction heads to estimate camera poses; when False, it uses the camera decoder approach.\n\n#### `ref_view_strategy` (default: \"saddle_balanced\")\n- **Type**: `str`\n- **Description**: Strategy for selecting the reference view from multiple input views. Options: `\"first\"`, `\"middle\"`, `\"saddle_balanced\"`, `\"saddle_sim_range\"`. Only applied when number of views ≥ 3. See [detailed documentation](funcs/ref_view_strategy.md) for strategy comparisons.\n- **Available strategies**:\n  - `\"saddle_balanced\"`: Selects view with balanced features across multiple metrics (recommended default)\n  - `\"saddle_sim_range\"`: Selects view with largest similarity range\n  - `\"first\"`: Always uses first view (not recommended, equivalent to no reordering for views < 3)\n  - `\"middle\"`: Uses middle view (recommended for video sequences)\n\n### 🔍 Feature Export Parameters\n\n#### `export_feat_layers` (default: [])\n- **Type**: `List[int]`\n- **Description**: List of layer indices to export intermediate features from. Features are stored in the `aux` dictionary of the Prediction object with keys like `feat_layer_0`, `feat_layer_1`, etc.\n\n### 🎥 Rendering Parameters\n\nThese arguments are only used when exporting Gaussian-splatting videos (include\n`\"gs_video\"` in `export_format`). They describe an auxiliary camera trajectory\nwith ``M`` views.\n\n#### `render_exts` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(M, 4, 4)`\n- **Description**: Camera extrinsics for the synthesized trajectory. If omitted,\n  the exporter falls back to the predicted poses.\n\n#### `render_ixts` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(M, 3, 3)`\n- **Description**: Camera intrinsics for each rendered frame. Leave `None` to\n  reuse the input intrinsics.\n\n#### `render_hw` (optional)\n- **Type**: `Optional[Tuple[int, int]]`\n- **Description**: Explicit output resolution `(height, width)` for the rendered\n  frames. Defaults to the input resolution when not provided.\n\n### ⚡ Processing Parameters\n\n#### `process_res` (default: 504)\n- **Type**: `int`\n- **Description**: Base resolution for processing. The model will resize images to this resolution for inference.\n\n#### `process_res_method` (default: \"upper_bound_resize\")\n- **Type**: `str`\n- **Description**: Method for resizing images to the target resolution.\n- **Options**:\n  - `\"upper_bound_resize\"`: Resize so that the specified dimension (504) becomes the longer side\n  - `\"lower_bound_resize\"`: Resize so that the specified dimension (504) becomes the shorter side\n- **Example**:\n  - Input: 1200×1600 → Output: 378×504 (with `process_res=504`, `process_res_method=\"upper_bound_resize\"`)\n  - Input: 504×672 → Output: 504×672 (no change needed)\n\n### 📦 Export Parameters\n\n#### `export_dir` (optional)\n- **Type**: `Optional[str]`\n- **Description**: Directory path where exported files will be saved. If not provided, no files will be exported.\n\n#### `export_format` (default: \"mini_npz\")\n- **Type**: `str`\n- **Description**: Format for exporting results. Supports multiple formats separated by `-`.\n- **Example**: `\"mini_npz-glb\"` exports both mini_npz and glb formats.\n\n#### 🌐 GLB Export Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"glb\"`.\n\n##### `conf_thresh_percentile` (default: 40.0)\n- **Type**: `float`\n- **Description**: Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out from the point cloud.\n\n##### `num_max_points` (default: 1,000,000)\n- **Type**: `int`\n- **Description**: Maximum number of points in the exported point cloud. If the point cloud exceeds this limit, it will be downsampled.\n\n##### `show_cameras` (default: True)\n- **Type**: `bool`\n- **Description**: Whether to include camera wireframes in the exported GLB file for visualization.\n\n#### 🎨 Feature Visualization Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"feat_vis\"`.\n\n##### `feat_vis_fps` (default: 15)\n- **Type**: `int`\n- **Description**: Frame rate for the output video when visualizing features across multiple images.\n\n#### ✨🎥 3DGS and 3DGS Video Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"gs_ply\"` or `\"gs_video\"`.\n\n##### `export_kwargs` (default: `{}`)\n- Type: `dict[str, dict[str, Any]]`\n- Description: Per-format extra arguments passed to export functions, mainly for `\"gs_ply\"` and `\"gs_video\"`.\n  - Access pattern: `export_kwargs[export_format][key] = value`\n  - Example:\n    ```python\n    {\n        \"gs_ply\": {\n            \"gs_views_interval\": 1,\n        },\n        \"gs_video\": {\n            \"trj_mode\": \"interpolate_smooth\",\n            \"chunk_size\": 1,\n            \"vis_depth\": None,\n        },\n    }\n    ```\n\n## 📤 Export Formats\n\nThe API supports multiple export formats for different use cases:\n\n### 📊 `mini_npz`\n- **Description**: Minimal NPZ format containing essential data\n- **Contents**: `depth`, `conf`, `exts`, `ixts`\n- **Use case**: Lightweight storage for depth data with camera parameters\n\n### 📦 `npz`\n- **Description**: Full NPZ format with comprehensive data\n- **Contents**: `depth`, `conf`, `exts`, `ixts`, `image`, etc.\n- **Use case**: Complete data export for advanced processing\n\n### 🌐 `glb`\n- **Description**: 3D visualization format with point cloud and camera poses\n- **Contents**:\n  - Point cloud with colors from original images\n  - Camera wireframes for visualization\n  - Confidence-based filtering and downsampling\n- **Use case**: 3D visualization, inspection, and analysis\n- **Features**:\n  - Automatic sky depth handling\n  - Confidence threshold filtering\n  - Background filtering (black/white)\n  - Scene scale normalization\n- **Parameters** (passed via `inference()` method directly):\n  - `conf_thresh_percentile` (float, default: 40.0): Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out.\n  - `num_max_points` (int, default: 1,000,000): Maximum number of points in the exported point cloud. If exceeded, points will be downsampled.\n  - `show_cameras` (bool, default: True): Whether to include camera wireframes in the exported GLB file for visualization.\n\n### ✨ `gs_ply`\n- **Description**: Gaussian Splatting point cloud format\n- **Contents**: 3DGS data in PLY format. Compatible with standard 3DGS viewers such as [SuperSplat](https://superspl.at/editor) (recommended), [SPARK](https://sparkjs.dev/viewer/).\n- **Use case**: Gaussian Splatting reconstruction\n- **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models.\n- **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)):\n  - `gs_views_interval`: Export to 3DGS every N views, default: `1`.\n\n### 🎥 `gs_video`\n- **Description**: Rasterized 3DGS to obtain videos\n- **Contents**: A video of 3DGS-rasterized views using either provided viewpoints or a predefined camera trajectory.\n- **Use case**: Video rendering for Gaussian Splatting\n- **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models.\n- **Note**: Can optionally use `render_exts`, `render_ixts`, and `render_hw` parameters in `inference()` method to specify novel viewpoints.\n- **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)):\n  - `extrinsics`: Optional world-to-camera poses for novel views. Falls back to the predicted poses of input views if not provided. (Alternatively, use `render_exts` parameter in `inference()`)\n  - `intrinsics`: Optional camera intrinsics for novel views. Falls back to the predicted intrinsics of input views if not provided. (Alternatively, use `render_ixts` parameter in `inference()`)\n  - `out_image_hw`: Optional output resolution `H x W`. Falls back to input resolution if not provided. (Alternatively, use `render_hw` parameter in `inference()`)\n  - `chunk_size`: Number of views rasterized per batch. Default: `8`.\n  - `trj_mode`: Predefined camera trajectory for novel-view rendering.\n  - `color_mode`: Same as `render_mode` in [gsplat](https://docs.gsplat.studio/main/apis/rasterization.html#gsplat.rasterization).\n  - `vis_depth`: How depth is combined with RGB. Default: `hcat` (horizontal concatenation).\n  - `enable_tqdm`: Whether to display a tqdm progress bar during rendering.\n  - `output_name`: File name of the rendered video.\n  - `video_quality`: Video quality to save. Default: `high`.\n    - `high`: High quality video (default)\n    - `medium`: Medium quality video (balance of storage space and quality)\n    - `low`: Low quality video (fewer storage space)\n\n### 🔍 `feat_vis`\n- **Description**: Feature visualization format\n- **Contents**: PCA-visualized intermediate features from specified layers\n- **Use case**: Model interpretability and feature analysis\n- **Note**: Requires `export_feat_layers` to be specified\n- **Parameters** (passed via `inference()` method directly):\n  - `feat_vis_fps` (int, default: 15): Frame rate for the output video when visualizing features across multiple images.\n\n### 🎨 `depth_vis`\n- **Description**: Depth visualization format\n- **Contents**: Color-coded depth maps alongside original images\n- **Use case**: Visual inspection of depth estimation quality\n\n### 🔗 Multiple Format Export\nYou can export multiple formats simultaneously by separating them with `-`:\n\n```python\n# Export both mini_npz and glb formats\nexport_format = \"mini_npz-glb\"\n\n# Export multiple formats\nexport_format = \"npz-glb-gs_ply\"\n```\n\n## ↩️ Return Value\n\nThe `inference()` method returns a `Prediction` object with the following attributes:\n\n### 📊 Core Outputs\n\n- **depth**: `np.ndarray` - Estimated depth maps with shape `(N, H, W)` where N is the number of images, H is height, and W is width.\n- **conf**: `np.ndarray` - Confidence maps with shape `(N, H, W)` indicating prediction reliability (optional, depends on model).\n\n### 📷 Camera Parameters\n\n- **extrinsics**: `np.ndarray` - Camera extrinsic matrices with shape `(N, 3, 4)` representing world-to-camera transformations. Only present if camera poses were estimated or provided as input.\n- **intrinsics**: `np.ndarray` - Camera intrinsic matrices with shape `(N, 3, 3)` containing focal length and principal point information. Only present if poses were estimated or provided as input.\n\n### 🎁 Additional Outputs\n\n- **processed_images**: `np.ndarray` - Preprocessed input images with shape `(N, H, W, 3)` in RGB format (0-255 uint8).\n- **aux**: `dict` - Auxiliary outputs including:\n  - `feat_layer_X`: Intermediate features from layer X (if `export_feat_layers` was specified)\n  - `gaussians`: 3D Gaussian Splats data (if `infer_gs=True`)\n\n### 💻 Usage Example\n\n```python\nprediction = model.inference(image=[\"img1.jpg\", \"img2.jpg\"])\n\n# Access depth maps\ndepth_maps = prediction.depth  # shape: (2, H, W)\n\n# Access confidence\nif hasattr(prediction, 'conf'):\n    confidence = prediction.conf\n\n# Access camera parameters (if available)\nif hasattr(prediction, 'extrinsics'):\n    camera_poses = prediction.extrinsics  # shape: (2, 4, 4)\n\nif hasattr(prediction, 'intrinsics'):\n    camera_intrinsics = prediction.intrinsics  # shape: (2, 3, 3)\n\n# Access intermediate features (if export_feat_layers was set)\nif hasattr(prediction, 'aux') and 'feat_layer_0' in prediction.aux:\n    features = prediction.aux['feat_layer_0']\n```\n"},"items":[{"name":"API.md","path":"docs/API.md","rawUrl":"https://raw.githubusercontent.com/ByteDance-Seed/Depth-Anything-3/HEAD/docs/API.md","title":"docs - API Interface Contract & Specifications","category":"project-spec","format":"markdown","content":"# 📚 DepthAnything3 API Documentation\n\n## 📑 Table of Contents\n\n1. [📖 Overview](#overview)\n2. [💡 Usage Examples](#usage-examples)\n3. [🔧 Core API](#core-api)\n   - [DepthAnything3 Class](#depthanything3-class)\n   - [inference() Method](#inference-method)\n4. [⚙️ Parameters](#parameters)\n   - [Input Parameters](#input-parameters)\n   - [Pose Alignment Parameters](#pose-alignment-parameters)\n   - [Feature Export Parameters](#feature-export-parameters)\n   - [Rendering Parameters](#rendering-parameters)\n   - [Processing Parameters](#processing-parameters)\n   - [Export Parameters](#export-parameters)\n5. [📤 Export Formats](#export-formats)\n6. [↩️ Return Value](#return-value)\n\n## 📖 Overview\n\nThis documentation provides comprehensive API reference for DepthAnything3, including usage examples, parameter specifications, export formats, and advanced features. It covers both basic pose and depth estimation workflows and advanced pose-conditioned processing with multiple export capabilities.\n\n## 💡 Usage Examples\n\nHere are quick examples to get you started:\n\n### 🚀 Basic Depth Estimation\n```python\nfrom depth_anything_3.api import DepthAnything3\n\n# Initialize and run inference\nmodel = DepthAnything3.from_pretrained(\"depth-anything/DA3NESTED-GIANT-LARGE\").to(\"cuda\")\nprediction = model.inference([\"image1.jpg\", \"image2.jpg\"])\n```\n\n### 📷 Pose-Conditioned Depth Estimation\n```python\nimport numpy as np\n\n# With camera parameters for better consistency\nprediction = model.inference(\n    image=[\"image1.jpg\", \"image2.jpg\"],\n    extrinsics=extrinsics_array,  # (N, 4, 4)\n    intrinsics=intrinsics_array   # (N, 3, 3)\n)\n```\n\n### 📤 Export Results\n```python\n# Export depth data and 3D visualization\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"mini_npz-glb\"\n)\n```\n\n### 🔍 Feature Extraction\n```python\n# Export intermediate features from specific layers\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"feat_vis\",\n    export_feat_layers=[0, 1, 2]  # Export features from layers 0, 1, 2\n)\n```\n\n### ✨ Advanced Export with Gaussian Splatting\n```python\n# Export multiple formats including Gaussian Splatting\n# Note: infer_gs=True requires da3-giant or da3nested-giant-large model\nmodel = DepthAnything3(model_name=\"da3-giant\").to(\"cuda\")\n\nprediction = model.inference(\n    image=image_paths,\n    extrinsics=extrinsics_array,\n    intrinsics=intrinsics_array,\n    export_dir=\"./output\",\n    export_format=\"npz-glb-gs_ply-gs_video\",\n    align_to_input_ext_scale=True,\n    infer_gs=True,  # Required for gs_ply and gs_video exports\n)\n```\n\n### 🎨 Advanced Export with Feature Visualization\n```python\n# Export with intermediate feature visualization\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"mini_npz-glb-depth_vis-feat_vis\",\n    export_feat_layers=[0, 5, 10, 15, 20],\n    feat_vis_fps=30,\n)\n```\n\n### 📐 Using Ray-Based Pose Estimation\n```python\n# Use ray-based pose estimation instead of camera decoder\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"glb\",\n    use_ray_pose=True,  # Enable ray-based pose estimation\n)\n```\n\n### 🎯 Reference View Selection\n```python\n# For multi-view inputs, automatically select the best reference view\nprediction = model.inference(\n    image=image_paths,\n    ref_view_strategy=\"saddle_balanced\",  # Default: balanced selection\n)\n\n# For video sequences, use middle frame as reference\nprediction = model.inference(\n    image=video_frames,\n    ref_view_strategy=\"middle\",  # Good for temporally ordered inputs\n)\n```\n\n## 🔧 Core API\n\n### 🔨 DepthAnything3 Class\n\nThe main API class that provides depth estimation capabilities with optional pose conditioning.\n\n#### 🎯 Initialization\n\n```python\nfrom depth_anything_3 import DepthAnything3\n\n# Initialize the model with a model name\nmodel = DepthAnything3(model_name=\"da3-large\")\nmodel = model.to(\"cuda\")  # Move to GPU\n```\n\n**Parameters:**\n- `model_name` (str, default: \"da3-large\"): The name of the model preset to use.\n  - **Available models:**\n    - 🦾 `\"da3-giant\"` - 1.15B params, any-view model with GS support\n    - ⭐ `\"da3-large\"` - 0.35B params, any-view model (recommended for most use cases)\n    - 📦 `\"da3-base\"` - 0.12B params, any-view model\n    - 🪶 `\"da3-small\"` - 0.08B params, any-view model\n    - 👁️ `\"da3mono-large\"` - 0.35B params, monocular depth only\n    - 📏 `\"da3metric-large\"` - 0.35B params, metric depth with sky segmentation\n    - 🎯 `\"da3nested-giant-large\"` - 1.40B params, nested model with all features\n\n### 🚀 inference() Method\n\nThe primary inference method that processes images and returns depth predictions.\n\n```python\nprediction = model.inference(\n    image=image_list,\n    extrinsics=extrinsics_array,      # Optional\n    intrinsics=intrinsics_array,      # Optional\n    align_to_input_ext_scale=True,   # Whether to align predicted poses to input scale\n    infer_gs=True,                   # Enable Gaussian branch for gs exports\n    use_ray_pose=False,              # Use ray-based pose estimation instead of camera decoder\n    ref_view_strategy=\"saddle_balanced\",  # Reference view selection strategy\n    render_exts=render_extrinsics,    # Optional renders for gs_video\n    render_ixts=render_intrinsics,    # Optional renders for gs_video\n    render_hw=(height, width),        # Optional renders for gs_video\n    process_res=504,\n    process_res_method=\"upper_bound_resize\",\n    export_dir=\"output_directory\",    # Optional\n    export_format=\"mini_npz\",\n    export_feat_layers=[],            # List of layer indices to export features from\n    conf_thresh_percentile=40.0,      # Confidence threshold percentile for depth map in GLB export\n    num_max_points=1_000_000,         # Maximum number of points to export in GLB export\n    show_cameras=True,                # Whether to show cameras in GLB export\n    feat_vis_fps=15,                  # Frames per second for feature visualization in feat_vis export\n    export_kwargs={}                  # Optional, additional arguments to export functions. export_format:key:val, see 'Parameters/Export Parameters' for details\n)\n```\n\n## ⚙️ Parameters\n\n### 📸 Input Parameters\n\n#### `image` (required)\n- **Type**: `List[Union[np.ndarray, Image.Image, str]]`\n- **Description**: List of input images. Can be numpy arrays, PIL Images, or file paths.\n- **Example**:\n  ```python\n  # From file paths\n  image = [\"image1.jpg\", \"image2.jpg\", \"image3.jpg\"]\n\n  # From numpy arrays\n  image = [np.array(img1), np.array(img2)]\n\n  # From PIL Images\n  image = [Image.open(\"image1.jpg\"), Image.open(\"image2.jpg\")]\n  ```\n\n#### `extrinsics` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(N, 4, 4)` where N is the number of input images\n- **Description**: Camera extrinsic matrices (world-to-camera transformation). When provided, enables pose-conditioned depth estimation mode.\n- **Note**: If not provided, the model operates in standard depth estimation mode.\n\n#### `intrinsics` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(N, 3, 3)` where N is the number of input images\n- **Description**: Camera intrinsic matrices containing focal length and principal point information. When provided, enables pose-conditioned depth estimation mode.\n\n### 🎯 Pose Alignment Parameters\n\n#### `align_to_input_ext_scale` (default: True)\n- **Type**: `bool`\n- **Description**: When True the predicted extrinsics are replaced with the input\n  ones and the depth maps are rescaled to match their metric scale. When False the\n  function returns the internally aligned poses computed via Umeyama alignment.\n\n#### `infer_gs` (default: False)\n- **Type**: `bool`\n- **Description**: Enable Gaussian Splatting branch for gaussian splatting exports. Required when using `gs_ply` or `gs_video` export formats.\n\n#### `use_ray_pose` (default: False)\n- **Type**: `bool`\n- **Description**: Use ray-based pose estimation instead of camera decoder for pose prediction. When True, the model uses ray prediction heads to estimate camera poses; when False, it uses the camera decoder approach.\n\n#### `ref_view_strategy` (default: \"saddle_balanced\")\n- **Type**: `str`\n- **Description**: Strategy for selecting the reference view from multiple input views. Options: `\"first\"`, `\"middle\"`, `\"saddle_balanced\"`, `\"saddle_sim_range\"`. Only applied when number of views ≥ 3. See [detailed documentation](funcs/ref_view_strategy.md) for strategy comparisons.\n- **Available strategies**:\n  - `\"saddle_balanced\"`: Selects view with balanced features across multiple metrics (recommended default)\n  - `\"saddle_sim_range\"`: Selects view with largest similarity range\n  - `\"first\"`: Always uses first view (not recommended, equivalent to no reordering for views < 3)\n  - `\"middle\"`: Uses middle view (recommended for video sequences)\n\n### 🔍 Feature Export Parameters\n\n#### `export_feat_layers` (default: [])\n- **Type**: `List[int]`\n- **Description**: List of layer indices to export intermediate features from. Features are stored in the `aux` dictionary of the Prediction object with keys like `feat_layer_0`, `feat_layer_1`, etc.\n\n### 🎥 Rendering Parameters\n\nThese arguments are only used when exporting Gaussian-splatting videos (include\n`\"gs_video\"` in `export_format`). They describe an auxiliary camera trajectory\nwith ``M`` views.\n\n#### `render_exts` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(M, 4, 4)`\n- **Description**: Camera extrinsics for the synthesized trajectory. If omitted,\n  the exporter falls back to the predicted poses.\n\n#### `render_ixts` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(M, 3, 3)`\n- **Description**: Camera intrinsics for each rendered frame. Leave `None` to\n  reuse the input intrinsics.\n\n#### `render_hw` (optional)\n- **Type**: `Optional[Tuple[int, int]]`\n- **Description**: Explicit output resolution `(height, width)` for the rendered\n  frames. Defaults to the input resolution when not provided.\n\n### ⚡ Processing Parameters\n\n#### `process_res` (default: 504)\n- **Type**: `int`\n- **Description**: Base resolution for processing. The model will resize images to this resolution for inference.\n\n#### `process_res_method` (default: \"upper_bound_resize\")\n- **Type**: `str`\n- **Description**: Method for resizing images to the target resolution.\n- **Options**:\n  - `\"upper_bound_resize\"`: Resize so that the specified dimension (504) becomes the longer side\n  - `\"lower_bound_resize\"`: Resize so that the specified dimension (504) becomes the shorter side\n- **Example**:\n  - Input: 1200×1600 → Output: 378×504 (with `process_res=504`, `process_res_method=\"upper_bound_resize\"`)\n  - Input: 504×672 → Output: 504×672 (no change needed)\n\n### 📦 Export Parameters\n\n#### `export_dir` (optional)\n- **Type**: `Optional[str]`\n- **Description**: Directory path where exported files will be saved. If not provided, no files will be exported.\n\n#### `export_format` (default: \"mini_npz\")\n- **Type**: `str`\n- **Description**: Format for exporting results. Supports multiple formats separated by `-`.\n- **Example**: `\"mini_npz-glb\"` exports both mini_npz and glb formats.\n\n#### 🌐 GLB Export Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"glb\"`.\n\n##### `conf_thresh_percentile` (default: 40.0)\n- **Type**: `float`\n- **Description**: Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out from the point cloud.\n\n##### `num_max_points` (default: 1,000,000)\n- **Type**: `int`\n- **Description**: Maximum number of points in the exported point cloud. If the point cloud exceeds this limit, it will be downsampled.\n\n##### `show_cameras` (default: True)\n- **Type**: `bool`\n- **Description**: Whether to include camera wireframes in the exported GLB file for visualization.\n\n#### 🎨 Feature Visualization Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"feat_vis\"`.\n\n##### `feat_vis_fps` (default: 15)\n- **Type**: `int`\n- **Description**: Frame rate for the output video when visualizing features across multiple images.\n\n#### ✨🎥 3DGS and 3DGS Video Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"gs_ply\"` or `\"gs_video\"`.\n\n##### `export_kwargs` (default: `{}`)\n- Type: `dict[str, dict[str, Any]]`\n- Description: Per-format extra arguments passed to export functions, mainly for `\"gs_ply\"` and `\"gs_video\"`.\n  - Access pattern: `export_kwargs[export_format][key] = value`\n  - Example:\n    ```python\n    {\n        \"gs_ply\": {\n            \"gs_views_interval\": 1,\n        },\n        \"gs_video\": {\n            \"trj_mode\": \"interpolate_smooth\",\n            \"chunk_size\": 1,\n            \"vis_depth\": None,\n        },\n    }\n    ```\n\n## 📤 Export Formats\n\nThe API supports multiple export formats for different use cases:\n\n### 📊 `mini_npz`\n- **Description**: Minimal NPZ format containing essential data\n- **Contents**: `depth`, `conf`, `exts`, `ixts`\n- **Use case**: Lightweight storage for depth data with camera parameters\n\n### 📦 `npz`\n- **Description**: Full NPZ format with comprehensive data\n- **Contents**: `depth`, `conf`, `exts`, `ixts`, `image`, etc.\n- **Use case**: Complete data export for advanced processing\n\n### 🌐 `glb`\n- **Description**: 3D visualization format with point cloud and camera poses\n- **Contents**:\n  - Point cloud with colors from original images\n  - Camera wireframes for visualization\n  - Confidence-based filtering and downsampling\n- **Use case**: 3D visualization, inspection, and analysis\n- **Features**:\n  - Automatic sky depth handling\n  - Confidence threshold filtering\n  - Background filtering (black/white)\n  - Scene scale normalization\n- **Parameters** (passed via `inference()` method directly):\n  - `conf_thresh_percentile` (float, default: 40.0): Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out.\n  - `num_max_points` (int, default: 1,000,000): Maximum number of points in the exported point cloud. If exceeded, points will be downsampled.\n  - `show_cameras` (bool, default: True): Whether to include camera wireframes in the exported GLB file for visualization.\n\n### ✨ `gs_ply`\n- **Description**: Gaussian Splatting point cloud format\n- **Contents**: 3DGS data in PLY format. Compatible with standard 3DGS viewers such as [SuperSplat](https://superspl.at/editor) (recommended), [SPARK](https://sparkjs.dev/viewer/).\n- **Use case**: Gaussian Splatting reconstruction\n- **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models.\n- **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)):\n  - `gs_views_interval`: Export to 3DGS every N views, default: `1`.\n\n### 🎥 `gs_video`\n- **Description**: Rasterized 3DGS to obtain videos\n- **Contents**: A video of 3DGS-rasterized views using either provided viewpoints or a predefined camera trajectory.\n- **Use case**: Video rendering for Gaussian Splatting\n- **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models.\n- **Note**: Can optionally use `render_exts`, `render_ixts`, and `render_hw` parameters in `inference()` method to specify novel viewpoints.\n- **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)):\n  - `extrinsics`: Optional world-to-camera poses for novel views. Falls back to the predicted poses of input views if not provided. (Alternatively, use `render_exts` parameter in `inference()`)\n  - `intrinsics`: Optional camera intrinsics for novel views. Falls back to the predicted intrinsics of input views if not provided. (Alternatively, use `render_ixts` parameter in `inference()`)\n  - `out_image_hw`: Optional output resolution `H x W`. Falls back to input resolution if not provided. (Alternatively, use `render_hw` parameter in `inference()`)\n  - `chunk_size`: Number of views rasterized per batch. Default: `8`.\n  - `trj_mode`: Predefined camera trajectory for novel-view rendering.\n  - `color_mode`: Same as `render_mode` in [gsplat](https://docs.gsplat.studio/main/apis/rasterization.html#gsplat.rasterization).\n  - `vis_depth`: How depth is combined with RGB. Default: `hcat` (horizontal concatenation).\n  - `enable_tqdm`: Whether to display a tqdm progress bar during rendering.\n  - `output_name`: File name of the rendered video.\n  - `video_quality`: Video quality to save. Default: `high`.\n    - `high`: High quality video (default)\n    - `medium`: Medium quality video (balance of storage space and quality)\n    - `low`: Low quality video (fewer storage space)\n\n### 🔍 `feat_vis`\n- **Description**: Feature visualization format\n- **Contents**: PCA-visualized intermediate features from specified layers\n- **Use case**: Model interpretability and feature analysis\n- **Note**: Requires `export_feat_layers` to be specified\n- **Parameters** (passed via `inference()` method directly):\n  - `feat_vis_fps` (int, default: 15): Frame rate for the output video when visualizing features across multiple images.\n\n### 🎨 `depth_vis`\n- **Description**: Depth visualization format\n- **Contents**: Color-coded depth maps alongside original images\n- **Use case**: Visual inspection of depth estimation quality\n\n### 🔗 Multiple Format Export\nYou can export multiple formats simultaneously by separating them with `-`:\n\n```python\n# Export both mini_npz and glb formats\nexport_format = \"mini_npz-glb\"\n\n# Export multiple formats\nexport_format = \"npz-glb-gs_ply\"\n```\n\n## ↩️ Return Value\n\nThe `inference()` method returns a `Prediction` object with the following attributes:\n\n### 📊 Core Outputs\n\n- **depth**: `np.ndarray` - Estimated depth maps with shape `(N, H, W)` where N is the number of images, H is height, and W is width.\n- **conf**: `np.ndarray` - Confidence maps with shape `(N, H, W)` indicating prediction reliability (optional, depends on model).\n\n### 📷 Camera Parameters\n\n- **extrinsics**: `np.ndarray` - Camera extrinsic matrices with shape `(N, 3, 4)` representing world-to-camera transformations. Only present if camera poses were estimated or provided as input.\n- **intrinsics**: `np.ndarray` - Camera intrinsic matrices with shape `(N, 3, 3)` containing focal length and principal point information. Only present if poses were estimated or provided as input.\n\n### 🎁 Additional Outputs\n\n- **processed_images**: `np.ndarray` - Preprocessed input images with shape `(N, H, W, 3)` in RGB format (0-255 uint8).\n- **aux**: `dict` - Auxiliary outputs including:\n  - `feat_layer_X`: Intermediate features from layer X (if `export_feat_layers` was specified)\n  - `gaussians`: 3D Gaussian Splats data (if `infer_gs=True`)\n\n### 💻 Usage Example\n\n```python\nprediction = model.inference(image=[\"img1.jpg\", \"img2.jpg\"])\n\n# Access depth maps\ndepth_maps = prediction.depth  # shape: (2, H, W)\n\n# Access confidence\nif hasattr(prediction, 'conf'):\n    confidence = prediction.conf\n\n# Access camera parameters (if available)\nif hasattr(prediction, 'extrinsics'):\n    camera_poses = prediction.extrinsics  # shape: (2, 4, 4)\n\nif hasattr(prediction, 'intrinsics'):\n    camera_intrinsics = prediction.intrinsics  # shape: (2, 3, 3)\n\n# Access intermediate features (if export_feat_layers was set)\nif hasattr(prediction, 'aux') and 'feat_layer_0' in prediction.aux:\n    features = prediction.aux['feat_layer_0']\n```\n","isInternal":false,"tokens":4896,"sizeBytes":19712}],"systemPromptSnippet":"<agent_rules repository=\"ByteDance-Seed/Depth-Anything-3\">\n\n<!-- Skill/Rule: docs - API Interface Contract & Specifications (docs/API.md) -->\n# 📚 DepthAnything3 API Documentation\n\n## 📑 Table of Contents\n\n1. [📖 Overview](#overview)\n2. [💡 Usage Examples](#usage-examples)\n3. [🔧 Core API](#core-api)\n   - [DepthAnything3 Class](#depthanything3-class)\n   - [inference() Method](#inference-method)\n4. [⚙️ Parameters](#parameters)\n   - [Input Parameters](#input-parameters)\n   - [Pose Alignment Parameters](#pose-alignment-parameters)\n   - [Feature Export Parameters](#feature-export-parameters)\n   - [Rendering Parameters](#rendering-parameters)\n   - [Processing Parameters](#processing-parameters)\n   - [Export Parameters](#export-parameters)\n5. [📤 Export Formats](#export-formats)\n6. [↩️ Return Value](#return-value)\n\n## 📖 Overview\n\nThis documentation provides comprehensive API reference for DepthAnything3, including usage examples, parameter specifications, export formats, and advanced features. It covers both basic pose and depth estimation workflows and advanced pose-conditioned processing with multiple export capabilities.\n\n## 💡 Usage Examples\n\nHere are quick examples to get you started:\n\n### 🚀 Basic Depth Estimation\n```python\nfrom depth_anything_3.api import DepthAnything3\n\n# Initialize and run inference\nmodel = DepthAnything3.from_pretrained(\"depth-anything/DA3NESTED-GIANT-LARGE\").to(\"cuda\")\nprediction = model.inference([\"image1.jpg\", \"image2.jpg\"])\n```\n\n### 📷 Pose-Conditioned Depth Estimation\n```python\nimport numpy as np\n\n# With camera parameters for better consistency\nprediction = model.inference(\n    image=[\"image1.jpg\", \"image2.jpg\"],\n    extrinsics=extrinsics_array,  # (N, 4, 4)\n    intrinsics=intrinsics_array   # (N, 3, 3)\n)\n```\n\n### 📤 Export Results\n```python\n# Export depth data and 3D visualization\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"mini_npz-glb\"\n)\n```\n\n### 🔍 Feature Extraction\n```python\n# Export intermediate features from specific layers\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"feat_vis\",\n    export_feat_layers=[0, 1, 2]  # Export features from layers 0, 1, 2\n)\n```\n\n### ✨ Advanced Export with Gaussian Splatting\n```python\n# Export multiple formats including Gaussian Splatting\n# Note: infer_gs=True requires da3-giant or da3nested-giant-large model\nmodel = DepthAnything3(model_name=\"da3-giant\").to(\"cuda\")\n\nprediction = model.inference(\n    image=image_paths,\n    extrinsics=extrinsics_array,\n    intrinsics=intrinsics_array,\n    export_dir=\"./output\",\n    export_format=\"npz-glb-gs_ply-gs_video\",\n    align_to_input_ext_scale=True,\n    infer_gs=True,  # Required for gs_ply and gs_video exports\n)\n```\n\n### 🎨 Advanced Export with Feature Visualization\n```python\n# Export with intermediate feature visualization\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"mini_npz-glb-depth_vis-feat_vis\",\n    export_feat_layers=[0, 5, 10, 15, 20],\n    feat_vis_fps=30,\n)\n```\n\n### 📐 Using Ray-Based Pose Estimation\n```python\n# Use ray-based pose estimation instead of camera decoder\nprediction = model.inference(\n    image=image_paths,\n    export_dir=\"./output\",\n    export_format=\"glb\",\n    use_ray_pose=True,  # Enable ray-based pose estimation\n)\n```\n\n### 🎯 Reference View Selection\n```python\n# For multi-view inputs, automatically select the best reference view\nprediction = model.inference(\n    image=image_paths,\n    ref_view_strategy=\"saddle_balanced\",  # Default: balanced selection\n)\n\n# For video sequences, use middle frame as reference\nprediction = model.inference(\n    image=video_frames,\n    ref_view_strategy=\"middle\",  # Good for temporally ordered inputs\n)\n```\n\n## 🔧 Core API\n\n### 🔨 DepthAnything3 Class\n\nThe main API class that provides depth estimation capabilities with optional pose conditioning.\n\n#### 🎯 Initialization\n\n```python\nfrom depth_anything_3 import DepthAnything3\n\n# Initialize the model with a model name\nmodel = DepthAnything3(model_name=\"da3-large\")\nmodel = model.to(\"cuda\")  # Move to GPU\n```\n\n**Parameters:**\n- `model_name` (str, default: \"da3-large\"): The name of the model preset to use.\n  - **Available models:**\n    - 🦾 `\"da3-giant\"` - 1.15B params, any-view model with GS support\n    - ⭐ `\"da3-large\"` - 0.35B params, any-view model (recommended for most use cases)\n    - 📦 `\"da3-base\"` - 0.12B params, any-view model\n    - 🪶 `\"da3-small\"` - 0.08B params, any-view model\n    - 👁️ `\"da3mono-large\"` - 0.35B params, monocular depth only\n    - 📏 `\"da3metric-large\"` - 0.35B params, metric depth with sky segmentation\n    - 🎯 `\"da3nested-giant-large\"` - 1.40B params, nested model with all features\n\n### 🚀 inference() Method\n\nThe primary inference method that processes images and returns depth predictions.\n\n```python\nprediction = model.inference(\n    image=image_list,\n    extrinsics=extrinsics_array,      # Optional\n    intrinsics=intrinsics_array,      # Optional\n    align_to_input_ext_scale=True,   # Whether to align predicted poses to input scale\n    infer_gs=True,                   # Enable Gaussian branch for gs exports\n    use_ray_pose=False,              # Use ray-based pose estimation instead of camera decoder\n    ref_view_strategy=\"saddle_balanced\",  # Reference view selection strategy\n    render_exts=render_extrinsics,    # Optional renders for gs_video\n    render_ixts=render_intrinsics,    # Optional renders for gs_video\n    render_hw=(height, width),        # Optional renders for gs_video\n    process_res=504,\n    process_res_method=\"upper_bound_resize\",\n    export_dir=\"output_directory\",    # Optional\n    export_format=\"mini_npz\",\n    export_feat_layers=[],            # List of layer indices to export features from\n    conf_thresh_percentile=40.0,      # Confidence threshold percentile for depth map in GLB export\n    num_max_points=1_000_000,         # Maximum number of points to export in GLB export\n    show_cameras=True,                # Whether to show cameras in GLB export\n    feat_vis_fps=15,                  # Frames per second for feature visualization in feat_vis export\n    export_kwargs={}                  # Optional, additional arguments to export functions. export_format:key:val, see 'Parameters/Export Parameters' for details\n)\n```\n\n## ⚙️ Parameters\n\n### 📸 Input Parameters\n\n#### `image` (required)\n- **Type**: `List[Union[np.ndarray, Image.Image, str]]`\n- **Description**: List of input images. Can be numpy arrays, PIL Images, or file paths.\n- **Example**:\n  ```python\n  # From file paths\n  image = [\"image1.jpg\", \"image2.jpg\", \"image3.jpg\"]\n\n  # From numpy arrays\n  image = [np.array(img1), np.array(img2)]\n\n  # From PIL Images\n  image = [Image.open(\"image1.jpg\"), Image.open(\"image2.jpg\")]\n  ```\n\n#### `extrinsics` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(N, 4, 4)` where N is the number of input images\n- **Description**: Camera extrinsic matrices (world-to-camera transformation). When provided, enables pose-conditioned depth estimation mode.\n- **Note**: If not provided, the model operates in standard depth estimation mode.\n\n#### `intrinsics` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(N, 3, 3)` where N is the number of input images\n- **Description**: Camera intrinsic matrices containing focal length and principal point information. When provided, enables pose-conditioned depth estimation mode.\n\n### 🎯 Pose Alignment Parameters\n\n#### `align_to_input_ext_scale` (default: True)\n- **Type**: `bool`\n- **Description**: When True the predicted extrinsics are replaced with the input\n  ones and the depth maps are rescaled to match their metric scale. When False the\n  function returns the internally aligned poses computed via Umeyama alignment.\n\n#### `infer_gs` (default: False)\n- **Type**: `bool`\n- **Description**: Enable Gaussian Splatting branch for gaussian splatting exports. Required when using `gs_ply` or `gs_video` export formats.\n\n#### `use_ray_pose` (default: False)\n- **Type**: `bool`\n- **Description**: Use ray-based pose estimation instead of camera decoder for pose prediction. When True, the model uses ray prediction heads to estimate camera poses; when False, it uses the camera decoder approach.\n\n#### `ref_view_strategy` (default: \"saddle_balanced\")\n- **Type**: `str`\n- **Description**: Strategy for selecting the reference view from multiple input views. Options: `\"first\"`, `\"middle\"`, `\"saddle_balanced\"`, `\"saddle_sim_range\"`. Only applied when number of views ≥ 3. See [detailed documentation](funcs/ref_view_strategy.md) for strategy comparisons.\n- **Available strategies**:\n  - `\"saddle_balanced\"`: Selects view with balanced features across multiple metrics (recommended default)\n  - `\"saddle_sim_range\"`: Selects view with largest similarity range\n  - `\"first\"`: Always uses first view (not recommended, equivalent to no reordering for views < 3)\n  - `\"middle\"`: Uses middle view (recommended for video sequences)\n\n### 🔍 Feature Export Parameters\n\n#### `export_feat_layers` (default: [])\n- **Type**: `List[int]`\n- **Description**: List of layer indices to export intermediate features from. Features are stored in the `aux` dictionary of the Prediction object with keys like `feat_layer_0`, `feat_layer_1`, etc.\n\n### 🎥 Rendering Parameters\n\nThese arguments are only used when exporting Gaussian-splatting videos (include\n`\"gs_video\"` in `export_format`). They describe an auxiliary camera trajectory\nwith ``M`` views.\n\n#### `render_exts` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(M, 4, 4)`\n- **Description**: Camera extrinsics for the synthesized trajectory. If omitted,\n  the exporter falls back to the predicted poses.\n\n#### `render_ixts` (optional)\n- **Type**: `Optional[np.ndarray]`\n- **Shape**: `(M, 3, 3)`\n- **Description**: Camera intrinsics for each rendered frame. Leave `None` to\n  reuse the input intrinsics.\n\n#### `render_hw` (optional)\n- **Type**: `Optional[Tuple[int, int]]`\n- **Description**: Explicit output resolution `(height, width)` for the rendered\n  frames. Defaults to the input resolution when not provided.\n\n### ⚡ Processing Parameters\n\n#### `process_res` (default: 504)\n- **Type**: `int`\n- **Description**: Base resolution for processing. The model will resize images to this resolution for inference.\n\n#### `process_res_method` (default: \"upper_bound_resize\")\n- **Type**: `str`\n- **Description**: Method for resizing images to the target resolution.\n- **Options**:\n  - `\"upper_bound_resize\"`: Resize so that the specified dimension (504) becomes the longer side\n  - `\"lower_bound_resize\"`: Resize so that the specified dimension (504) becomes the shorter side\n- **Example**:\n  - Input: 1200×1600 → Output: 378×504 (with `process_res=504`, `process_res_method=\"upper_bound_resize\"`)\n  - Input: 504×672 → Output: 504×672 (no change needed)\n\n### 📦 Export Parameters\n\n#### `export_dir` (optional)\n- **Type**: `Optional[str]`\n- **Description**: Directory path where exported files will be saved. If not provided, no files will be exported.\n\n#### `export_format` (default: \"mini_npz\")\n- **Type**: `str`\n- **Description**: Format for exporting results. Supports multiple formats separated by `-`.\n- **Example**: `\"mini_npz-glb\"` exports both mini_npz and glb formats.\n\n#### 🌐 GLB Export Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"glb\"`.\n\n##### `conf_thresh_percentile` (default: 40.0)\n- **Type**: `float`\n- **Description**: Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out from the point cloud.\n\n##### `num_max_points` (default: 1,000,000)\n- **Type**: `int`\n- **Description**: Maximum number of points in the exported point cloud. If the point cloud exceeds this limit, it will be downsampled.\n\n##### `show_cameras` (default: True)\n- **Type**: `bool`\n- **Description**: Whether to include camera wireframes in the exported GLB file for visualization.\n\n#### 🎨 Feature Visualization Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"feat_vis\"`.\n\n##### `feat_vis_fps` (default: 15)\n- **Type**: `int`\n- **Description**: Frame rate for the output video when visualizing features across multiple images.\n\n#### ✨🎥 3DGS and 3DGS Video Parameters\n\nThese parameters are passed directly to the `inference()` method and only apply when `export_format` includes `\"gs_ply\"` or `\"gs_video\"`.\n\n##### `export_kwargs` (default: `{}`)\n- Type: `dict[str, dict[str, Any]]`\n- Description: Per-format extra arguments passed to export functions, mainly for `\"gs_ply\"` and `\"gs_video\"`.\n  - Access pattern: `export_kwargs[export_format][key] = value`\n  - Example:\n    ```python\n    {\n        \"gs_ply\": {\n            \"gs_views_interval\": 1,\n        },\n        \"gs_video\": {\n            \"trj_mode\": \"interpolate_smooth\",\n            \"chunk_size\": 1,\n            \"vis_depth\": None,\n        },\n    }\n    ```\n\n## 📤 Export Formats\n\nThe API supports multiple export formats for different use cases:\n\n### 📊 `mini_npz`\n- **Description**: Minimal NPZ format containing essential data\n- **Contents**: `depth`, `conf`, `exts`, `ixts`\n- **Use case**: Lightweight storage for depth data with camera parameters\n\n### 📦 `npz`\n- **Description**: Full NPZ format with comprehensive data\n- **Contents**: `depth`, `conf`, `exts`, `ixts`, `image`, etc.\n- **Use case**: Complete data export for advanced processing\n\n### 🌐 `glb`\n- **Description**: 3D visualization format with point cloud and camera poses\n- **Contents**:\n  - Point cloud with colors from original images\n  - Camera wireframes for visualization\n  - Confidence-based filtering and downsampling\n- **Use case**: 3D visualization, inspection, and analysis\n- **Features**:\n  - Automatic sky depth handling\n  - Confidence threshold filtering\n  - Background filtering (black/white)\n  - Scene scale normalization\n- **Parameters** (passed via `inference()` method directly):\n  - `conf_thresh_percentile` (float, default: 40.0): Lower percentile for adaptive confidence threshold. Points below this confidence percentile will be filtered out.\n  - `num_max_points` (int, default: 1,000,000): Maximum number of points in the exported point cloud. If exceeded, points will be downsampled.\n  - `show_cameras` (bool, default: True): Whether to include camera wireframes in the exported GLB file for visualization.\n\n### ✨ `gs_ply`\n- **Description**: Gaussian Splatting point cloud format\n- **Contents**: 3DGS data in PLY format. Compatible with standard 3DGS viewers such as [SuperSplat](https://superspl.at/editor) (recommended), [SPARK](https://sparkjs.dev/viewer/).\n- **Use case**: Gaussian Splatting reconstruction\n- **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models.\n- **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)):\n  - `gs_views_interval`: Export to 3DGS every N views, default: `1`.\n\n### 🎥 `gs_video`\n- **Description**: Rasterized 3DGS to obtain videos\n- **Contents**: A video of 3DGS-rasterized views using either provided viewpoints or a predefined camera trajectory.\n- **Use case**: Video rendering for Gaussian Splatting\n- **Requirements**: Must set `infer_gs=True` when calling `inference()`. Only supported by `da3-giant` and `da3nested-giant-large` models.\n- **Note**: Can optionally use `render_exts`, `render_ixts`, and `render_hw` parameters in `inference()` method to specify novel viewpoints.\n- **Additional configs**, provided via `export_kwargs` (see [Export Parameters](#export-parameters)):\n  - `extrinsics`: Optional world-to-camera poses for novel views. Falls back to the predicted poses of input views if not provided. (Alternatively, use `render_exts` parameter in `inference()`)\n  - `intrinsics`: Optional camera intrinsics for novel views. Falls back to the predicted intrinsics of input views if not provided. (Alternatively, use `render_ixts` parameter in `inference()`)\n  - `out_image_hw`: Optional output resolution `H x W`. Falls back to input resolution if not provided. (Alternatively, use `render_hw` parameter in `inference()`)\n  - `chunk_size`: Number of views rasterized per batch. Default: `8`.\n  - `trj_mode`: Predefined camera trajectory for novel-view rendering.\n  - `color_mode`: Same as `render_mode` in [gsplat](https://docs.gsplat.studio/main/apis/rasterization.html#gsplat.rasterization).\n  - `vis_depth`: How depth is combined with RGB. Default: `hcat` (horizontal concatenation).\n  - `enable_tqdm`: Whether to display a tqdm progress bar during rendering.\n  - `output_name`: File name of the rendered video.\n  - `video_quality`: Video quality to save. Default: `high`.\n    - `high`: High quality video (default)\n    - `medium`: Medium quality video (balance of storage space and quality)\n    - `low`: Low quality video (fewer storage space)\n\n### 🔍 `feat_vis`\n- **Description**: Feature visualization format\n- **Contents**: PCA-visualized intermediate features from specified layers\n- **Use case**: Model interpretability and feature analysis\n- **Note**: Requires `export_feat_layers` to be specified\n- **Parameters** (passed via `inference()` method directly):\n  - `feat_vis_fps` (int, default: 15): Frame rate for the output video when visualizing features across multiple images.\n\n### 🎨 `depth_vis`\n- **Description**: Depth visualization format\n- **Contents**: Color-coded depth maps alongside original images\n- **Use case**: Visual inspection of depth estimation quality\n\n### 🔗 Multiple Format Export\nYou can export multiple formats simultaneously by separating them with `-`:\n\n```python\n# Export both mini_npz and glb formats\nexport_format = \"mini_npz-glb\"\n\n# Export multiple formats\nexport_format = \"npz-glb-gs_ply\"\n```\n\n## ↩️ Return Value\n\nThe `inference()` method returns a `Prediction` object with the following attributes:\n\n### 📊 Core Outputs\n\n- **depth**: `np.ndarray` - Estimated depth maps with shape `(N, H, W)` where N is the number of images, H is height, and W is width.\n- **conf**: `np.ndarray` - Confidence maps with shape `(N, H, W)` indicating prediction reliability (optional, depends on model).\n\n### 📷 Camera Parameters\n\n- **extrinsics**: `np.ndarray` - Camera extrinsic matrices with shape `(N, 3, 4)` representing world-to-camera transformations. Only present if camera poses were estimated or provided as input.\n- **intrinsics**: `np.ndarray` - Camera intrinsic matrices with shape `(N, 3, 3)` containing focal length and principal point information. Only present if poses were estimated or provided as input.\n\n### 🎁 Additional Outputs\n\n- **processed_images**: `np.ndarray` - Preprocessed input images with shape `(N, H, W, 3)` in RGB format (0-255 uint8).\n- **aux**: `dict` - Auxiliary outputs including:\n  - `feat_layer_X`: Intermediate features from layer X (if `export_feat_layers` was specified)\n  - `gaussians`: 3D Gaussian Splats data (if `infer_gs=True`)\n\n### 💻 Usage Example\n\n```python\nprediction = model.inference(image=[\"img1.jpg\", \"img2.jpg\"])\n\n# Access depth maps\ndepth_maps = prediction.depth  # shape: (2, H, W)\n\n# Access confidence\nif hasattr(prediction, 'conf'):\n    confidence = prediction.conf\n\n# Access camera parameters (if available)\nif hasattr(prediction, 'extrinsics'):\n    camera_poses = prediction.extrinsics  # shape: (2, 4, 4)\n\nif hasattr(prediction, 'intrinsics'):\n    camera_intrinsics = prediction.intrinsics  # shape: (2, 3, 3)\n\n# Access intermediate features (if export_feat_layers was set)\nif hasattr(prediction, 'aux') and 'feat_layer_0' in prediction.aux:\n    features = prediction.aux['feat_layer_0']\n```\n\n\n</agent_rules>"}