# Technical Documentation: NVIDIAGameWorks/kaolin > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/NVIDIAGameWorks/kaolin) ### Overview (README.md) # Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research > **TL;DR** — Kaolin is NVIDIA's PyTorch library of reusable, GPU-optimized modules distilled from 3D deep learning research. > > - Representation-agnostic **physics** (meshes, splats, point clouds) with Simplicits > - **Differentiable rendering** (DIB-R, nvdiffrast, easy_render PBR) > - First-class **3D Gaussian splats** (PLY/USD I/O, densification, gsplat bridge) > - GPU **octree** acceleration structure: Structured Point Clouds (SPC) > - **Conversions** between 3D representations, **quaternion** ops, and **USD** I/O

Kaolin Physics simulation with Kaolin

[](https://kaolin.readthedocs.io/en/latest/) [](https://github.com/NVIDIAGameWorks/kaolin/releases) [](LICENSE) [](https://developer.nvidia.com/kaolin) Kaolin packages reusable building blocks from NVIDIA 3D research into a cohesive PyTorch API — continuously improving representation-agnostic physics simulation, fast conversions between representations, quaternion math, batched mesh and splat containers, I/O, visualization and more. See [kaolin.readthedocs.io](https://kaolin.readthedocs.io/en/latest/) for tutorials and API reference, and [developer.nvidia.com/kaolin](https://developer.nvidia.com/kaolin) for the NVIDIA Kaolin hub. ## SIGGRAPH 2026 Join us in Los Angeles! Two sessions showcasing new Kaolin capabilities. [Full details here](https://kaolin.readthedocs.io/en/latest/notes/siggraph2026.html).

**Talk — Any Representation, Any Hardware, All Interactions** Accelerating Interactive Prototypes Over Cutting Edge AI & 3D Research Sun **19 Jul 2026**, 3:00–3:20 pm PDT · Room **408 A** Maria Shugrina (NVIDIA / University of Toronto) Introduces Kaolin's new **web client-server framework** (`kaolin/visualize/dash`) for rapid prototyping of interactive browser interfaces over emerging AI and 3D research — patterns distilled from building interactive tools at SIGGRAPH technical papers, Real-Time Live, and Labs. Available on the [`web_framework_prerelease` branch](https://github.com/NVIDIAGameWorks/kaolin/tree/web_framework_prerelease) (not yet merged to `master`). [Schedule →](https://s2026.conference-schedule.org/presentation/?id=gensub_229&sess=sess169)

**Hands-on Lab — From 3D Captures to Simulated Digital Environments** Thu **23 Jul 2026**, 10:15–11:45 am PDT · **Concourse Hall** Clement Fuji Tsang, Vismay Modi, Maria Shugrina (NVIDIA) A **capture-to-simulation pipeline** for in-the-wild 3D Gaussian Splat scenes: segment objects, predict volumetric mechanical properties, run mixed splat–mesh physics — powered by recent Kaolin features. Export shareable **USD files with Kaolin's custom physics schema**. **Bring your laptop** and follow along hands-on. Full Conference badge required. [Schedule →](https://s2026.conference-schedule.org/presentation/?id=gensub_123&sess=sess257)

## Features

Physics (Simplicits) Simulate meshes, splats, and point clouds with collisions. Warp-accelerated, representation-agnostic. [Docs](https://kaolin.readthedocs.io/en/latest/notes/simplicits.html) · [Mesh](examples/tutorial/physics/simplicits_mesh.ipynb) · [Splat](examples/tutorial/physics/simplicits_gaussians_splatting.ipynb) · [3DGRUT](examples/tutorial/physics/simulatable_3dgrut.ipynb)	3D Gaussian Splats `GaussianSplatModel`, PLY/USD I/O, densification, gsplat camera converters. [Tutorial](examples/tutorial/working_with_gaussians.ipynb) · [Simulate](examples/tutorial/physics/simplicits_gaussians_splatting.ipynb) · [Interactive viz](examples/tutorial/nerfstudio_gsplat_interactive_visualizer.ipynb)	Differentiable Rendering DIB-R, nvdiffrast, `easy_render` PBR, spherical harmonics and spherical gaussians lighting. [Docs](https://kaolin.readthedocs.io/en/latest/notes/diff_render.html) · [DIB-R](examples/tutorial/dibr_tutorial.ipynb) · [Easy render](examples/tutorial/easy_mesh_render.ipynb) · [Camera](examples/tutorial/camera_and_rasterization.ipynb) · [Lighting](https://kaolin.readthedocs.io/en/latest/notes/differentiable_lighting.html)
Structured Point Clouds GPU octree acceleration structure with ray tracing and feature grids. [Docs](https://kaolin.readthedocs.io/en/latest/notes/spc_summary.html) · [Tutorial](examples/tutorial/understanding_spcs_tutorial.ipynb) · [API](https://kaolin.readthedocs.io/en/latest/modules/kaolin.ops.spc.html)	USD Pipeline Import/export meshes, point clouds, gaussians, and physics materials with Kaolin's custom schema. [API](https://kaolin.readthedocs.io/en/latest/modules/kaolin.io.usd.html) · [Checkpoints](https://kaolin.readthedocs.io/en/latest/notes/checkpoints.html) · [GLTF viz](examples/tutorial/gltf_viz.ipynb)	Visualization Jupyter 3D viewer, Timelapse checkpoints, and web client-server framework. [Docs](https://kaolin.readthedocs.io/en/latest/notes/visualizer.html) · [Interactive](examples/tutorial/interactive_visualizer.ipynb) · [Checkpoints](https://kaolin.readthedocs.io/en/latest/notes/checkpoints.html) · [Web framework branch](https://github.com/NVIDIAGameWorks/kaolin/tree/web_framework_prerelease)
Conversions Fast GPU conversions between meshes, voxel grids, point clouds, gaussians, and more. [Docs](https://kaolin.readthedocs.io/en/latest/notes/conversions.html) · [DMTet](examples/tutorial/dmtet_tutorial.ipynb) · [FlexiCubes](https://kaolin.readthedocs.io/en/latest/notes/volumetric_meshes.html)	Quaternions Differentiable quaternion and rigid-transform utilities for 3D deep learning. [Docs](https://kaolin.readthedocs.io/en/latest/notes/quaternions.html) · [Tutorial](examples/tutorial/quaternion_tutorial.ipynb) · [API](https://kaolin.readthedocs.io/en/latest/modules/kaolin.math.html)	Surface Meshes Batched `SurfaceMesh` container with auto-computed attributes and I/O. [Docs](https://kaolin.readthedocs.io/en/latest/notes/surface_meshes.html) · [Tutorial](examples/tutorial/working_with_meshes.ipynb) · [Easy render](examples/tutorial/easy_mesh_render.ipynb)

*Experimental:* [Newton coupling](kaolin/experimental/newton/README.md) — Simplicits soft bodies with rigid bodies, MPM, and articulated robots ([rigid](examples/tutorial/physics/newton_rigidbody_coupling.ipynb) · [MPM](examples/tutorial/physics/newton_mpm_coupling_oneway.ipynb) · [Franka](examples/tutorial/physics/newton_franka_coupling.ipynb)). See the [tutorial index](https://kaolin.readthedocs.io/en/latest/notes/tutorial_index.html) and [API reference](https://kaolin.readthedocs.io/en/latest/) at [kaolin.readthedocs.io](https://kaolin.readthedocs.io/en/latest/). ## Installation Starting with v0.12.0, Kaolin supports installation with pre-built wheels: ```bash # Replace TORCH_VERSION and CUDA_VERSION with your torch / cuda versions pip install kaolin==0.18.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-{TORCH_VERSION}_cu{CUDA_VERSION}.html ``` For example, kaolin 0.18.0 with PyTorch 2.8.0 and CUDA 12.8: ```bash pip install kaolin==0.18.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.8.0_cu128.html ``` See the [installation guide](https://kaolin.readthedocs.io/en/latest/notes/installation.html) for the full torch/CUDA compatibility matrix and [source install](https://kaolin.readthedocs.io/en/latest/notes/installation.html#install-from-source) instructions. ```bash python -c "import kaolin; print(kaolin.__version__)" ``` ## Quickstart ```python import kaolin print(kaolin.__version__) ``` Load a 3D Gaussian splat from PLY or USD: ```python import kaolin from kaolin.rep import GaussianSplatModel gs = kaolin.io.import_gaussiancloud("scene.ply") print(gs) # GaussianSplatModel with positions, scales, rotations, opacities, ... ``` Simulate a mesh with Simplicits — see the [physics tutorial](examples/tutorial/physics/simplicits_mesh.ipynb). Render a mesh with the easy PBR API: ```python import kaolin as kal mesh = kal.io.obj.import_mesh("model.obj") camera = kal.render.easy_render.default_camera(512) lighting = kal.render.easy_render.default_lighting() result = kal.render.easy_render.render_mesh(camera, mesh, lighting=lighting) ``` ## News ### Unreleased (master or staging) Recent work on `master` since [v0.18.0](https://github.com/NVIDIAGameWorks/kaolin/releases/tag/v0.18.0): - **[FreeForm / RKPM](https://research.nvidia.com/labs/sil/projects/freeform/)** (CVPR 2026) — mesh-free, reduced-order deformable simulation for meshes and Gaussian splats. Builds skinning eigenmodes with a Reproducing Kernel Particle Method (RKPM) basis instead of per-shape neural-field optimization — about **40× faster training** and lower error vs. FEM. Now integrated in Kaolin Simplicits. - **Web client-server framework** (`kaolin/visualize/dash`) — rapid prototyping of interactive Web UIs over AI and 3D research ([`web_framework_prerelease` branch](https://github.com/NVIDIAGameWorks/kaolin/tree/web_framework_prerelease); [SIGGRAPH 2026 talk](https://s2026.conference-schedule.org/presentation/?id=gensub_229&sess=sess169)) - **`GaussianSplatModel`** and **`PointSamples`** tensor-container API - **PLY/USD gaussian I/O** with feature preservation - **USD physics schema** — materials, skinned physics, subset features - **gsplat** batched camera converters - **Newton coupling** — soft bodies with rigid/MPM/Franka ([notebooks](examples/tutorial/physics/newton_rigidbody_coupling.ipynb)) - **Simplicits Easy API** save/load redesign and collision friction fixes - **FlexiCubes** now Apache 2.0 at [`kaolin/ops/conversions/flexicubes/`](kaolin/ops/conversions/flexicubes/flexicubes.py) ### v0.18.0 highlights - [Collisions](https://kaolin.readthedocs.io/en/latest/modules/kaolin.physics.simplicits.html#kaolin.physics.simplicits.SimplicitsScene.enable_collisions) in the physics module - [3D gaussians → voxelgrid](https://kaolin.readthedocs.io/en/latest/modules/kaolin.ops.conversions.html#kaolin.ops.conversions.gs_to_voxelgrid) conversion and volume densifier - [Mesh + gaussian physics with 3DGRUT rendering](examples/tutorial/physics/simulatable_3dgrut.ipynb) - FlexiCubes relicensed to Apache 2.0 See [release notes](https://github.com/NVIDIAGameWorks/kaolin/releases/tag/v0.18.0) for details. ## Tutorials Notebooks live under [`examples/tutorial/`](examples/tutorial/). Highlights by topic: **Physics** - [Simulate a mesh with Simplicits](examples/tutorial/physics/simplicits_mesh.ipynb) - [Simulate a Gaussian splat](examples/tutorial/physics/simplicits_gaussians_splatting.ipynb) - [Mesh + splat physics with 3DGRUT](examples/tutorial/physics/simulatable_3dgrut.ipynb) - [Newton rigid-body coupling](examples/tutorial/physics/newton_rigidbody_coupling.ipynb) **Gaussians** - [Working with GaussianSplatModel](examples/tutorial/working_with_gaussians.ipynb) **Rendering** - [DIB-R differentiable rendering](examples/tutorial/dibr_tutorial.ipynb) - [Easy mesh render](examples/tutorial/easy_mesh_render.ipynb) - [Camera and rasterization](examples/tutorial/camera_and_rasterization.ipynb) **Structured Point Clouds** - [Understanding SPCs](examples/tutorial/understanding_spcs_tutorial.ipynb) **Visualization** - [Interactive visualizer](examples/tutorial/interactive_visualizer.ipynb) - [GLTF visualization](examples/tutorial/gltf_viz.ipynb) Full index: [tutorial index](https://kaolin.readthedocs.io/en/latest/notes/tutorial_index.html). ## Ecosystem Projects built with Kaolin: - [**FreeForm / RKPM**](https://research.nvidia.com/labs/sil/projects/freeform/) — mesh-free reduced-order simulation via RKPM skinning eigenmodes (CVPR 2026) - [**VoMP**](https://research.nvidia.com/labs/sil/projects/vomp/) — feed-forward volumetric mechanical property fields for splats, meshes, and NeRFs (ICLR 2026) - [**ArtisanGS**](https://research.nvidia.com/labs/sil/projects/ArtisanGS/) — interactive Gaussian splat selection and segmentation with AI + human in the loop - [**TRON**](https://research.nvidia.com/labs/sil/projects/tron/) — relightable 3D Gaussian reconstructions with a single-step neural renderer - [**3DGRUT**](https://github.com/nv-tlabs/3DGRUT) — ray tracing and hybrid rasterization of Gaussian particles - [**3DGUT**](https://research.nvidia.com/labs/toronto-ai/3DGUT/) — joint mesh and Gaussian splat rendering (CVPR 2025 oral) - [**Diffusion Texture Painting**](https://research.nvidia.com/labs/toronto-ai/DiffusionTexturePainting/) — interactive diffusion-based texture painting on 3D meshes (SIGGRAPH 2024) - [**NVIDIA Kaolin Wisp**](https://github.com/NVIDIAGameWorks/kaolin-wisp) — neural fields engine (NeRF, NGLOD, instant-ngp) - [**gsplat**](https://github.com/nerfstudio-project/gsplat) — CUDA Gaussian splatting with Kaolin camera bridge - [**NVIDIA Newton**](https://newton-physics.github.io/newton/) — physics engine with experimental [Kaolin coupling](kaolin/experimental/newton/README.md) - [**Neural Geometric LOD (nglod)**](https://github.com/nv-tlabs/nglod) — SPC ray tracing - [**FlexiCubes**](https://github.com/nv-tlabs/FlexiCubes) — gradient-based mesh extraction (SIGGRAPH 2023) - [**DefTet**](https://github.com/nv-tlabs/DefTet) — deformable tetrahedral mesh reconstruction - [**DIB-R**](https://github.com/nv-tlabs/DIB-R-Single-Image-3D-Reconstruction) — single-image 3D reconstruction - [**gradSim**](https://github.com/gradsim/gradsim) — differentiable simulation - [**Text2Mesh**](https://github.com/threedle/text2mesh) — text-driven mesh stylization ## Contributing Please review our [contribution guidelines](CONTRIBUTING.md). ## License Kaolin is released under the [Apache License 2.0](LICENSE). A default `import kaolin` gives you the full Apache-licensed library. The [`kaolin/non_commercial/`](kaolin/non_commercial/) package is **legacy only** — kept for backward compatibility with older import paths (e.g. the pre-Apache FlexiCubes copy). New code should use the Apache-licensed modules under `kaolin/ops/`, `kaolin/rep/`, and the rest of the package tree. ## Citation If you use Kaolin in your research, please cite: ```bibtex @software{KaolinLibrary, author = {Tsang, Clement Fuji and Shugrina, Maria and Lafleche, Jean-Francois and Perel, Or and Loop, Charles and Takikawa, Towaki and Modi, Vismay and Zook, Alexander and Wang, Jiehan and Chen, Wenzheng and Shen, Tianchang and Gao, Jun and Jatavallabhula, Krishna Murthy and Smith, Edward and Rozantsev, Artem and Fidler, Sanja and State, Gavriel and Gorski, Jason and Xiang, Tommy and Li, Jianing and Li, Michael and Lebaredian, Rev}, title = {{Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research}}, version = {0.18.0}, date = {2024-11-20}, url = {https://github.com/NVIDIAGameWorks/kaolin} } ``` ## Contributors **Current team:** Clement Fuji Tsang (Technical Lead), Maria (Masha) Shugrina (Manager), Charles Loop, Vismay Modi, Or Perel **Other major contributors:** Alexander Zook, Donglai Xiang, Wenzheng Chen, Sanja Fidler, Jun Gao, Jason Gorski, Jean-Francois Lafleche, Rev Lebaredian, Jianing Li, Michael Li, Krishna Murthy Jatavallabhula, Artem Rozantsev, Tianchang (Frank) Shen, Edward Smith, Gavriel State, Towaki Takikawa, Jiehan Wang, Tommy Xiang --- ### README (`docs/README.md`) # Documenting This guide is for developers who write API documentation. To build the documentation, run `pip install -r tools/doc_requirements.txt` to install the dependencies for documentation. Then, run ```make html``` on Linux ```make.bat html``` on Windows ## Documenting Python API The best way to document our Python API is to do so directly in the code. That way it's always extracted from a location where it's closest to the actual code and most likely to be correct. Instead of using the older and more cumbersome restructredText Docstring specification we have adopted the more streamlined [Google Python Style Docstring][#5] format. This is how you would document an API function in Python: ```python def answer_question(question): """This function can answer some questions. It currently only answers a limited set of questions so don't expect it to know everything. Args: question (str): The question passed to the function, trailing question mark is not necessary and casing is not important. Returns: answer (str): The answer to the question or ``None`` if it doesn't know the answer. """ if question.lower().startswith("what is the answer to life, universe, and everything"): return str(42) else: return None ``` After running the documentation generation system we will get this as the output: One note: The high-level structure is essentially in four parts: * A one-liner describing the function (without details or corner cases) * A paragraph that gives more detail on the function behavior (if necessary) * An `Args:` section (if the function takes arguments, note that `self` is not considered an argument) * A `Returns:` section (if the function can return something other than `None`) We want to draw your attention to the following: Indentation is key when writing docstrings. The documentation system is clever enough to remove uniform indentation. That is, as long as all the lines have the same amount of padding that padding will be ignored and not passed onto the restructured text processor. Fortunately clang-format leaves this funky formatting alone - respecting the raw string qualifier. Let's now turn our attention to how we document modules and their attributes. We should of course only document modules that are part of our API (not internal helper modules) and only public attributes. Below is a detailed example: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` This is what the documentation would look like: As we have mentioned we should not mix the `Attributes:` style of documentation with inline documentation of attributes. Notice how `module_level_variable3` appears in a separate block from all the other attributes that were documented. It is even after the TODO section. Choose one approach for your module and stick to it. There are valid reasons to pick one style above the other but don't cross the streams! For instructions on how to document classes, exceptions, etc please consult the [Sphinx Napoleon Extension Guide][#7]. ### Adding New Python Modules When adding a new python binding module to the core of kaolin, a basic .rst will be automatically generated in docs/module/ when running ```make html``` (the automatic generation is not supported yet on windows). The resulting .rst file will look like: ``` .. _moduleName: ============ .. automodule:: :members: :undoc-members: :show-inheritance: ``` If you want the .rst to not be generated, you must add the corresponding python path in [this list][#8]. To add more content such as an introduction the .rst have to be modified following RestructuredText syntax. [#1]: https://www.python.org/dev/peps/pep-0257/ [#2]: https://www.python.org/dev/peps/pep-0287/ [#3]: https://devguide.python.org/documenting/ [#4]: https://docs.python.org/3/library/typing.html [#5]: http://google.github.io/styleguide/pyguide.html#38-comments-and-docstrings [#6]: https://pybind11.readthedocs.io/en/stable/basics.html?highlight=py%3A%3Aarg#keyword-arguments [#7]: https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html [#8]: https://github.com/NVIDIAGameWorks/kaolin/tree/master/docs/kaolin_ext.py#L21 --- ### Index (`docs/index.rst`) Welcome to Kaolin Library Documentation ======================================= .. image:: ../assets/kaolin.png `NVIDIA Kaolin library `_ provides a PyTorch API for working with a variety of 3D representations and includes a growing collection of GPU-optimized operations such as modular differentiable rendering, fast conversions between representations, data loading, 3D checkpoints, differentiable camera API, differentiable lighting with spherical harmonics and spherical gaussians, powerful octree acceleration structure called Structured Point Clouds, interactive 3D visualizer for jupyter notebooks, convenient batched mesh container, quaternion operations, representation-agnostic physics simulation and more. See :ref:`Installation `, :ref:`API Overview ` and :ref:`Tutorials ` to get started! .. toctree:: :titlesonly: :maxdepth: 1 :caption: Tutorials: notes/tutorial_index notes/siggraph2026 notes/siggraph2025 notes/cvpr2025 notes/simplicits notes/conversions notes/volumetric_meshes notes/surface_meshes notes/diff_render notes/differentiable_camera notes/differentiable_lighting notes/pbr_shader notes/visualizer notes/spc_summary notes/quaternions notes/checkpoints .. toctree:: :titlesonly: :maxdepth: 1 :caption: API Reference: modules/kaolin.ops modules/kaolin.math modules/kaolin.metrics modules/kaolin.io modules/kaolin.physics modules/kaolin.render modules/kaolin.rep modules/kaolin.utils modules/kaolin.visualize modules/kaolin.non_commercial modules/kaolin.experimental --- ### Readthedocs Requirements (`docs/readthedocs_requirements.txt`) numpy==1.26.3 scipy==1.14.0 https://download-r2.pytorch.org/whl/cpu/torch-2.10.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl https://download-r2.pytorch.org/whl/cpu/torchvision-0.25.0%2Bcpu-cp312-cp312-manylinux_2_28_x86_64.whl --- ### Modules Kaolin.Experimental.Newton (`docs/modules/kaolin.experimental.newton.rst`) .. _kaolin.experimental.newton: kaolin.experimental.newton ========================== API --- .. automodule:: kaolin.experimental.newton :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Io.Materials (`docs/modules/kaolin.io.materials.rst`) .. _kaolin.io.materials: kaolin.io.materials =================== .. currentmodule:: kaolin.io.materials API --- Functions =================== API --- .. automodule:: kaolin.io.materials :members: :exclude-members: MaterialError, MaterialLoadError, MaterialFileError, MaterialNotFoundError Exceptions ---------- .. autoclass:: MaterialError .. autoclass:: MaterialLoadError .. autoclass:: MaterialFileError .. autoclass:: MaterialNotFoundError --- ### Modules Kaolin.Io.Obj (`docs/modules/kaolin.io.obj.rst`) .. _kaolin.io.obj: kaolin.io.obj ============= .. currentmodule:: kaolin.io.obj API --- Functions --------- .. automodule:: kaolin.io.obj :members: :exclude-members: return_type, ignore_error_handler, skip_error_handler, default_error_handler, create_missing_materials_error_handler, MaterialError, MaterialLoadError, MaterialFileError, MaterialNotFoundError Error Handler ------------- .. autofunction:: ignore_error_handler .. autofunction:: skip_error_handler .. autofunction:: default_error_handler .. autofunction:: create_missing_materials_error_handler --- ### Modules Kaolin.Io (`docs/modules/kaolin.io.rst`) .. _kaolin.io: kaolin.io ========= IO directory contains all the functionalities to interact with data files. :ref:`obj module` and :ref:`usd module ` contains importer to .obj and o, importers / exporters to .usd(a) files, :ref:`dataset module` contains helper features for caching data, and preprocessing whole datasets, and :ref:`materials module` contains Materials definition that should be used throughout Kaolin. .. toctree:: :maxdepth: 2 :titlesonly: kaolin.io.dataset kaolin.io.materials kaolin.io.gltf kaolin.io.obj kaolin.io.off kaolin.io.ply kaolin.io.render kaolin.io.shapenet kaolin.io.usd kaolin.io.modelnet kaolin.io.shrec kaolin.io.utils kaolin.io.mesh kaolin.io.gaussians --- ### Modules Kaolin.Io.Shapenet (`docs/modules/kaolin.io.shapenet.rst`) kaolin.io.shapenet ================== API --- .. automodule:: kaolin.io.shapenet :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Io.Usd (`docs/modules/kaolin.io.usd.rst`) .. _kaolin.io.usd: kaolin.io.usd ============= Universal Scene Description --------------------------- Universal Scene Description (USD) is an open-source 3D scene description file format developed by Pixar and designed to be versatile, extensible and interchangeable between different 3D tools. Single models and animations as well as large organized scenes composed of any number of assets can be defined in USD, making it suitable for organizing entire datasets into interpretable, subsets based on tags, class or other metadata label. Kaolin includes base I/O operations for USD and also leverages this format to export 3D checkpoints. Use kaolin.io.usd to read and write USD files (try :code:`tutorials/usd_kitcheset.py`), and :code:`kaolin.visualize.Timelapse` to export 3D checkpoints (try :code:`tutorials/visualize_main.py`). As a first step to familiarizing yourself with USD, we suggest following this `tutorial `_. More tutorials and documentation can be found `here `_. Viewing USD Files ~~~~~~~~~~~~~~~~~ USD files can be visualized in `NVIDIA Omniverse `_. For easy mesh visualization, see also `Kaolin Jupyter Visualizer recipe for meshes `_. Alternatively, you may use Pixar's USDView which can be obtained by visiting `https://developer.nvidia.com/usd `_ and selecting the corresponding platform under *USD Pre-Built Libraries and Tools*. API --- Functions --------- .. automodule:: kaolin.io.usd :members: :exclude-members: mesh_return_type, MaterialError, MaterialLoadError, MaterialFileError, MaterialWriteError, MaterialNotFoundError, MaterialNotSupportedError, Material, PBRMaterial --- ### Modules Kaolin.Math (`docs/modules/kaolin.math.rst`) .. _kaolin.math: kaolin.math =========== This module contains all the basic math features such as matrix transformations and quaternions. .. toctree:: :maxdepth: 2 :titlesonly: kaolin.math.quat --- ### Modules Kaolin.Metrics (`docs/modules/kaolin.metrics.rst`) .. _kaolin.metrics: kaolin.metrics ============== Metrics are differentiable operators that can be used to compute loss or accuracy. We currently provide an IoU for voxelgrid, sided distance based metrics such as chamfer distance, point_to_mesh_distance and other simple regularization such as uniform_laplacian_smoothing. For tetrahedral mesh, we support the equivolume and AMIPS losses. .. toctree:: :maxdepth: 2 :titlesonly: kaolin.metrics.pointcloud kaolin.metrics.render kaolin.metrics.trianglemesh kaolin.metrics.voxelgrid kaolin.metrics.tetmesh --- ### Modules Kaolin.Metrics.Tetmesh (`docs/modules/kaolin.metrics.tetmesh.rst`) .. _kaolin.metrics.tetmesh: kaolin.metrics.tetmesh ====================== API --- .. automodule:: kaolin.metrics.tetmesh :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Non Commercial (`docs/modules/kaolin.non_commercial.rst`) .. _kaolin.non_commercial: kaolin.non\_commercial ====================== License ------- This submodule contains features under `NSCL license `_ restricted to non commercial usage for research and evaluation purposes. API --- .. automodule:: kaolin.non_commercial :members: :undoc-members: :exclude-members: FlexiCubes :show-inheritance: --- ### Modules Kaolin.Ops.Batch (`docs/modules/kaolin.ops.batch.rst`) .. _kaolin.ops.batch: kaolin.ops.batch ================ .. _batching: Batching -------- Batching data in 3D can be tricky due to the heterogeneous sizes. For instance, point clouds can have different number of points, which means we can't always just concatenate the tensors on a batch axis. Kaolin supports different batching strategies: .. _exact: Exact ~~~~~ Exact batching is the logical representation for homogeneous data. For instance, if you sample the same numbers of points from a batch of meshes, you would just have a single tensor of shape :math:`(\text{batch_size}, \text{number_of_points}, 3)`. .. _padded: Padded ~~~~~~ Heterogeneous tensors are padded to identical dimensions with a constant value so that they can be concatenated on a batch axis. This is similar to padding for the batching of image data of different shapes. .. note:: The last dimension must always be of the size of the element, e.g. 3 for 3D points (element of point clouds) or 1 for a grayscale pixel (element of grayscale textures). For instance, for two textures :math:`T_0` and :math:`T_1` of shape :math:`(32, 32, 3)` and :math:`(64, 16, 3)` the batched tensor will be of shape :math:`(2, max(32, 64), max(32, 16), 3) = (2, 64, 32, 3)` and the padding value will be :math:`0`. :math:`T_0` will be padded on the 1st axis by :math:`32` while :math:`T_1` will be padded on the 2nd axis by :math:`16`. You can also enforce a specific maximum shape (if you want to have a fix memory consumption or use optimization like cudnn algorithm selection). For instance, you can force :math:`T_0` and :math:`T_1` to be batched with a maximum shape of :math:`(128, 128)`, the batched tensor will be of shape :math:`(2, 128, 128, 3)`, :math:`T_0` will be padded on the 1st axis and 2nd axis by 96 and :math:`T_1` will be padded on the 1st axis by :math:`64` and on the 2nd axis by :math:`112`. For more information on how to do padded batching check :func:`kaolin.ops.batch.list_to_padded` Related attributes: ................... .. _padded_shape_per_tensor: * :attr:`shape_per_tensor`: 2D :class:`torch.LongTensor` stores the shape of each sub-tensor except the last dimension in the padded tensor. E.g., in the example above :attr:`shape_per_tensor` would be ``torch.LongTensor([[32, 32], [64, 16]])``. Refer to :func:`kaolin.ops.batch.get_shape_per_tensor` for more information. .. _packed: Packed ~~~~~~ Heterogeneous tensors are reshaped to 2D :math:`(-1, \text{last_dimension})` and concatenated on the first axis. This is similar to packed sentences in NLP. .. note:: The last dimension must always be of the size of the element, e.g. 3 for 3D points (element of point clouds) or 1 for a grayscale pixel (element of grayscale textures). For instance, for two textures :math:`T_0` and :math:`T_1` of shape :math:`(32, 32, 3)` and :math:`(64, 16, 3)` The batched tensor will be of shape :math:`(32 * 32 + 64 * 16, 3)`. :math:`T_0` will be reshaped to :math:`(32 * 32, 3)` and :math:`T_1` will be reshaped :math:`(64 * 16, 3)`, before being concatenated on the first axis. For more information on how to do padded batching check :func:`kaolin.ops.batch.list_to_packed` Related attributes: ................... .. _packed_shape_per_tensor: * :attr:`shape_per_tensor`: 2D :class:`torch.LongTensor` stores the shape of each sub-tensor except the last dimension in the padded tensor. E.g., in the example above :attr:`shape_per_tensor` would be ``torch.LongTensor([[32, 32], [64, 16]])``. Refer to :func:`kaolin.ops.batch.get_shape_per_tensor` for more information. .. _packed_first_idx: * :attr:`first_idx`: 1D :class:`torch.LongTensor` stores the first index of each subtensor and the last index + 1 on the first axis in the packed tensor. E.g., in the example above :attr:`first_idx` would be ``torch.LongTensor([0, 1024, 2048])``. This attribute are used for delimiting each subtensor into the packed tensor, for instance, to slice or index. Refer to :func:`kaolin.ops.batch.get_first_idx` for more information. API --- .. automodule:: kaolin.ops.batch :platform: Windows-x86_64, Linux-x86_64 :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Ops.Conversions (`docs/modules/kaolin.ops.conversions.rst`) .. _kaolin.ops.conversions: kaolin.ops.conversions ====================== API --- .. autoclass:: kaolin.ops.conversions.FlexiCubes :members: :special-members: __call__ .. automodule:: kaolin.ops.conversions :members: :undoc-members: :exclude-members: FlexiCubes :show-inheritance: --- ### Modules Kaolin.Ops.Gaussians (`docs/modules/kaolin.ops.gaussians.rst`) .. _kaolin.ops.gaussians: kaolin.ops.gaussians *********************** `Gaussian Splats `_ is a novel 3D representation consisting of a collection of optimizable 3D Gaussian particles carrying values like alpha and radiance. kaolin does not address specifics regarding the optimization and rendering of this representation, which are already addressed by other frameworks. Rather, it supports additional novel operations which are not handled by other common packages. To maintain compatibility with other frameworks in the broader sense, kaolin makes minimal assumptions about the exact fields tracked by the 3D Gaussians. At the bare minimum, gaussians are expected to keep track of their mean, covariance rotation & scale components, and opacity. Densification ================== The marriage of high quality reconstructions available with `Gaussian Splats `_ and physics simulations now supported by kaolin paves the way to new and exciting interactive opportunities. To improve the accuracy of such simulations, kaolin includes a CUDA based densification module which attempts to sample additional points within the volume of shapes represented with 3D Gaussians. API --- .. automodule:: kaolin.ops.gaussians :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Ops.Mesh (`docs/modules/kaolin.ops.mesh.rst`) .. _kaolin.ops.mesh: kaolin.ops.mesh *********************** A mesh is a 3D object representation consisting of a collection of vertices and polygons. Triangular meshes ================== Triangular meshes comprise of a set of triangles that are connected by their common edges or corners. In Kaolin, they are usually represented as a set of two tensors: * ``vertices``: A :class:`torch.Tensor`, of shape :math:`(\text{batch_size}, \text{num_vertices}, 3)`, contains the vertices coordinates. * ``faces``: A :class:`torch.LongTensor`, of shape :math:`(\text{batch_size}, \text{num_faces}, 3)`, contains the mesh topology, by listing the vertices index for each face. Both tensors can be combined using :func:`kaolin.ops.mesh.index_vertices_by_faces`, to form ``face_vertices``, of shape :math:`(\text{batch_size}, \text{num_faces}, 3, 3)`, listing the vertices coordinate for each face. Tetrahedral meshes ================== A tetrahedron or triangular pyramid is a polyhedron composed of four triangular faces, six straight edges, and four vertex corners. Tetrahedral meshes inside Kaolin are composed of two tensors: * ``vertices``: A :class:`torch.Tensor`, of shape :math:`(\text{batch_size}, \text{num_vertices}, 3)`, contains the vertices coordinates. * ``tet``: A :class:`torch.LongTensor`, of shape :math:`(\text{batch_size}, \text{num_tet}, 4)`, contains the tetrahedral mesh topology, by listing the vertices index for each tetrahedron. Both tensors can be combined, to form ``tet_vertices``, of shape :math:`(\text{batch_size}, \text{num_tet}, 4, 3)`, listing the tetrahedrons vertices coordinates for each face. API --- .. automodule:: kaolin.ops.mesh :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Ops (`docs/modules/kaolin.ops.rst`) .. _kaolin.ops: kaolin.ops ========== Operators are primitive processing functions for batched 3D models (:ref:`meshes`, ::ref:`voxelgrids` and point clouds). Tensor batching operators are in :ref:`kaolin.ops.batch`, conversions of 3D models between different representations are in :ref:`kaolin.ops.conversions`. .. toctree:: :maxdepth: 2 :titlesonly: kaolin.ops.batch kaolin.ops.coords kaolin.ops.conversions kaolin.ops.pointcloud kaolin.ops.gcn kaolin.ops.mesh kaolin.ops.gaussians kaolin.ops.random kaolin.ops.reduction kaolin.ops.spc kaolin.ops.voxelgrid --- ### Modules Kaolin.Ops.Spc (`docs/modules/kaolin.ops.spc.rst`) .. _kaolin.ops.spc: kaolin.ops.spc ############## API --- .. automodule:: kaolin.ops.spc :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Physics.Common (`docs/modules/kaolin.physics.common.rst`) .. _kaolin.physics.common: kaolin.physics.common ======================== API --- .. automodule:: kaolin.physics.common :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Physics.Materials (`docs/modules/kaolin.physics.materials.rst`) .. _kaolin.physics.materials: kaolin.physics.materials ======================== API --- .. automodule:: kaolin.physics.materials :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Physics (`docs/modules/kaolin.physics.rst`) .. _kaolin.physics: kaolin.physics ============== The ``kaolin.physics`` module provides representation-agnostic physics simulation (see the :ref:`Simplicits notes ` for an overview). .. toctree:: :maxdepth: 2 :titlesonly: kaolin.physics.common kaolin.physics.materials kaolin.physics.simplicits kaolin.physics.utils API --- Classes ^^^^^^^ * :ref:`PhysicsPoints ` * :ref:`SkinnedPoints ` * :ref:`SkinnedPhysicsPoints ` * :ref:`SimplicitsObject ` * :ref:`SimplicitsScene ` --- ### Modules Kaolin.Physics.Simplicits.PhysicsPoints (`docs/modules/kaolin.physics.simplicits.PhysicsPoints.rst`) :orphan: .. _kaolin.physics.simplicits.PhysicsPoints: kaolin.physics.simplicits.PhysicsPoints ======================================= API --- .. autoclass:: kaolin.physics.simplicits.PhysicsPoints :special-members: __init__, __len__ :members: :inherited-members: :show-inheritance: --- ### Modules Kaolin.Physics.Simplicits.SimplicitsObject (`docs/modules/kaolin.physics.simplicits.SimplicitsObject.rst`) :orphan: .. _kaolin.physics.simplicits.SimplicitsObject: kaolin.physics.simplicits.SimplicitsObject ========================================== API --- .. autoclass:: kaolin.physics.simplicits.SimplicitsObject :special-members: __init__, __len__ :members: :inherited-members: :show-inheritance: --- ### Modules Kaolin.Physics.Simplicits.SimplicitsScene (`docs/modules/kaolin.physics.simplicits.SimplicitsScene.rst`) :orphan: .. _kaolin.physics.simplicits.SimplicitsScene: kaolin.physics.simplicits.SimplicitsScene ========================================= API --- .. autoclass:: kaolin.physics.simplicits.SimplicitsScene :special-members: __init__ :members: :inherited-members: :show-inheritance: --- ### Modules Kaolin.Physics.Simplicits.SkinnedPhysicsPoints (`docs/modules/kaolin.physics.simplicits.SkinnedPhysicsPoints.rst`) :orphan: .. _kaolin.physics.simplicits.SkinnedPhysicsPoints: kaolin.physics.simplicits.SkinnedPhysicsPoints ============================================== API --- .. autoclass:: kaolin.physics.simplicits.SkinnedPhysicsPoints :special-members: __init__, __len__ :members: :inherited-members: :show-inheritance: --- ### Modules Kaolin.Physics.Simplicits.SkinnedPoints (`docs/modules/kaolin.physics.simplicits.SkinnedPoints.rst`) :orphan: .. _kaolin.physics.simplicits.SkinnedPoints: kaolin.physics.simplicits.SkinnedPoints ======================================= API --- .. autoclass:: kaolin.physics.simplicits.SkinnedPoints :special-members: __init__, __len__ :members: :inherited-members: :show-inheritance: --- ### Modules Kaolin.Physics.Simplicits (`docs/modules/kaolin.physics.simplicits.rst`) .. _kaolin.physics.simplicits: kaolin.physics.simplicits ========================= API --- Classes ^^^^^^^ * :ref:`PhysicsPoints ` * :ref:`SkinnedPoints ` * :ref:`SkinnedPhysicsPoints ` * :ref:`SimplicitsObject ` * :ref:`SimplicitsScene ` Other ^^^^^ .. automodule:: kaolin.physics.simplicits :members: :undoc-members: :show-inheritance: :exclude-members: SimplicitsMLP, SimplicitsRKPM, PhysicsPoints, SkinnedPoints, SkinnedPhysicsPoints, SimplicitsObject, SimplicitsScene --- ### Modules Kaolin.Physics.Utils (`docs/modules/kaolin.physics.utils.rst`) .. _kaolin.physics.utils: kaolin.physics.utils ==================== API --- .. automodule:: kaolin.physics.utils :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Render.Camera.Camera (`docs/modules/kaolin.render.camera.camera.rst`) :orphan: .. _kaolin.render.camera.Camera: kaolin.render.camera.Camera =========================== API --- .. autoclass:: kaolin.render.camera.Camera :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Render.Camera.Camera Extrinsics (`docs/modules/kaolin.render.camera.camera_extrinsics.rst`) :orphan: .. _kaolin.render.camera.CameraExtrinsics: kaolin.render.camera.CameraExtrinsics ===================================== API --- .. autoclass:: kaolin.render.camera.CameraExtrinsics :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Render.Camera.Camera Intrinsics (`docs/modules/kaolin.render.camera.camera_intrinsics.rst`) :orphan: .. _kaolin.render.camera.CameraIntrinsics: kaolin.render.camera.CameraIntrinsics ===================================== API --- .. autoclass:: kaolin.render.camera.CameraIntrinsics :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Render.Camera.Extrinsics Rep (`docs/modules/kaolin.render.camera.extrinsics_rep.rst`) :orphan: .. _kaolin.render.camera.ExtrinsicsRep: kaolin.render.camera.ExtrinsicsRep ================================== API --- .. autoclass:: kaolin.render.camera.ExtrinsicsRep :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Render.Camera.Orthographic Intrinsics (`docs/modules/kaolin.render.camera.orthographic_intrinsics.rst`) :orphan: .. _kaolin.render.camera.OrthographicIntrinsics: kaolin.render.camera.OrthographicIntrinsics =========================================== API --- .. autoclass:: kaolin.render.camera.OrthographicIntrinsics :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Render.Camera.Pinhole Intrinsics (`docs/modules/kaolin.render.camera.pinhole_intrinsics.rst`) :orphan: .. _kaolin.render.camera.PinholeIntrinsics: kaolin.render.camera.PinholeIntrinsics ====================================== API --- .. autoclass:: kaolin.render.camera.PinholeIntrinsics :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Render.Camera (`docs/modules/kaolin.render.camera.rst`) .. _kaolin.render.camera: kaolin.render.camera ==================== Kaolin provides extensive camera API. For an overview, see the :ref:`Camera class docs `. API --- Classes ^^^^^^^ * :ref:`Camera ` * :ref:`CameraExtrinsics ` * :ref:`CameraIntrinsics ` * :ref:`PinholeIntrinsics ` * :ref:`OrthographicIntrinsics ` * :ref:`ExtrinsicsRep ` .. _camera-conversions: Camera Conversions ^^^^^^^^^^^^^^^^^^ Aligning camera conventions across different codebases can take time and care. Kaolin ships with converters between :ref:`kaolin.render.camera.Camera ` and camera conventions in several popular codebases, including: * `nerfstudio gsplat `_ * `INRIA gaussian splats `_ * `polyscope `_ Community contributions are welcome to expand this set. .. autofunction:: kaolin.render.camera.kaolin_camera_to_gsplat_nerfstudio .. autofunction:: kaolin.render.camera.gsplat_nerfstudio_camera_to_kaolin .. autofunction:: kaolin.render.camera.kaolin_camera_to_gsplat_inria .. autofunction:: kaolin.render.camera.gsplat_inria_camera_to_kaolin .. autofunction:: kaolin.render.camera.kaolin_camera_to_polyscope .. autofunction:: kaolin.render.camera.polyscope_camera_to_kaolin Functions ^^^^^^^^^ .. automodule:: kaolin.render.camera :members: :exclude-members: Camera, CameraExtrinsics, CameraIntrinsics, PinholeIntrinsics, OrthographicIntrinsics, ExtrinsicsRep, kaolin_camera_to_gsplat_inria, gsplat_inria_camera_to_kaolin, kaolin_camera_to_gsplat_nerfstudio, gsplat_nerfstudio_camera_to_kaolin, kaolin_camera_to_polyscope, polyscope_camera_to_kaolin :undoc-members: :show-inheritance: --- ### Modules Kaolin.Render.Easy Render (`docs/modules/kaolin.render.easy_render.rst`) kaolin.render.easy\_render ========================== Common Utilities ---------------- .. autofunction:: kaolin.render.easy_render.default_camera .. autofunction:: kaolin.render.easy_render.default_lighting .. autofunction:: kaolin.render.easy_render.default_material Easy Rendering of Meshes ------------------------ .. autoclass:: kaolin.render.easy_render.RenderPass :members: :undoc-members: .. autofunction:: kaolin.render.easy_render.render_mesh Misc Utilities -------------- .. automodule:: kaolin.render.easy_render :members: :member-order: bysource :undoc-members: :show-inheritance: :exclude-members: default_camera, default_lighting, default_material, render_mesh, RenderPass --- ### Modules Kaolin.Render (`docs/modules/kaolin.render.rst`) .. _kaolin.render: This submodule contains all the features related to rendering, including differentiable rasterizer, raytracer, lighting, materials and our Camera API. kaolin.render ============= .. toctree:: :maxdepth: 2 :titlesonly: kaolin.render.camera kaolin.render.easy_render kaolin.render.lighting kaolin.render.materials kaolin.render.mesh kaolin.render.spc --- ### Modules Kaolin.Render.Spc (`docs/modules/kaolin.render.spc.rst`) .. _kaolin.render.spc: kaolin.render.spc ================= API --- .. automodule:: kaolin.render.spc :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Rep.Gaussians (`docs/modules/kaolin.rep.gaussians.rst`) :orphan: .. _kaolin.rep.GaussianSplatModel: kaolin.rep.GaussianSplatModel ============================= API --- .. autoclass:: kaolin.rep.GaussianSplatModel :special-members: __init__, __len__, __getitem__, __setitem__ :members: :inherited-members: :show-inheritance: --- ### Modules Kaolin.Rep.Point Samples (`docs/modules/kaolin.rep.point_samples.rst`) :orphan: .. _kaolin.rep.PointSamples: kaolin.rep.PointSamples ======================= API --- .. autoclass:: kaolin.rep.PointSamples :special-members: __init__, __len__, __getitem__, __setitem__ :members: :inherited-members: :show-inheritance: --- ### Modules Kaolin.Rep (`docs/modules/kaolin.rep.rst`) .. _kaolin.rep: kaolin.rep ========== This module includes higher-level Kaolin classes ("representations"). API --- Classes ^^^^^^^ * :ref:`SurfaceMesh ` * :ref:`Spc ` * :ref:`GaussianSplatModel ` * :ref:`PointSamples ` * :ref:`TensorContainerBase ` Other ^^^^^^^^^ .. automodule:: kaolin.rep :members: :exclude-members: SurfaceMesh, Spc, GaussianSplatModel, PointSamples, TensorContainerBase :undoc-members: :show-inheritance: --- ### Modules Kaolin.Rep.Spc (`docs/modules/kaolin.rep.spc.rst`) :orphan: .. _kaolin.rep.Spc: kaolin.rep.Spc =========================== API --- .. autoclass:: kaolin.rep.Spc :members: :undoc-members: :show-inheritance: --- ### Modules Kaolin.Rep.Surface Mesh (`docs/modules/kaolin.rep.surface_mesh.rst`) :orphan: .. _kaolin.rep.SurfaceMesh: SurfaceMesh =========================== Tutorial -------- For a walk-through of :class:`kaolin.rep.SurfaceMesh` features, see `working_with_meshes.ipynb `_. API --- * :ref:`Overview ` * :ref:`Supported Attributes ` * :ref:`Batching ` * :ref:`Attribute Access and Auto-Computability ` * :ref:`Inspecting and Copying ` * :ref:`Tensor Operations ` .. autoclass:: kaolin.rep.SurfaceMesh :members: :undoc-members: :member-order: bysource :exclude-members: Batching, attribute_info_string, set_batching, to_batched, getattr_batched, cat, vertices, face_vertices, normals, face_normals, vertex_normals, uvs, face_uvs, faces, face_normals_idx, face_uvs_idx, material_assignments, materials, cuda, cpu, to, float_tensors_to, detach, get_attributes, has_attribute, has_or_can_compute_attribute, probably_can_compute_attribute, get_attribute, get_or_compute_attribute, check_sanity, to_string, as_dict, describe_attribute, unset_attributes_return_none, allow_auto_compute, batching, convert_attribute_batching .. _rubric mesh batching: .. rubric:: Supported Batching Strategies ``SurfaceMesh`` can be instantiated with any of the following batching strategies, and supports conversions between batching strategies. Current batching strategy of a ``mesh`` object can be read from ``mesh.batching`` or by running ``print(mesh)``. For example:: mesh = kaolin.io.obj.load_mesh(path) print(mesh) mesh.to_batched() print(mesh) .. autoclass:: kaolin.rep.SurfaceMesh.Batching :members: .. automethod:: attribute_info_string .. automethod:: check_sanity .. automethod:: set_batching .. automethod:: to_batched .. automethod:: getattr_batched .. automethod:: cat .. automethod:: convert_attribute_batching .. _rubric mesh attribute access: .. rubric:: Attribute Access By default, ``SurfaceMesh`` will attempt to auto-compute missing attributes on access. These attributes will be cached, unless their ancestors have ``requires_grad == True``. This behavior of the ``mesh`` object can be changed at construction time (``allow_auto_compute=False``) or by setting ``mesh.allow_auto_compute`` later. In addition to this convenience API, explicit methods for attribute access are also supported. For example, using **convenience API**:: # Caching is enabled by default mesh = kaolin.io.obj.load_mesh(path, with_normals=False) print(mesh) print(mesh.has_attribute('face_normals')) # False fnorm = mesh.face_normals # Auto-computed print(mesh.has_attribute('face_normals')) # True (cached) # Caching is disabled when gradients need to flow mesh = kaolin.io.obj.load_mesh(path, with_normals=False) mesh.vertices.requires_grad = True # causes caching to be off print(mesh.has_attribute('face_normals')) # False fnorm = mesh.face_normals # Auto-computed print(mesh.has_attribute('face_normals')) # False (caching disabled) For example, using **explicit API**:: mesh = kaolin.io.obj.load_mesh(path, with_normals=False) print(mesh.has_attribute('face_normals')) # False fnorm = mesh.get_or_compute_attribute('face_normals', should_cache=False) print(mesh.has_attribute('face_normals')) # False .. automethod:: get_attributes .. automethod:: has_attribute .. automethod:: has_or_can_compute_attribute .. automethod:: probably_can_compute_attribute .. automethod:: get_attribute .. automethod:: get_or_compute_attribute .. _rubric mesh inspecting: .. rubric:: Inspecting and Copying Meshes To make it easier to work with, ``SurfaceMesh`` supports detailed print statements, as well as ``len()``, ``copy()``, ``deepcopy()`` and can be converted to a dictionary. Supported operations:: import copy mesh_copy = copy.copy(mesh) mesh_copy = copy.deepcopy(mesh) batch_size = len(mesh) # Print default attributes print(mesh) # Print more detailed attributes print(mesh.to_string(detailed=True, print_stats=True)) # Print specific attribute print(mesh.describe_attribute('vertices')) .. automethod:: to_string .. automethod:: describe_attribute .. automethod:: as_dict .. _rubric mesh tensor ops: .. rubric:: Tensor Operations Convenience operations for device and type conversions of some or all member tensors. .. automethod:: cuda .. automethod:: cpu .. automethod:: to .. automethod:: float_tensors_to .. automethod:: detach .. rubric:: Other --- ### Modules Kaolin.Rep.Tensor Container (`docs/modules/kaolin.rep.tensor_container.rst`) :orphan: .. _kaolin.rep.TensorContainerBase: kaolin.rep.TensorContainerBase ============================== API --- .. autoclass:: kaolin.rep.TensorContainerBase :special-members: __init__ :members: :inherited-members: :show-inheritance: --- ### Modules Kaolin.Utils (`docs/modules/kaolin.utils.rst`) .. _kaolin.utils: kaolin.utils ============ .. toctree:: :maxdepth: 2 :titlesonly: kaolin.utils.testing kaolin.utils.bundled_data kaolin.utils.env_vars kaolin.utils.log --- ### Modules Kaolin.Visualize (`docs/modules/kaolin.visualize.rst`) .. _kaolin.visualize: kaolin.visualize ================ API --- .. automodule:: kaolin.visualize :members: :inherited-members: :undoc-members: :show-inheritance: --- ### Notes Checkpoints (`docs/notes/checkpoints.rst`) .. _3d_viz: 3D Checkpoint Visualization =========================== .. image:: ../img/koala.jpg Visualizing 3D inputs and outputs of your model during training is an essential diagnostic tool. Kaolin provides a :ref:`simple API to checkpoint` **batches of meshes, pointclouds and voxelgrids**, as well as **colors and textures**, saving them in :ref:`the USD format`, which is efficient at storing time-varying 3D assets, as constant attributes do not need to be replicated in time (e.g. if mesh topology is fixed, face attribute would only be saved once). These checkpoints can then be visualized locally using any USD Viewer or (with fewer features) by launching :ref:`Kaolin Dash3D` on the commandline, allowing remote visualization through a web browser. .. _writing checkpoints: Writing Checkpoints: -------------------- In a common scenario, model performance is visualized for a small evaluation batch. Bootstrap 3D checkpoints in your python training code by configuring a :class:`~kaolin.visualize.Timelapse` object:: import kaolin timelapse = kaolin.visualize.Timelapse(viz_log_dir) The ``viz_log_dir`` is the directory where checkpoints will be saved. Timelapse will create files and subdirectories under this path, so providing a dedicated ``viz_log_dir`` separate from your other logs and configs will help keep things clean. The :class:`~kaolin.visualize.Timelapse` API supports point clouds, voxel grids and meshes, as well as colors and textures. Saving Fixed Data ^^^^^^^^^^^^^^^^^ To save any iteration-independent data, call ``timelapse`` before your training loop without providing an ``iteration`` parameter, e.g.:: timelapse.add_mesh_batch(category='ground_truth', faces_list=face_list, vertices_list=gt_vert_list) timelapse.add_pointcloud_batch(category='input', pointcloud_list=input_pt_clouds) The ``category`` identifies the meaning of the data. In this toy example, the model learns to turn the ``'input'`` pointcloud into the ``'output'`` mesh. Both the ``'ground_truth'`` mesh and the ``'input'`` pointcloud batches are only saved once for easy visual comparison. Saving Time-varying Data ^^^^^^^^^^^^^^^^^^^^^^^^ To checkpoint time-varying data during training, simply call :meth:`~kaolin.visualize.Timelapse.add_mesh_batch`, :meth:`~kaolin.visualize.Timelapse.add_pointcloud_batch` or :meth:`~kaolin.visualize.Timelapse.add_voxelgrid_batch`, for example:: if iteration % checkpoint_interval == 0: timelapse.add_mesh_batch(category='output', iteration=iteration, faces_list=face_list, vertices_list=out_vert_list) .. Tip:: For any data type, only time-varying data needs to be saved at every iteration. E.g., if your output mesh topology is fixed, only save ``faces_list`` once, and then call ``add_mesh_batch`` with only the predicted ``vertices_list``. This will cut down your checkpoint size. Saving Colors and Appearance ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ We are working on adding support for colors and semantic ids to point cloud and voxel grid checkpoints. Mesh API supports multiple time-varying materials by specifying a :class:`kaolin.render.materials.PBRMaterial`. For an example of using materials, see `test_timelapse.py `_. Sample Code ^^^^^^^^^^^ We provide a `script `_ that writes mock checkpoints, which can be run as follows:: python examples/tutorial/visualize_main.py \ --test_objs=path/to/object1.obj,path/to/object2.obj \ --output_dir=path/to/logdir In addition, see :ref:`diff_render` tutorial. .. _file format: Understanding the File Format: ------------------------------ Kaolin :class:`~kaolin.visualize.Timelapse` writes checkpoints using Universal Scene Descriptor (USD) file format (`Documentation `_), developed with wide support for use cases in visual effects, including time-varying data. This allows reducing redundancy in written data across time. After checkpointing with :class:`~kaolin.visualize.Timelapse`, the input ``viz_log_dir`` will contain a similar file structure:: ground_truth/mesh_0.usd ground_truth/mesh_1.usd ground_truth/mesh_... ground_truth/textures input/pointcloud_0.usd input/pointcloud_1.usd input/pointcloud_... output/mesh_0.usd output/mesh_1.usd output/mesh_... output/pointcloud_0.usd output/pointcloud_1.usd output/pointcloud_... output/textures Here, the root folder names correspond to the ``category`` parameter provided to :class:`~kaolin.visualize.Timelapse` functions. Each element of the batch of every type is saved in its own numbered ``.usd`` file. Each USD file can be viewed on its own using any USD viewer, such as `NVIDIA Omniverse View `_, or the whole log directory can be visualized using the tools below. .. Caution:: Timelapse is designed to only save one visualization batch for every category and type. Saving multiple batches without interleaving the data can be accomplished by creating custom categories. .. _dash 3d: Visualizing with Kaolin Dash3D: ------------------------------- .. image:: ../img/dash3d_viz.jpg Viewing USD files locally requires local access to a GPU and to the saved checkpoints, which is not always possible. We are also developing a lightweight ``kaolin-dash3d`` visualizer, which allows visualizing local and remote checkpoints without specialized hardware or applications. This tool is bundled with Kaolin as a command-line utility. To start Dash3D on the machine that stores the checkpoints, run:: kaolin-dash3d --logdir=$TIMELAPSE_DIR --port=8080 The ``logdir`` is the directory :class:`kaolin.visualize.Timelapse` was configured with. This command will launch a web server that will stream geometry to web clients. To connect, simply visit ``http://ip.of.machine:8080`` (or `localhost:8080 `_ if connecting locally or with ssh port forwarding). Try it now: ^^^^^^^^^^^^^ See Dash3D in action by running it on our test samples and visiting `localhost:8080 `_:: kaolin-dash3d --logdir=$KAOLIN_ROOT/tests/samples/timelapse/notexture/ --port=8080 .. Caution:: Dash3d is an experimental feature. It only supports **triangle meshes** and **pointclouds** and cannot visualize colors, ids or textures. The web client was tested the most on `Google Chrome `_. --- ### Notes Conversions (`docs/notes/conversions.rst`) Conversions across 3D Representations ************************************* .. _rep_conversions: .. image:: ../img/conversions.jpg Kaolin Library supports conversions between many 3D representations commonly used for deep learning with a plain PyTorch API. From Mesh ========= * mesh to pointcloud: * :any:`kaolin.ops.mesh.sample_points` * :any:`kaolin.ops.mesh.packed_sample_points` * mesh to voxels: * :any:`kaolin.ops.conversions.trianglemeshes_to_voxelgrids` * mesh to octree (:ref:`Structured Point Clouds ` or SPC): * :any:`kaolin.ops.conversions.unbatched_mesh_to_spc` From Signed Distance Field (SDF) ================================ * SDF to tetrahedral mesh: * :any:`kaolin.ops.conversions.marching_tetrahedra` * SDF to voxelgrid: * :any:`kaolin.ops.conversions.pointclouds_to_voxelgrids` From Point Cloud ================ * point cloud to voxels: * :any:`kaolin.ops.conversions.pointclouds_to_voxelgrids` * point cloud to octree (:ref:`Structured Point Clouds ` or SPC): * :any:`kaolin.ops.conversions.unbatched_pointcloud_to_spc` From Voxels =========== * point cloud to mesh: * :any:`kaolin.ops.conversions.voxelgrids_to_cubic_meshes` * :any:`kaolin.ops.conversions.voxelgrids_to_trianglemeshes` Hybrid Representations ====================== * SDF and voxelgrid to tetrahedral mesh: * :any:`kaolin.ops.conversions.FlexiCubes` (See also :ref:`Volumetric Mesh Optimization `) --- ### Notes Cvpr2025 (`docs/notes/cvpr2025.rst`) CVPR 2025 Tutorial ================== .. raw:: html

3D Deep Learning, Gaussian Splats and Physics Simulation
a Hands-On Lab with NVIDIA Kaolin Library

.. image:: ../img/cvpr2025.jpg .. raw:: html

Attend

Content

Organizers

Run It Yourself

Bring your laptop to this hands-on course from basics to the bleeding edge of 3D research, powered by a dedicated GPU back end for each user and NVIDIA Kaolin Library, a PyTorch-based toolkit for fast, GPU-optimized operations and interactive applications on Gaussian splats, meshes, and volumetric data. This lab will focus on new research allowing interactive physics simulation with contact for 3D Gaussian Splats and meshes, rendered jointly using latest advances (`3DGUT `_, a CVPR oral) and viewed interactively in a Jupyter notebook. Complete coding examples are released together with this tutorial, and are supported by Kaolin Library v0.18.0. .. _cvpr2025 attend: When & Where: ------------- Join us in Nashville, Tennessee on **Wednesday, June 11th, 8AM-noon CDT during CVPR 2025**. See the `Official Schedule `_ for room location. Don't forget to bring your laptop! .. _cvpr2025 content: Content: -------- .. rst-class:: wrap-table .. list-table:: Approximate Tutorial Schedule :widths: 20 35 45 :header-rows: 1 * - Time - Topic - Supporting Materials * - 8-8:15am - Introduction - `Kaolin Documentation `_ * - 8:15-8:30am - Setting up on the cluster - Per-attendee GPU back end reserved by the `Deep Learning Institute `_ * - 8:30-9:15am - Kaolin basics (hands on) - Load and manipulate mesh attributes and PBR materials, control cameras, differentiable rendering, interactive viewing of any render function. See `notebook `__. * - 9:15-10:00am - Introduction to Kaolin Physics simulation (hands on) - Learn theoretical background on representation-agnostic physics simulation method `Simplicits `_, and the :ref:`API ` for its latest implementation in Kaolin, now accelerated with `NVIDIA Warp `_. See `notebook `__. * - 10-10:20am - Break - At least one lecturer will be available to help or answer questions. **New attendees can join at this point.** * - 10:20-11am - Physics simulation for 3D Gaussian Splat objects with collisions, interactive viewing in Jupyter (hands-on) - Directly simulate 3D Gaussian Splat objects and view them interactively within a Jupyter notebook. Enable **collision resolution** between objects. Theoretical primer on collisions. See `notebook `__ (to be updated soon!). Previous version (slower, no collisions) covered in this `video `_. * - 11-11:45am - 3D Gaussians and meshes simulated and rendered together (hands-on) - Add meshes to the mix and simulate meshes and Gaussian-based radiance fields together, interactively view and jointly render them using latest research advances from `NVIDIA 3DGUT `_, a CVPR oral. See `notebook `_ (coming soon!). Discussion of open problems. * - 11:45am-noon - Q & A - Instructors available for help with individual coding examples. .. _cvpr2025 run it yourself: Run It Yourself: ---------------- Install Kaolin v0.18.0 following `our installation instructions `_, and follow our `tutorial notebooks `_ as well as the `Warp notebook `_ .. _cvpr2025 organizers: Organizers: ----------- .. image:: ../img/people/vismay.jpg :height: 0px :width: 0px .. image:: ../img/people/clement.jpg :height: 0px :width: 0px .. image:: ../img/people/or_camel.jpg :height: 0px :width: 0px .. image:: ../img/people/masha_kaleidoscope.jpg :height: 0px :width: 0px .. raw:: html

Clement Fuji Tsang

Clement is a Senior Research Scientist at NVIDIA, leading Kaolin Library development and working on Deep Learning applied to 3D and computer vision. Previously Clement was working on operators fusion and TensorRT integration in MXNet, as well as large scale training of Deep Learning models. His current focus is to develop and share Deep Learning solutions that are efficient and scalable on GPUs for 3D, computer vision and NLP tasks. He has been presenting Kaolin at SIGGRAPH 2022, 2024, and multiple GTCs.

Vismay Modi

Vismay is a Research Scientist at NVIDIA, working on Kaolin's representation-agnostic physics simulator. His focus is to enable interactive simulation of 3D objects in various representations, empowering artists, researchers and engineers to easily prototype, animate and simulate their generated or reconstructed 3D assets. His research goal is to ensure that simulation tools support a diverse set of interactive physics-based phenomena, including elasto-dynamics, muscle activation, joints, cloth, collisions with frictional contact, on any 3D representation, including NeRFs, 3D Gaussian splats, CT scans and more.

Or Perel

Or is a Research Scientist at NVIDIA Toronto AI Lab and a Ph.D. student at the University of Toronto. Previously, he worked at Amazon Rekognition and Autodesk. He obtained his M.Sc. in Computer Sciences from Tel Aviv University, under the supervision of Prof. Daniel Cohen-Or. His research lies at the convergence of computer vision, graphics, and machine learning, with particular interest in AI-driven 3D simulations, including realistic reconstructions and interactive workflows for manipulating them.

Masha (Maria) Shugrina

Masha is a Senior Research Scientist at the NVIDIA Toronto AI Lab, where she manages a subgroup focused on interactive applications of AI and on efforts to accelerate research, including the NVIDIA Kaolin Library. Her core research interest is advancing techniques that integrate AI into the interactive loop. She defended her PhD at the University of Toronto, and Master’s at MIT. She has also worked as a Research Engineer at Adobe and Senior Software Engineer and Tech Lead at Google.

--- ### Notes Diff Render (`docs/notes/diff_render.rst`) .. _diff_render: Differentiable Rendering ************************ .. image:: ../img/clock.gif Differentiable rendering can be used to optimize the underlying 3D properties, like geometry and lighting, by back-propagating gradients from the loss in the image space. Kaolin Library integrates techniques from `DIB-R `_ and `DIB-R++ `_ published techniques, as well as follow up improvements. Many Kaolin utilities also integrate with an alternative `nvdiffrast `_ differentiable rendering utility, also from NVIDIA. We provide an end-to-end basic tutorial using the :mod:`kaolin.render.mesh` functionality for mesh optimization in `examples/tutorial/dibr_tutorial.ipynb `_. See also :ref:`Differentiable Camera `, :ref:`Differentiable Lighting `, and :ref:`Easy PBR Shader `. References ========== * DIB-R: `"Learning to predict 3d objects with an interpolation-based differentiable renderer." `_ Chen, Wenzheng, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. NeurIPS 2019. * `"DIB-R++: learning to predict lighting and material with a hybrid differentiable renderer." `_ Chen, Wenzheng, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. NeurIPS 2021. * Nvdiffrast: `"Modular primitives for high-performance differentiable rendering." `_ Laine, Samuli, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. SIGGRAPH (TOG) 2020. --- ### Notes Differentiable Camera (`docs/notes/differentiable_camera.rst`) Differentiable Camera ********************* .. _differentiable_camera: Tutorials ========= Kaolin github includes a number of recipes for camera manipulation, including: * `cameras_differentiable.py `_: optimize a camera position * `camera_transforms.py `_: using :func:`Camera.transform()` function * `camera_ray_tracing.py `_: how to design a ray generating function using :class:`Camera` objects * `camera_properties.py `_: exposing some the camera attributes and properties * `camera_opengl_shaders.py `_: Using the camera with glumpy * `camera_movement.py `_: Manipulating a camera position and zoom * `camera_init_simple.py `_: Making Camera objects with the flexible :func:`Camera.from_args()` constructor * `camera_init_explicit.py `_: Making :class:`CameraIntrinsics` and :class:`CameraExtrinsics` with all the different constructors available * `camera_coordinate_systems.py `_: Changing coordinate system in a :class:`Camera` object Camera class ============ .. _camera_class: :class:`kaolin.render.camera.Camera` is a one-stop class for all camera related differentiable / non-differentiable transformations. Camera objects are represented by *batched* instances of 2 submodules: - :ref:`CameraExtrinsics `: The extrinsics properties of the camera (position, orientation). These are usually embedded in the view matrix, used to transform vertices from world space to camera space. - :ref:`CameraIntrinsics `: The intrinsics properties of the lens (such as field of view / focal length in the case of pinhole cameras). Intrinsics parameters vary between different lens type, and therefore multiple CameraIntrinsics subclasses exist, to support different types of cameras: pinhole / perspective, orthographic, fisheye, and so forth. For pinehole and orthographic lens, the intrinsics are embedded in a projection matrix. The intrinsics module can be used to transform vertices from camera space to Normalized Device Coordinates. .. note:: To avoid tedious invocation of camera functions through ``camera.extrinsics.someop()`` and ``camera.intrinsics.someop()``, kaolin overrides the ``__get_attributes__`` function to forward any function calls of ``camera.someop()`` to the appropriate extrinsics / intrinsics submodule. The entire pipeline of transformations can be summarized as (ignoring homogeneous coordinates):: World Space Camera View Space V ---CameraExtrinsics.transform()---> V' ---CameraIntrinsics.transform()--- Shape~(B, 3) (view matrix) Shape~(B, 3) | | (linear lens: projection matrix) | + homogeneus -> 3D | V Normalized Device Coordinates (NDC) Shape~(B, 3) When using view / projection matrices, conversion to homogeneous coordinates is required. Alternatively, the `transform()` function takes care of such projections under the hood when needed. How to apply transformations with kaolin's Camera: 1. Linear camera types, such as the commonly used pinhole camera, support the :func:`view_projection_matrix()` method. The returned matrix can be used to transform vertices through pytorch's matrix multiplication, or even be passed to shaders as a uniform. 2. All Cameras are guaranteed to support a general :func:`transform()` function which maps coordinates from world space to Normalized Device Coordinates space. For some lens types which perform non linear transformations, the :func:`view_projection_matrix()` is non-defined. Therefore the camera transformation must be applied through a dedicated function. For linear cameras, :func:`transform()` may use matrices under the hood. 3. Camera parameters may also be queried directly. This is useful when implementing camera params aware code such as ray tracers. How to control kaolin's Camera: - :class:`CameraExtrinsics`: is packed with useful methods for controlling the camera position and orientation: :func:`translate() `, :func:`rotate() `, :func:`move_forward() `, :func:`move_up() `, :func:`move_right() `, :func:`cam_pos() `, :func:`cam_up() `, :func:`cam_forward() `, :func:`cam_up() `. - :class:`CameraIntrinsics`: exposes a lens :func:`zoom() ` operation. The exact functionality depends on the camera type. How to optimize the Camera parameters: - Both :class:`CameraExtrinsics`: and :class:`CameraIntrinsics` maintain :class:`torch.Tensor` buffers of parameters which support pytorch differentiable operations. - Setting ``camera.requires_grad_(True)`` will turn on the optimization mode. - The :func:`gradient_mask` function can be used to mask out gradients of specific Camera parameters. .. note:: :class:`CameraExtrinsics`: supports multiple representions of camera parameters (see: :func:`switch_backend `). Specific representations are better fit for optimization (e.g.: they maintain an orthogonal view matrix). Kaolin will automatically switch to using those representations when gradient flow is enabled For non-differentiable uses, the default representation may provide better speed and numerical accuracy. Other useful camera properties: - Cameras follow pytorch in part, and support arbitrary ``dtype`` and ``device`` types through the :func:`to()`, :func:`cpu()`, :func:`cuda()`, :func:`half()`, :func:`float()`, :func:`double()` methods and :func:`dtype`, :func:`device` properties. - :class:`CameraExtrinsics`: and :class:`CameraIntrinsics`: individually support the :func:`requires_grad` property. - Cameras implement :func:`torch.allclose` for comparing camera parameters under controlled numerical accuracy. The operator ``==`` is reserved for comparison by ref. - Cameras support batching, either through construction, or through the :func:`cat()` method. .. note:: Since kaolin's cameras are batched, the view/projection matrices are of shapes :math:`(\text{num_cameras}, 4, 4)`, and some operations, such as :func:`transform()` may return values as shapes of :math:`(\text{num_cameras}, \text{num_vectors}, 3)`. Concluding remarks on coordinate systems and other confusing conventions: - kaolin's Cameras assume column major matrices, for example, the inverse view matrix (cam2world) is defined as: .. math:: \begin{bmatrix} r1 & u1 & f1 & px \\ r2 & u2 & f2 & py \\ r3 & u3 & f3 & pz \\ 0 & 0 & 0 & 1 \end{bmatrix} This sometimes causes confusion as the view matrix (world2cam) uses a transposed 3x3 submatrix component, which despite this transposition is still column major (observed through the last `t` column): .. math:: \begin{bmatrix} r1 & r2 & r3 & tx \\ u1 & u2 & u3 & ty \\ f1 & f2 & f3 & tz \\ 0 & 0 & 0 & 1 \end{bmatrix} - kaolin's cameras do not assume any specific coordinate system for the camera axes. By default, the right handed cartesian coordinate system is used. Other coordinate systems are supported through :func:`change_coordinate_system() ` and the ``coordinates.py`` module:: Y ^ | |---------> X / Z - kaolin's NDC space is assumed to be left handed (depth goes inwards to the screen). The default range of values is [-1, 1]. CameraExtrinsics class ====================== .. _camera_extrinsics_class: :class:`kaolin.render.camera.CameraExtrinsics` holds the extrinsics parameters of a camera: position and orientation in space. This class maintains the view matrix of camera, used to transform points from world coordinates to camera / eye / view space coordinates. This view matrix maintained by this class is column-major, and can be described by the 4x4 block matrix: .. math:: \begin{bmatrix} R & t \\ 0 & 1 \end{bmatrix} where **R** is a 3x3 rotation matrix and **t** is a 3x1 translation vector for the orientation and position respectively. This class is batched and may hold information from multiple cameras. :class:`CameraExtrinsics` relies on a dynamic representation backend to manage the tradeoff between various choices such as speed, or support for differentiable rigid transformations. Parameters are stored as a single tensor of shape :math:`(\text{num_cameras}, K)`, where K is a representation specific number of parameters. Transformations and matrices returned by this class support differentiable torch operations, which in turn may update the extrinsic parameters of the camera:: convert_to_mat Backend ---- > Extrinsics Representation R View Matrix M Shape (num_cameras, K), Shape (num_cameras, 4, 4) < ---- convert_from_mat .. note:: Unless specified manually with :func:`switch_backend`, kaolin will choose the optimal representation backend depending on the status of ``requires_grad``. .. note:: Users should be aware, but not concerned about the conversion from internal representations to view matrices. kaolin performs these conversions where and if needed. Supported backends: - **"matrix_se3"**\: A flattened view matrix representation, containing the full information of special euclidean transformations (translations and rotations). This representation is quickly converted to a view matrix, but differentiable ops may cause the view matrix to learn an incorrect, non-orthogonal transformation. - **"matrix_6dof_rotation"**\: A compact representation with 6 degrees of freedom, ensuring the view matrix remains orthogonal under optimizations. The conversion to matrix requires a single Gram-Schmidt step. .. seealso:: `On the Continuity of Rotation Representations in Neural Networks, Zhou et al. 2019 `_ Unless stated explicitly, the definition of the camera coordinate system used by this class is up to the choice of the user. Practitioners should be mindful of conventions when pairing the view matrix managed by this class with a projection matrix. CameraIntrinsics class ====================== .. _camera_intrinsics_class: :class:`kaolin.render.camera.CameraIntrinsics` holds the intrinsics parameters of a camera: how it should project from camera space to normalized screen / clip space. The instrinsics are determined by the camera type, meaning parameters may differ according to the lens structure. Typical computer graphics systems commonly assume the intrinsics of a pinhole camera (see: :class:`PinholeIntrinsics` class). One implication is that some camera types do not use a linear projection (i.e: Fisheye lens). There are therefore numerous ways to use CameraIntrinsics subclasses: 1. Access intrinsics parameters directly. This may typically benefit use cases such as ray generators. 2. The :func:`transform()` method is supported by all CameraIntrinsics subclasses, both linear and non-linear transformations, to project vectors from camera space to normalized screen space. This method is implemented using differential pytorch operations. 3. Certain CameraIntrinsics subclasses which perform linear projections, may expose the transformation matrix via dedicated methods. For example, :class:`PinholeIntrinsics` exposes a :func:`projection_matrix()` method. This may typically be useful for rasterization based rendering pipelines (i.e: OpenGL vertex shaders). This class is batched and may hold information from multiple cameras. Parameters are stored as a single tensor of shape :math:`(\text{num_cameras}, K)` where K is the number of intrinsic parameters. currently there are two subclasses of intrinsics: :class:`kaolin.render.camera.OrthographicIntrinsics` and :class:`kaolin.render.camera.PinholeIntrinsics`. Ray Generation ====================== Starting with kaolin 0.16.0, :class:`kaolin.render.camera.Camera` supports ray generation of pinhole and orthographic cameras via :func:`generate_rays() `. The full functional api is included in the :ref:`raygen.py module`, and allows for lower level operations such as :func:`generate_centered_custom_resolution_pixel_coords()` for creating a custom pixel-grid to guide ray-generation. This is useful for supporting ray-tracing with lower resolution image planes, or implementing more advanced effects like ray-jittering. .. image:: ../img/camera_raygen_grid.png API Documentation: ------------------ * Check all the camera classes and functions at the :ref:`API documentation`. --- ### Notes Differentiable Lighting (`docs/notes/differentiable_lighting.rst`) Differentiable Lighting *********************** .. _differentiable_lighting: .. image:: ../img/lighting.gif Kaolin Library implements low-level Spherical Harmonic (``sh``) and Spherical Gaussian (``sg``) lighting, including diffuse and specular lighting effects. These lighting models are different from light sources like point light and spot light, common in computer graphics, and instead approximate a wide range of lighting conditions with a fixed number of optimizable parameters. For example, Spherical Gaussians are used in `"DIB-R++: learning to predict lighting and material with a hybrid differentiable renderer." `_ Chen, Wenzheng, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. NeurIPS 2021. See low-level lighting functions in :py:mod:`kaolin.render.lighting`, as well as :ref:`Easy PBR Shader `, which incorporates SphericalGaussian lighting as an argument into an easy-to-render function for meshes. Tutorials: Lighting Optimization ================================ Optimize Diffuse and Specular Lighting with Spherical Gaussians ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The `examples/tutorial/sg_specular_lighting.ipynb `_ tutorial shows a synthetic example of optimizing for lighting effects represented as Spherical Gaussians using standard PyTorch optimizers and photometric loss. .. raw:: html

Optimizing Diffuse Lighting ^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. image:: ../img/diffuse.png The `examples/tutorial/diffuse_lighting.ipynb `_ tutorial shows how to optimize lighting parameters with spherical gaussians and spherical harmonics. Tutorials: Visualization ======================== Other tutorials include Spherical Gaussian lighting with :any:`kaolin.render.easy_render.render_mesh` function, and show its usage for visualization and synthetic data rendering. For example: * `GLTF Visualizer `_ showing how to render GLTF / glb meshes, such as those from the Objaverse datset, with PBR materials and lighting. * `Easy Mesh Render Tutorial `_ showing how to render meshes from different formats with PBR materials and lighting. --- ### Notes Installation (`docs/notes/installation.rst`) :orphan: .. _installation: Installation ============ Most functions in Kaolin use PyTorch with custom high-performance code in C++ and CUDA. For this reason, full Kaolin functionality is only available for systems with an NVIDIA GPU, supporting CUDA. While it is possible to install Kaolin on other systems, only a fraction of operations will be available for a CPU-only install. Requirements ------------ * Linux, Windows, or macOS (CPU-only) * Python >= 3.9, <= 3.12 * `CUDA `_ >= 10.0 (with 'nvcc' installed) See `CUDA Toolkit Archive `_ to install older version. * torch >= 2.0, <= 2.5.1 Quick Start (Linux, Windows) ---------------------------- | Make sure any of the supported CUDA and torch versions below are pre-installed. | The latest version of Kaolin can be installed with pip: .. code-block:: bash $ pip install kaolin==0.18.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-{TORCH_VER}_cu{CUDA_VER}.html .. Note:: Replace *TORCH_VER* and *CUDA_VER* with any of the compatible options below. .. rst-class:: center-align-center-col +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ | **torch / CUDA** | **cu118** | **cu121** | **cu124** | **cu126** | **cu128** | **cu129** | +==================+===========+===========+===========+===========+===========+===========+ | **torch-2.8.0** | | | | ✓ | ✓ | ✓ | +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ | **torch-2.7.1** | ✓ | | | ✓ | ✓ | | +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ | **torch-2.7.0** | ✓ | | | ✓ | ✓ | | +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ | **torch-2.6.0** | ✓ | | ✓ | ✓ | | | +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ | **torch-2.5.1** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ | **torch-2.5.0** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ | **torch-2.4.1** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ | **torch-2.4.0** | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ | **torch-2.3.1** | ✓ | ✓ | | | | | +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ | **torch-2.3.0** | ✓ | ✓ | | | | | +------------------+-----------+-----------+-----------+-----------+-----------+-----------+ For example, to install kaolin for torch 2.8.0 and CUDA 12.9: .. code-block:: bash $ pip install kaolin==0.18.0 -f https://nvidia-kaolin.s3.us-east-2.amazonaws.com/torch-2.8.0_cu129.html You can check https://nvidia-kaolin.s3.us-east-2.amazonaws.com/index.html to see all the wheels available. Installation from source ------------------------ .. Note:: We recommend installing Kaolin into a virtual environment. For instance to create a new environment with `Anaconda `_: .. code-block:: bash $ conda create --name kaolin python=3.9 $ conda activate kaolin 1. Clone Repository ^^^^^^^^^^^^^^^^^^^ Clone and optionally check out an `official release `_: .. code-block:: bash $ git clone --recursive https://github.com/NVIDIAGameWorks/kaolin $ cd kaolin $ git checkout v0.18.0 # optional 2. Install dependencies ^^^^^^^^^^^^^^^^^^^^^^^ You can install the dependencies running: .. code-block:: bash $ pip install -r tools/build_requirements.txt -r tools/viz_requirements.txt -r tools/requirements.txt 2. Test CUDA ^^^^^^^^^^^^ You can verify that CUDA is properly installed at the desired version with nvcc by running the following: .. code-block:: bash $ nvidia-smi $ nvcc --version 3. Install Pytorch ^^^^^^^^^^^^^^^^^^ Follow `official instructions `_ to install PyTorch of a supported version. Kaolin may be able to work with other PyTorch versions, but we only explicitly test within the version range 2.3.0 to 2.8.0. See below for overriding PyTorch version check during install. Here is how to install the latest Pytorch version supported by Kaolin for cuda 12.8: .. code-block:: bash $ pip install torch==2.8.0 --extra-index-url https://download.pytorch.org/whl/cu128 4. Optional Environment Variables ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ * If trying Kaolin with an unsupported PyTorch version, set: ``export IGNORE_TORCH_VER=1`` * If using heterogeneous GPU setup, set the architectures for which to compile the CUDA code, e.g.: ``export TORCH_CUDA_ARCH_LIST="7.0 7.5"`` * In some setups, there may be a conflict between cub available with cuda install > 11 and ``third_party/cub`` that kaolin includes as a submodule. If conflict occurs or cub is not found, set ``CUB_HOME`` to the cuda one, e.g. typically on Linux: ``export CUB_HOME=/usr/local/cuda-*/include/`` 5. Install Kaolin ^^^^^^^^^^^^^^^^^ .. code-block:: bash $ python setup.py develop .. Note:: Kaolin can be installed without GPU, however, CPU support is limited and many CUDA-only functions will be missing. Testing your installation ------------------------- Run a quick test of your installation and version: .. code-block:: bash $ python -c "import kaolin; print(kaolin.__version__)" Running tests ^^^^^^^^^^^^^ For an exhaustive check, install testing dependencies and run tests as follows: .. code-block:: bash $ pip install -r tools/ci_requirements.txt $ export CI='true' # on Linux $ set CI='true' # on Windows $ pytest --import-mode=importlib -s tests/python/ .. Note:: These tests rely on CUDA operations and will fail if you installed on CPU only, where not all functionality is available. --- ### Notes Overview (`docs/notes/overview.rst`) :orphan: .. _overview: API Overview ============ Below is a summary of Kaolin functionality. Refer to :ref:`tutorial_index` for specific use cases, examples and recipes that use these building blocks. Operators for 3D Data: ^^^^^^^^^^^^^^^^^^^^^^ :ref:`kaolin/ops` contains operators for efficient processing functions of batched 3d models and tensors. We provide, conversions between 3d representations, primitives batching of heterogenenous data, and efficient mainstream functions on meshes and voxelgrids. .. toctree:: :maxdepth: 2 ../modules/kaolin.ops I/O: ^^^^ :ref:`kaolin/io` contains functionality to interact with files. We provide, importers and exporters to popular format such as .obj and .usd, but also utility functions and classes to preprocess and cache datasets with specific transforms. .. toctree:: :maxdepth: 2 ../modules/kaolin.io Metrics: ^^^^^^^^ :ref:`kaolin/metrics` contains functions to compute distance and losses such as point_to_mesh distance, chamfer distance, IoU, or laplacian smoothing. .. toctree:: :maxdepth: 2 ../modules/kaolin.metrics Differentiable Rendering: ^^^^^^^^^^^^^^^^^^^^^^^^^ :ref:`kaolin/render` provide functions related to differentiable rendering, such a DIB-R rasterization, application of camera projection / translation / rotation, lighting, and textures. .. toctree:: :maxdepth: 2 ../modules/kaolin.render 3D Checkpoints and Visualization: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ :ref:`kaolin/visualize` contains utilities for writing 3D checkpoints for visualization. Currently we provide timelapse exporter that can be quickly picked up by the `Omniverse Kaolin App `_. .. toctree:: :maxdepth: 2 ../modules/kaolin.visualize Utilities: ^^^^^^^^^^ :ref:`kaolin/utils` contains utility functions to help development of application or research scripts. We provide functions to display and check informations about tensors, and features to fix seed. .. toctree:: :maxdepth: 2 ../modules/kaolin.utils Non Commercial ^^^^^^^^^^^^^^ :ref:`kaolin/non_commercial` contains features under `NSCL license `_ restricted to non commercial usage for research and evaluation purposes. .. toctree:: :maxdepth: 2 ../modules/kaolin.non_commercial Licenses ======== Most of Kaolin's repository is under `Apache v2.0 license `_, except under :ref:`kaolin/non_commercial` which is under `NSCL license `_ restricted to non commercial usage for research and evaluation purposes. For example, FlexiCubes method is included under :ref:`non_commercial`. Default `kaolin` import includes Apache-licensed components: .. code-block:: python import kaolin The non-commercial components need to be explicitly imported as: .. code-block:: python import kaolin.non_commercial --- ### Notes Pbr Shader (`docs/notes/pbr_shader.rst`) Easy PBR Shader (USD, gltf, obj) ******************************** .. _pbr_shader: .. image:: ../img/easy_render_urchin.jpg .. raw:: html

Differentiable rendering of an in-the-wild Sea Urchin Shell by Drakery from Sketchfab, with no specialized geometry or material preprocessing.

It can be challenging to compose custom differentiable rendering pipelines, especially given the wide range of conventions, and this step can take some ramp-up time for research projects. With Kaolin Library v0.16.0, we are piloting a standard differentiable shader :any:`kaolin.render.easy_render.render_mesh` that works out-of-the-box for many meshes, supporting :ref:`Spherical Gaussian lighting `, and a partial set of physics based rendering (PBR) material maps, including albedo textures, roughness and specular workflows and normal maps. See End-to-end Tutorial: `examples/tutorial/easy_mesh_render.ipynb `_. .. Caution:: There are many differentiable rendering variants, and this particular module has not yet been research-tested for specific applications. We did find it robust across many in-the-wild meshes any welcome your feedback on our `GitHub `_. **Please cite Kaolin if this is useful in your research** (`citation `_). Consistent Convensions for USD, GLTF, GLB, OBJ ============================================== One of the issues with differentiably rendering existing meshes is that input file formats as well as readers into tensors used in deep learning frameworks like PyTorch have a wide range of convensions. Since v0.16.0, Kaolin has started to make the effort to make meshes imported from ``.USD``, ``.GLTF``, ``.obj`` using :any:`kaolin.io.mesh.import_mesh` follow consistent convensions, both for imported geometry, represented as :class:`kaolin.rep.SurfaceMesh` and materials, represented as :class:`kaolin.render.materials.PBRMaterial`. Given staggering number of subtleties, we cannot guarantee consistency or full support for many aspects of these formats, but latest release is a step toward enabling 3D DL researchers to more easily integrate real meshes into their research, rather than creating them from scratch. For example, we would like to enable research into iterative workflows for artists, allowing use of traditional authoring tools in conjunction with new AI algorithms. To load a mesh into PyTorch (see also :ref:`Working With Surface Meshes `):: mesh = kaolin.io.import_mesh('usd_gltf_or_obj_path', triangulate=True).cuda() Standard Differentiable PBR Shader ================================== To render imported (or constructed) triangular meshes (note that you can specify ``triangluate=True`` on import), simply call :any:`kaolin.render.easy_render.render_mesh` to output full composited pass, as well as normals, albedo, specular, alpha and face_id passes. You will first need to create :class:`kaolin.render.camera.Camera` (see :ref:`Differentiable Camera `) or use a default:: # Use default camera, lighting or construct your own camera = kaolin.render.easy_render.default_camera(512).cuda() lighting = kaolinl.render.easy_render.default_lighting().cuda() # Render render_res = kal.render.easy_render.render_mesh(in_cam, mesh, lighting=lighting, **kwargs) .. Tip:: **Both** DIB-R and nvdiffrast :ref:`pipelines ` are supported, by passing in ``backend="cuda"`` and ``backend="nvdiffrast"``, respectively. See :any:`documentation ` for all available options. A Note on the Backends ====================== See :ref:`Differentiable Rendering ` for background on different backends. As noted above, the :any:`easy_render.render_mesh ` supports two back ends. We recommend using `nvdiffrast `_. If this library is installed, a default CUDA based context will be created and reused on the provided GPU device. Note that not all nvdiffrast capabilities (like anti-aliasing) are currently available through the high-level function. Interactive Jupyter Viewing =========================== The rendering function is fully compatible with :ref:`Kaolin Interactive 3D Viewer `, enabling easy inspection of rendering passes right in your debugging notebook: .. raw:: html

Interactively view/debug the output of any custom render function in a Jupyter notebook
(here showing differentiable rendering of a Sea Urchin Shell by Drakery from Sketchfab).

--- ### Notes Quaternions (`docs/notes/quaternions.rst`) Quaternions in PyTorch ********************** .. _quaternions: Quanternions are handy mathematical tools for working with rotations and 3D transformations, and since v0.16.0 Kaolin Library bundles a module of low-level convenience operations with quaternions, targeting common use cases and exposing a plain PyTorch interface to all functions. See :py:mod:`kaolin.math.quat` documentations for a full list of functions and a short python recipe in `examples/recipes/quat/quat_transforms.py `_. --- ### Notes Siggraph2025 (`docs/notes/siggraph2025.rst`) SIGGRAPH 2025 Tutorial ====================== .. raw:: html

Tackling Gaussian Splats, Physics Simulation, and Visualization With NVIDIA Kaolin and Warp Libraries
Hands-On Class

.. video:: ../videos/grut_spray_can_enc.mp4 :height: 0 :width: 0 .. video:: ../videos/flexisim.mp4 :height: 0 :width: 0 .. raw:: html

.. raw:: html

Attend

Content

Organizers

Run It Yourself

Bring your laptop to this hands-on class from basics to the bleeding edge of 3D research, powered by a dedicated GPU back end for each user and NVIDIA Kaolin Library, a PyTorch-based toolkit for fast, GPU-optimized operations and interactive applications on Gaussian splats, meshes, and volumetric data. This lab will focus on new research allowing interactive physics simulation with contact for 3D Gaussians Splats and meshes, along with an introduction to `NVIDIA Warp `_, all viewed interactively in Jupyter notebooks hosted on the `NVIDIA Deep Learning Institute `_. This tutorial will be focused on the newly added collision features to simplicits and will also provide an introduction to Warp. .. _siggraph2025 attend: When & Where: ------------- Join us in Vancouver, Canada on **Sunday, August 10th, 12:30-2pm PST during SIGGRAPH 2025**, Rooms 121-122 in the West Building. Don't forget to bring your laptop! .. _siggraph2025 content: Content: -------- .. rst-class:: wrap-table .. list-table:: Approximate Tutorial Schedule :widths: 20 35 45 :header-rows: 1 * - Time - Topic - Supporting Materials * - 12:30-12:40pm - Introduction and setting up on Cloud - `Kaolin Documentation `_, Per-attendee GPU back end reserved by the `Deep Learning Institute `_ * - 12:40-12:50pm - Kaolin basics (hands on) - Load and manipulate mesh attributes and PBR materials, control cameras, differentiable rendering, interactive viewing of any render function. See `notebook `__. * - 12:50-1:30pm - Introduction to Kaolin Physics simulation (hands on) - Learn theoretical background on representation-agnostic physics simulation method `Simplicits `_, and the :ref:`API ` for its latest implementation in Kaolin, now accelerated with `NVIDIA Warp `__. See `notebook `__. * - 1:30-1:35pm - 3D Gaussians and meshes simulated and rendered together (overview). - Add meshes to the mix and simulate meshes and Gaussian-based radiance fields together, interactively view and jointly render them using latest research advances from `NVIDIA 3DGUT `_, a CVPR oral. See `notebook `__. * - 1:35-1:55pm - Introduction to Warp and FlexiSim (hands on) - Make and debug Warp kernel for differentiable physics, see `notebook `__. See `FlexiSim page `_. * - 1:55-2:00pm - Conclusion and Q & A - Instructors available for help with individual coding examples. .. _siggraph2025 run it yourself: Run It Yourself: ---------------- Install Kaolin v0.18.0 following `our installation instructions `_, and follow our `tutorial notebooks `_ as well as the `Warp notebook `_ .. _siggraph2025 organizers: Organizers: ----------- .. image:: ../img/people/vismay.jpg :height: 0px :width: 0px .. image:: ../img/people/clement.jpg :height: 0px :width: 0px .. image:: ../img/people/gilles.jpg :height: 0px :width: 0px .. image:: ../img/people/masha_kaleidoscope.jpg :height: 0px :width: 0px .. raw:: html