Rasterization using 3d gaussians

[fwilliams]

Summary

This PR adds a dense, differentiable rasterize-from-world path for Gaussian splatting that renders images directly from world-space 3D Gaussians (means/quats/log-scales) instead of relying only on 2D projected conics. It supports UT-based projection for OpenCV camera models, integrates into the GaussianSplat3d public API, and includes extensive regression/gradient tests.

Key changes

• New public API

  • GaussianSplat3d.render_images_from_world(...) (Python + C++ bindings)
  • Supports CameraModel + RollingShutterType and packed OpenCV distortion coefficients.

• New kernels + autograd (dense-only)

  • CUDA forward/backward for rasterization from world-space 3D Gaussians.
  • Autograd wrapper enforces correct backward arity (one grad per forward input).

• Camera model consolidation + reuse

  • Shared CameraModel / RollingShutterType enums (C++) and Python IntEnum wrappers in fvdb.enums.
  • Shared OpenCV distortion math and pose interpolation helpers used across UT + rasterize-from-world.

• Masks + backgrounds parity

  • Optional tile masks and per-camera backgrounds with behavior aligned to the classic rasterizer.
  • Deadlock fixes: block-wide early exits for masked/no-intersection edge tiles in both classic and from-world codepaths.

• Docs + naming cleanup

  • Expanded Python docstrings for render_images_from_world.
  • Removed deprecated aliases and aligned naming (no *_3dgs suffix in the public API).

Test plan

• C++
  • ./build.sh ctest
• Python
  • Run from /tmp so Python imports the installed extension:
    • conda run -n fvdb python -m pytest -q --rootdir /path/to/fvdb-core/tests /path/to/fvdb-core/tests

Notes / follow-ups

• Rasterize-from-world is currently CUDA-only and dense-only (no jagged/sparse output).
• CameraModel / RollingShutterType are kept explicit in some dispatch APIs; folding them into RenderSettings could be a follow-up if we want a single transport object everywhere.

[Copilot]

Pull request overview

This pull request adds 3D Gaussian Splatting (3DGS) rasterization capability that enables rendering directly from world-space Gaussians rather than from 2D projections. This allows geometry gradients to flow through the rasterization step, which is particularly useful for non-pinhole camera models with distortion.

Changes:

• Adds new render_images_from_world_3dgs method that performs ray-ellipsoid rasterization directly in world space
• Introduces CameraModel and RollingShutterType enums for flexible camera projection support
• Implements CUDA kernels for forward and backward passes of the 3DGS rasterizer with support for multiple camera models (pinhole, orthographic, and OpenCV distortion variants)

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/fvdb/detail/ops/gsplat/GaussianCameraModels.h	Defines camera model and rolling shutter enums (contains critical duplication bug)
src/fvdb/detail/ops/gsplat/GaussianRasterizeFromWorld3DGS.cuh	Shared utilities for ray generation, distortion, and coordinate transformations
src/fvdb/detail/ops/gsplat/GaussianRasterizeFromWorld3DGSForward.cu	CUDA kernel for forward rasterization pass
src/fvdb/detail/ops/gsplat/GaussianRasterizeFromWorld3DGSBackward.cu	CUDA kernel for backward gradient computation
src/fvdb/detail/autograd/GaussianRasterizeFromWorld3DGS.{h,cpp}	PyTorch autograd wrapper for the rasterization operation
src/fvdb/GaussianSplat3d.{h,cpp}	Main API implementation routing to appropriate projection and rasterization paths
src/python/GaussianSplatBinding.cpp	Python bindings for the new enums and method
fvdb/gaussian_splatting.py	Python wrapper for the new rendering method
fvdb/enums.py	Python enum definitions for camera models and rolling shutter types
fvdb/_fvdb_cpp.pyi, fvdb/init.pyi, fvdb/init.py	Type stubs and exports for the new functionality
tests/unit/test_rasterize_world3dgs.py	Unit tests for gradient correctness and finite difference validation
src/CMakeLists.txt	Build configuration for new source files
src/fvdb/detail/ops/gsplat/GaussianProjectionUT.h	Refactored to use shared enum definitions

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 6 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[swahtz]

Could we get a description of the PR please?

[fwilliams]

@swahtz : updated:

• I refactored the rasterize-from-world dispatch APIs to take a RenderSettings object (for image size/origin/tileSize), matching the other rasterize utilities.
• I kept CameraModel and RollingShutterType as explicit parameters for now.

Folding camera model + rolling-shutter policy into RenderSettings would be a broader API change since RenderSettings is used across several rasterize/projection utilities that don’t need UT/rolling-shutter camera semantics. Happy to do that as a follow-up if we want to standardize camera transport further.

[harrism]

There are a lot of things that are inconsistent with current rasterization implementation. It would be good to harmonize that. My main concern is the deadlock test. Is it necessary? Was this discovered in code in the wild? Do we need to test for deadlock both in Python and C++? I don't think so.

[swahtz]

There's quite a few inconsistencies in this rasterization kernel compared to the precedents set by the other rasterization kernels (building a RasterizeCommonArgs, variable naming and documentation of functions). I'd prefer to see those specific aspects addressed to be more maintainable and other comments could be left for later issues.

[swahtz]

Thank you for making some of the requested changes.

I think in the next steps, we should first refactor this to match the patterns of other rasterization functions (RasterizeCommonArgs, etc.). After that I think we should discuss with @harrism refactoring of common functionality. There's a few changes I've also wanted to be make to the general pipeline but have held off such as separating Camera definition settings; Rasterizer method-agnostic settings and Rasterizer-specific settings in a semantically useful way as well as build in the ability use other shading evaluation methods and not just force all rasterization to use the SH evaluation path.