Gaussian Rotation

[crocis]

I have noticed that when you compute the Gaussian rotation with GSRegresser, you do not transform it with the camera extrinsics. Is it an error or am I wrong?

[ShunyuanZheng]

That's right, we do not regress the correspondence between camera extrinsics and rotation. However, in our experiments, it is better to regress such an anisotropic rotation pattern than to fix it with an isotropic one (fix the rotation to [1, 0, 0, 0] and regress a single-dimensional Gaussian scale) as in some other works.

[crocis]

Does it mean that you take the Gaussian rotation and apply the extrinsics to it so that the Gaussian is oriented based on the input view?

[ShunyuanZheng]

Actually, I do not fully understand what 'apply the extrinsics to Gaussians' means. The rotation is regressed by the network and directly sent to the Gaussian Rasterizer afterwards.

[crocis]

Let me explain what I mean. Given an input view, GPS-Gaussian computes Gaussians with rotations for each of them. If you move the input view, GPS-Gaussian must compute different Gaussian rotations according to the new position and orientation of the input view. Does GPS-Gaussian modify the Gaussian rotations when the input view moves? I have not found the code that does that.

[ShunyuanZheng]

Do you mean moving the camera? Since the rotation is only related to the input 2D images, the moving input views should lead to the re-computation of rotations. But the computation is only related to the input images in GPS-Gaussian, which means it will not be related to the new position and orientation.

[crocis]

Therefore, if I move the RGBD camera, the Gaussian orientations won't change. Right?

[ShunyuanZheng]

Theoretically, it might be the case.

[crocis]

From what I understand the Gaussian rotations computed by GPS-Gaussian are defined in the coordinate space of the input camera used to compute them. If you want the Gaussian rotations in the world coordinate space, you need to transform them according to the pose of the input camera.