Fisheye-GS: Lightweight and Extensible Gaussian Splatting Module for Fisheye Cameras

Zimu Liao,Siyan Chen,Rong Fu,Yi Wang,Zhongling Su,Hao Luo,Li Ma,Linning Xu,Bo Dai,Hengjie Li,Zhilin Pei,Xingcheng Zhang

2024-09-11

Abstract:Recently, 3D Gaussian Splatting (3DGS) has garnered attention for its high fidelity and real-time rendering. However, adapting 3DGS to different camera models, particularly fisheye lenses, poses challenges due to the unique 3D to 2D projection calculation. Additionally, there are inefficiencies in the tile-based splatting, especially for the extreme curvature and wide field of view of fisheye lenses, which are crucial for its broader real-life applications. To tackle these challenges, we introduce Fisheye-GS.This innovative method recalculates the projection transformation and its gradients for fisheye cameras. Our approach can be seamlessly integrated as a module into other efficient 3D rendering methods, emphasizing its extensibility, lightweight nature, and modular design. Since we only modified the projection component, it can also be easily adapted for use with different camera models. Compared to methods that train after undistortion, our approach demonstrates a clear improvement in visual quality.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to adapt 3D Gaussian Splatting (3DGS) technology to the fisheye camera model in order to overcome the challenges that traditional 3DGS encounters when dealing with the unique curvature and wide field - of - view of fisheye lenses. Specifically:

1. **Recalculation of Projection Transformation**: The images of fisheye lenses have significant radial distortion, which makes the traditional perspective projection method no longer applicable. The paper proposes an equidistant projection model for fisheye lenses and recalculates the projection transformation and its gradient to ensure image quality and accuracy under extreme curvature and wide field - of - view.

2. **Light - weight Modular Design**: The method proposed in the paper (Fisheye - GS) can be seamlessly integrated as a light - weight module into other efficient 3D rendering methods, emphasizing its scalability, light - weight characteristics and modular design. By only modifying the projection component, Fisheye - GS can easily adapt to different camera models.

3. **Improvement of Visual Quality**: Compared with the method of first performing distortion - removal processing on the image and then training, Fisheye - GS shows obvious advantages in visual quality. Experimental results show that Fisheye - GS can generate high - quality, distortion - free images in various 3D graphics applications.

### Formula Summary

1. **Equidistant Projection Model**:
- Relationship between projection distance \(r_d\) and incident angle \(\theta\):
\[
r_d = f\theta
\]
where \(f\) is the focal length of the camera.

- Calculation of pixel coordinates \((x_p, y_p)\):
\[
x_p = c_x + f_x\theta\frac{x_c}{l_z}, \quad y_p = c_y + f_y\theta\frac{y_c}{l_z}
\]
where \(l_z=\sqrt{x_c^2 + y_c^2}\), \((c_x, c_y)\) are the pixel coordinates of the optical center.

2. **Jacobian Matrix of Projection Transformation**:
- Calculation of Jacobian matrix \(J_\phi\):
\[
J_\phi=\frac{\partial\mu_p}{\partial\mu_c}=\begin{bmatrix}
\frac{f_x x_c^2}{z_c l_z^2}+\frac{f_x y_c^2}{z_c l_z^2}\theta\frac{x_c y_c}{l_z^3}&\frac{f_x x_c y_c}{z_c l_z^2}-\frac{f_x y_c^2}{z_c l_z^2}\theta\frac{x_c y_c}{l_z^3}&-\frac{f_x x_c}{l_z^2}\\
\frac{f_y x_c y_c}{z_c l_z^2}-\frac{f_y x_c^2}{z_c l_z^2}\theta\frac{x_c y_c}{l_z^3}&\frac{f_y y_c^2}{z_c l_z^2}+\frac{f_y x_c^2}{z_c l_z^2}\theta\frac{x_c y_c}{l_z^3}&-\frac{f_y y_c}{l_z^2}
\end{bmatrix}
\]

3. **Calculation of Projection Covariance**:
- Calculation of projection covariance \(\Sigma_p\):
\[
\Sigma_p = T\Sigma T^T
\]
where \(T = JW\).

4. **Gradient Calculation**:
- Chain rule of gradient propagation:
\[