Abstract:Representing underwater 3D scenes is a valuable yet complex task, as attenuation and scattering effects during underwater imaging significantly couple the information of the objects and the water. This coupling presents a significant challenge for existing methods in effectively representing both the objects and the water medium simultaneously. To address this challenge, we propose Aquatic-GS, a hybrid 3D representation approach for underwater scenes that effectively represents both the objects and the water medium. Specifically, we construct a Neural Water Field (NWF) to implicitly model the water parameters, while extending the latest 3D Gaussian Splatting (3DGS) to model the objects explicitly. Both components are integrated through a physics-based underwater image formation model to represent complex underwater scenes. Moreover, to construct more precise scene geometry and details, we design a Depth-Guided Optimization (DGO) mechanism that uses a pseudo-depth map as auxiliary guidance. After optimization, Aquatic-GS enables the rendering of novel underwater viewpoints and supports restoring the true appearance of underwater scenes, as if the water medium were absent. Extensive experiments on both simulated and real-world datasets demonstrate that Aquatic-GS surpasses state-of-the-art underwater 3D representation methods, achieving better rendering quality and real-time rendering performance with a 410x increase in speed. Furthermore, regarding underwater image restoration, Aquatic-GS outperforms representative dewatering methods in color correction, detail recovery, and stability. Our models, code, and datasets can be accessed at <a class="link-external link-https" href="https://aquaticgs.github.io" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
### Problems the Paper Attempts to Solve
This paper aims to address the complex challenges in underwater 3D scene representation. Specifically, the attenuation and scattering effects present during underwater imaging significantly couple object information with water medium information, making it extremely difficult for existing methods to effectively represent both objects and the water medium simultaneously. The paper proposes a hybrid 3D representation method named Aquatic-GS to overcome this challenge.
### Background and Problem Description
1. **Complexity of Underwater Imaging**:
- **Attenuation**: When light reflects from an object to the camera, it gets absorbed by the water, especially red light, which is more easily absorbed than blue and green light, causing color distortion in the captured images.
- **Scattering**: Particularly backscattering, when suspended particles in the water scatter ambient light to the camera, leading to image blurring and reduced contrast.
- **Distance Dependence**: These effects become more pronounced as the distance between the object and the camera increases, further exacerbating the coupling of information.
2. **Limitations of Existing Methods**:
- **NeRF (Neural Radiance Fields)**: While capable of efficiently encoding complex scenes, its implicit modeling strategy often results in blurred details, high geometric noise, and high computational costs when representing objects.
- **3DGS (3D Gaussian Splatting)**: Although it performs well in representing opaque objects, it encounters inevitable artifacts and inaccurate geometric structures when dealing with rich semi-transparent water media.
### Solution
To effectively represent objects and water media in underwater scenes, the paper proposes Aquatic-GS, a hybrid 3D representation method combining explicit representation and implicit modeling. Specifically:
1. **Neural Water Field**:
- Implicitly models the distribution of water parameters, considering the spatial non-uniformity of water parameters in real environments.
- Uses neural networks to learn the spatial distribution of water parameters, including attenuation coefficients and backscattering coefficients.
2. **3D Gaussian Splatting**:
- Explicitly models the appearance and geometric structure of the scene, using a set of learnable 3D Gaussian primitives to capture the true appearance and geometric information of the scene.
- Integrates water parameters and object information through a physics-based underwater image formation model (UIF model) to generate complex underwater scenes.
3. **Depth-Guided Optimization**:
- Introduces pseudo-depth maps as auxiliary guidance to optimize the geometric structure and distant details of the scene.
- Includes four specifically designed loss functions: transmittance regularization, depth variance minimization regularization, coarse depth supervision, and block frequency domain supervision to ensure more accurate scene representation.
### Experimental Results
The paper evaluates the performance of Aquatic-GS on multiple real-world and simulated datasets, showing that:
- In the task of underwater novel view synthesis, Aquatic-GS outperforms existing NeRF-based methods, achieving higher rendering quality and faster real-time rendering speed.
- In the task of underwater image restoration, Aquatic-GS surpasses representative dewatering methods in terms of color correction, detail recovery, and stability.
### Conclusion
By combining the advantages of explicit representation and implicit modeling, Aquatic-GS effectively addresses the key challenges in underwater scene representation, providing a more comprehensive and accurate method for underwater scene representation.