Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections

Dongbin Zhang,Chuming Wang,Weitao Wang,Peihao Li,Minghan Qin,Haoqian Wang

2024-07-14

Abstract:Novel view synthesis from unconstrained in-the-wild images remains a meaningful but challenging task. The photometric variation and transient occluders in those unconstrained images make it difficult to reconstruct the original scene accurately. Previous approaches tackle the problem by introducing a global appearance feature in Neural Radiance Fields (NeRF). However, in the real world, the unique appearance of each tiny point in a scene is determined by its independent intrinsic material attributes and the varying environmental impacts it receives. Inspired by this fact, we propose Gaussian in the wild (GS-W), a method that uses 3D Gaussian points to reconstruct the scene and introduces separated intrinsic and dynamic appearance feature for each point, capturing the unchanged scene appearance along with dynamic variation like illumination and weather. Additionally, an adaptive sampling strategy is presented to allow each Gaussian point to focus on the local and detailed information more effectively. We also reduce the impact of transient occluders using a 2D visibility map. More experiments have demonstrated better reconstruction quality and details of GS-W compared to NeRF-based methods, with a faster rendering speed. Video results and code are available at <a class="link-external link-https" href="https://eastbeanzhang.github.io/GS-W/" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of achieving high-quality novel view synthesis from unconstrained collections of wild images. Specifically, the paper focuses on how to reconstruct the original scene and synthesize new views from images taken at different times and settings, which include transient occluders such as pedestrians or vehicles and dynamic appearance changes like varying skies, weather, and lighting. These images typically exhibit significant photometric variations and transient occlusions, making accurate scene reconstruction very challenging.

The paper proposes a method called "Gaussian in the Wild" (GS-W), which represents the scene using 3D Gaussian points and introduces independent intrinsic and dynamic appearance features for each point to capture the invariant appearance of the scene as well as dynamic changes such as lighting and weather. Additionally, the method introduces an adaptive sampling strategy that allows each Gaussian point to more effectively focus on local and detailed information, while using 2D visibility maps to reduce the impact of transient occluders. Experimental results show that compared to NeRF-based methods, GS-W not only performs better in terms of reconstruction quality and detail but also improves rendering speed by over 1000 times.