Abstract:3D semantic scene completion (SSC) is an ill-posed perception task that requires inferring a dense 3D scene from limited observations. Previous camera-based methods struggle to predict accurate semantic scenes due to inherent geometric ambiguity and incomplete observations. In this paper, we resort to stereo matching technique and bird's-eye-view (BEV) representation learning to address such issues in SSC. Complementary to each other, stereo matching mitigates geometric ambiguity with epipolar constraint while BEV representation enhances the hallucination ability for invisible regions with global semantic context. However, due to the inherent representation gap between stereo geometry and BEV features, it is non-trivial to bridge them for dense prediction task of SSC. Therefore, we further develop a unified occupancy-based framework dubbed BRGScene, which effectively bridges these two representations with dense 3D volumes for reliable semantic scene completion. Specifically, we design a novel Mutual Interactive Ensemble (MIE) block for pixel-level reliable aggregation of stereo geometry and BEV features. Within the MIE block, a Bi-directional Reliable Interaction (BRI) module, enhanced with confidence re-weighting, is employed to encourage fine-grained interaction through mutual guidance. Besides, a Dual Volume Ensemble (DVE) module is introduced to facilitate complementary aggregation through channel-wise recalibration and multi-group voting. Our method outperforms all published camera-based methods on SemanticKITTI for semantic scene completion. Our code is available on <a class="link-external link-https" href="https://github.com/Arlo0o/StereoScene" rel="external noopener nofollow">this https URL</a>.

Learning Robust Image-Based Rendering on Sparse Scene Geometry Via Depth Completion

Scalable Image-Based Indoor Scene Rendering with Reflections

Interactive arbitrary-view rendering system using fast IBR and depth correction

CRF-Based Reconstruction from Narrow-Baseline Image Sequences.

Depth-image-based Rendering with Spatial and Temporal Texture Synthesis for 3DTV

Arbitrary view generation based on DIBR

Robust Bayesian Scene Reconstruction by Leveraging Retrieval-Augmented Priors

SIRe-IR: Inverse Rendering for BRDF Reconstruction with Shadow and Illumination Removal in High-Illuminance Scenes

Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images

PBIR-NIE: Glossy Object Capture under Non-Distant Lighting

Robust Geometry-Preserving Depth Estimation Using Differentiable Rendering

Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Image

Multi-view Inverse Rendering for Large-scale Real-world Indoor Scenes

RigNet++: Efficient Repetitive Image Guided Network for Depth Completion

GS-IR: 3D Gaussian Splatting for Inverse Rendering

Efficient Depth Completion Using Learned Bases

Incorporating dense metric depth into neural 3D representations for view synthesis and relighting

GBR: Generative Bundle Refinement for High-fidelity Gaussian Splatting and Meshing

ProbIBR: Fast Image-Based Rendering with Learned Probability-Guided Sampling

Bridging Stereo Geometry and BEV Representation with Reliable Mutual Interaction for Semantic Scene Completion

S$^3$-NeRF: Neural Reflectance Field from Shading and Shadow under a Single Viewpoint