Abstract:3D point clouds are rich in geometric structure information, while 2D images contain important and continuous texture information. Combining 2D information to achieve better 3D semantic segmentation has become mainstream in 3D scene understanding. Albeit the success, it still remains elusive how to fuse and process the cross-dimensional features from these two distinct spaces. Existing state-of-the-art usually exploit bidirectional projection methods to align the cross-dimensional features and realize both 2D & 3D semantic segmentation tasks. However, to enable bidirectional mapping, this framework often requires a symmetrical 2D-3D network structure, thus limiting the network's flexibility. Meanwhile, such dual-task settings may distract the network easily and lead to over-fitting in the 3D segmentation task. As limited by the network's inflexibility, fused features can only pass through a decoder network, which affects model performance due to insufficient depth. To alleviate these drawbacks, in this paper, we argue that despite its simplicity, projecting unidirectionally multi-view 2D deep semantic features into the 3D space aligned with 3D deep semantic features could lead to better feature fusion. On the one hand, the unidirectional projection enforces our model focused more on the core task, i.e., 3D segmentation; on the other hand, unlocking the bidirectional to unidirectional projection enables a deeper cross-domain semantic alignment and enjoys the flexibility to fuse better and complicated features from very different spaces. In joint 2D-3D approaches, our proposed method achieves superior performance on the ScanNetv2 benchmark for 3D semantic segmentation.

Fused Network for View Synthesis.

Asymmetric Bidirectional View Synthesis for Free Viewpoint and Three-Dimensional Video

Recurrent Volume-Based 3-D Feature Fusion for Real-Time Multiview Object Pose Estimation.

Synthesis-Aware Region-Based 3D Video Coding.

Improved Multi-View Depth Estimation For View Synthesis In 3d Video Coding

Depth-image Based View Synthesis for Threedimensional Television

Temporal3D: 2D-to-3d Video Conversion Network with Multi-frame Fusion

A Novel Method of Multi-view Virtual Image Synthesis for Auto-stereoscopic Display

Stereoscopic to Multi-view Video Conversion System FPGA Implementation

Learning-Based Stereoscopic View Synthesis with Cascaded Deep Neural Networks

FusionDepth: Complement Self-Supervised Monocular Depth Estimation with Cost Volume

Real-Time Free Viewpoint Video Synthesis System Based on DIBR and A Depth Estimation Network

Towards Deeper and Better Multi-view Feature Fusion for 3D Semantic Segmentation

Virtual-Real Fusion with Dynamic Scene from Videos

ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction

Stereoscopic view synthesis with progressive structure reconstruction and scene constraints

Real-Time Unsupervised Multi-View Depth Estimation Network for Virtual View Synthesis

Scale-Consistent Fusion: From Heterogeneous Local Sampling to Global Immersive Rendering

Virtual Multi-view Fusion for 3D Semantic Segmentation

StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videos