Abstract:2D-3D joint learning is essential and effective for fundamental 3D vision tasks, such as 3D semantic segmentation, due to the complementary information these two visual modalities contain. Most current 3D scene semantic segmentation methods process 2D images "as they are", i.e., only real captured 2D images are used. However, such captured 2D images may be redundant, with abundant occlusion and/or limited field of view (FoV), leading to poor performance for the current methods involving 2D inputs. In this paper, we propose a general learning framework for joint 2D-3D scene understanding by selecting informative virtual 2D views of the underlying 3D scene. We then feed both the 3D geometry and the generated virtual 2D views into any joint 2D-3D-input or pure 3D-input based deep neural models for improving 3D scene understanding. Specifically, we generate virtual 2D views based on an information score map learned from the current 3D scene semantic segmentation results. To achieve this, we formalize the learning of the information score map as a deep reinforcement learning process, which rewards good predictions using a deep neural network. To obtain a compact set of virtual 2D views that jointly cover informative surfaces of the 3D scene as much as possible, we further propose an efficient greedy virtual view coverage strategy in the normal-sensitive 6D space, including 3-dimensional point coordinates and 3-dimensional normal. We have validated our proposed framework for various joint 2D-3D-input or pure 3D-input based deep neural models on two real-world 3D scene datasets, i.e., ScanNet v2 and S3DIS, and the results demonstrate that our method obtains a consistent gain over baseline models and achieves new top accuracy for joint 2D and 3D scene semantic segmentation. Code is available at https://github.com/smy-THU/VirtualViewSelection.

Variational Inference for Scalable 3D Object-centric Learning

Variable-Viewpoint Representations for 3D Object Recognition

Dynamic Scene Understanding through Object-Centric Voxelization and Neural Rendering

3D Object Recognition By Corresponding and Quantizing Neural 3D Scene Representations

Incremental Joint Learning of Depth, Pose and Implicit Scene Representation on Monocular Camera in Large-scale Scenes

Learning Disentangled Representation for Multi-View 3D Object Recognition.

Learning 3D object-centric representation through prediction

DynaVol: Unsupervised Learning for Dynamic Scenes through Object-Centric Voxelization

Learning Virtual View Selection for 3D Scene Semantic Segmentation

Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose

AUTO3D: Novel view synthesis through unsupervisely learned variational viewpoint and global 3D representation

Slot-guided Volumetric Object Radiance Fields

Equivariant Spatio-Temporal Self-Supervision for LiDAR Object Detection

Unsupervised Object-Centric Learning from Multiple Unspecified Viewpoints

3D Open-Vocabulary Panoptic Segmentation with 2D-3D Vision-Language Distillation

Scaled Inverse Graphics: Efficiently Learning Large Sets of 3D Scenes

Unsupervised Discovery of Object-Centric Neural Fields

Learning Canonical 3D Object Representation for Fine-Grained Recognition

Self-Supervised Multi-View Learning via Auto-Encoding 3D Transformations

Learning the Global Descriptor for 3-D Object Recognition Based on Multiple Views Decomposition

Large Scale Joint Semantic Re-Localisation and Scene Understanding via Globally Unique Instance Coordinate Regression