Abstract:2D-3D joint learning is essential and effective for fundamental 3D vision tasks, such as 3D semantic segmentation, due to the complementary information these two visual modalities contain. Most current 3D scene semantic segmentation methods process 2D images "as they are", i.e., only real captured 2D images are used. However, such captured 2D images may be redundant, with abundant occlusion and/or limited field of view (FoV), leading to poor performance for the current methods involving 2D inputs. In this paper, we propose a general learning framework for joint 2D-3D scene understanding by selecting informative virtual 2D views of the underlying 3D scene. We then feed both the 3D geometry and the generated virtual 2D views into any joint 2D-3D-input or pure 3D-input based deep neural models for improving 3D scene understanding. Specifically, we generate virtual 2D views based on an information score map learned from the current 3D scene semantic segmentation results. To achieve this, we formalize the learning of the information score map as a deep reinforcement learning process, which rewards good predictions using a deep neural network. To obtain a compact set of virtual 2D views that jointly cover informative surfaces of the 3D scene as much as possible, we further propose an efficient greedy virtual view coverage strategy in the normal-sensitive 6D space, including 3-dimensional point coordinates and 3-dimensional normal. We have validated our proposed framework for various joint 2D-3D-input or pure 3D-input based deep neural models on two real-world 3D scene datasets, i.e., ScanNet v2 and S3DIS, and the results demonstrate that our method obtains a consistent gain over baseline models and achieves new top accuracy for joint 2D and 3D scene semantic segmentation. Code is available at https://github.com/smy-THU/VirtualViewSelection.

Best view selection of 3D models based on unsupervised feature learning and discrimination ability

A classification-based approach for best view selection of 3D models

Viewpoint-Aware Representation for Sketch-Based 3D Model Retrieval

Feature Representation for 3D Object Retrieval Based on Unconstrained Multi-View

View-based 3D object retrieval with discriminative views.

Exploring Discriminative Views for 3D Object Retrieval.

Adaptive unsupervised multi-view feature selection for visual concept recognition

Fast Best Viewpoint Selection with Geometry-Enhanced Multiple Views and Cross-Modal Distillation

Collaborative and Discriminative Subspace Learning for unsupervised multi-view feature selection

3D-NVS: A 3D Supervision Approach for Next View Selection

Structure-aware Viewpoint Selection for Volume Visualization

View-Based Discriminative Probabilistic Modeling for 3D Object Retrieval and Recognition

Similarity Voting Based Viewpoint Selection for Volumes

Unsupervised Multi-View CNN for Salient View Selection and 3D Interest Point Detection

Multi-level regularization-based unsupervised multi-view feature selection with adaptive graph learning

A Unified Feature Representation and Learning Framework for 3D Shape

Perceptual-Based Automatic Viewpoint Selection

Fast View-Based 3d Model Retrieval Via Unsupervised Multiple Feature Fusion and Online Projection Learning

Learning Virtual View Selection for 3D Scene Semantic Segmentation

3D shape classification based on convolutional neural networks fusing multi-view information

Unsupervised Cross-View Feature Selection on incomplete data