Abstract:As two fundamental representation modalities of 3D objects, 3D point clouds and multi-view 2D images record shape information from different domains of geometric structures and visual appearances. In the current deep learning era, remarkable progress in processing such two data modalities has been achieved through respectively customizing compatible 3D and 2D network architectures. However, unlike multi-view image-based 2D visual modeling paradigms, which have shown leading performance in several common 3D shape recognition benchmarks, point cloud-based 3D geometric modeling paradigms are still highly limited by insufficient learning capacity, due to the difficulty of extracting discriminative features from irregular geometric signals. In this paper, we explore the possibility of boosting deep 3D point cloud encoders by transferring visual knowledge extracted from deep 2D image encoders under a standard teacher-student distillation workflow. Generally, we propose PointMCD, a unified multi-view cross-modal distillation architecture, including a pretrained deep image encoder as the teacher and a deep point encoder as the student. To perform heterogeneous feature alignment between 2D visual and 3D geometric domains, we further investigate visibility-aware feature projection (VAFP), by which point-wise embeddings are reasonably aggregated into view-specific geometric descriptors. By pair-wisely aligning multi-view visual and geometric descriptors, we can obtain more powerful deep point encoders without exhausting and complicated network modification. Experiments on 3D shape classification, part segmentation, and unsupervised learning strongly validate the effectiveness of our method. The code and data will be publicly available at <a class="link-external link-https" href="https://github.com/keeganhk/PointMCD" rel="external noopener nofollow">this https URL</a>.

Viewpoint Estimation for Workpieces with Deep Transfer Learning from Cold to Hot

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Monocular Depth Estimation Based on Unsupervised Learning

Domain Adaptation for Viewpoint Estimation with Image Generation

Efficient Encoding and Aligning Viewpoints for 6D Pose Estimation of Unseen Industrial Parts

Cad-based Viewpoint Estimation of Texture-Less Object for Purposive Perception Using Domain Adaptation.

Viewpoint Estimation Using Triplet Loss with A Novel Viewpoint-based Input Selection Strategy

Model-Based Active Viewpoint Transfer For Purposive Perception

VIEWPOINT ESTIMATION IN IMAGES BY A KEY-POINT BASED DEEP NEURAL NETWORK

Semantic Translation With Convolutional Encoder-Decoder Networks For Viewpoint Estimation

Deep Transfer Neural Network Using Hybrid Representations of Domain Discrepancy

Monocular Viewpoints Estimation for Generic Objects in the Wild

Outline viewpoint feature histogram: An improved point cloud descriptor for recognition and grasping of workpieces

View-Based 3-D CAD Model Retrieval With Deep Residual Networks.

Learning Monocular Depth by Distilling Cross-domain Stereo Networks

HVDistill: Transferring Knowledge from Images to Point Clouds via Unsupervised Hybrid-View Distillation

PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognition

Towards 3D Human Pose Estimation in the Wild: a Weakly-supervised Approach

Neural View Synthesis and Matching for Semi-Supervised Few-Shot Learning of 3D Pose

Deep Networks for Image-to-Image Prediction