Abstract:How to effectively retrieve desired 3D models with simple queries is a long-standing problem in computer vision community. The model-based approach is quite straightforward but nontrivial, since people could not always have the desired 3D query model available by side. Recently, large amounts of wide-screen electronic devices are prevail in our daily lives, which makes the sketch-based 3D shape retrieval a promising candidate due to its simpleness and efficiency. The main challenge of sketch-based approach is the huge modality gap between sketch and 3D shape. In this paper, we proposed a novel deep correlated holistic metric learning (DCHML) method to mitigate the discrepancy between sketch and 3D shape domains. The proposed DCHML trains two distinct deep neural networks (one for each domain) jointly, which learns two deep nonlinear transformations to map features from both domains into a new feature space. The proposed loss, including discriminative loss and correlation loss, aims to increase the discrimination of features within each domain as well as the correlation between different domains. In the new feature space, the discriminative loss minimizes the intra-class distance of the deep transformed features and maximizes the inter-class distance of the deep transformed features to a large margin within each domain, while the correlation loss focused on mitigating the distribution discrepancy across different domains. Different from existing deep metric learning methods only with loss at the output layer, our proposed DCHML is trained with loss at both hidden layer and output layer to further improve the performance by encouraging features in the hidden layer also with desired properties. Our proposed method is evaluated on three benchmarks, including 3D Shape Retrieval Contest 2013, 2014, and 2016 benchmarks, and the experimental results demonstrate the superiority of our proposed method over the state-of-the-art methods.

Deep Correlated Joint Network for 2-D Image-Based 3-D Model Retrieval

Cross-modal Retrieval Based on Deep Correlated Network

Joint Local Correlation and Global Contextual Information for Unsupervised 3D Model Retrieval and Classification

Deep Correlated Holistic Metric Learning for Sketch-Based 3D Shape Retrieval

M-GCN: Multi-Branch Graph Convolution Network for 2D Image-based on 3D Model Retrieval.

Learning Joint Embedding for Cross-Modal Retrieval

Convolutional Deep Learning for 3D Object Retrieval

Unsupervised Cross-media Graph Convolutional Network for 2D Image-based 3D Model Retrieval

Multi-Modal Retrieval Via Deep Textual-Visual Correlation Learning

Cross-Modality Retrieval by Joint Correlation Learning

JFLN: Joint Feature Learning Network for 2D Sketch Based 3D Shape Retrieval

Learning Pairwise Neural Network Encoder for Depth Image-based 3D Model Retrieval

Multi-graph Convolutional Network for Unsupervised 3D Shape Retrieval

Deep Point Convolutional Approach for 3D Model Retrieval.

Multiple Discrimination and Pairwise CNN for view-based 3D object retrieval

Joint Intermediate Domain Generation and Distribution Alignment for 2D Image-Based 3D Objects Retrieval.

CLN: Cross-Domain Learning Network for 2D Image-Based 3D Shape Retrieval

Correlated and Individual Multi-Modal Deep Learning for RGB-D Object Recognition.

HGAN: Holistic Generative Adversarial Networks for Two-dimensional Image-based Three-dimensional Object Retrieval

Cross‐modal Semantic Correlation Learning by Bi‐CNN Network

Deep Feature Correlation Learning for Multi-Modal Remote Sensing Image Registration