Abstract:The traditional 3D object retrieval (3DOR) task is under the close-set setting, which assumes the categories of objects in the retrieval stage are all seen in the training stage. Existing methods under this setting may tend to only lazily discriminate their categories, while not learning a generalized 3D object embedding. Under such circumstances, it is still a challenging and open problem in real-world applications due to the existence of various unseen categories. In this paper, we first introduce the open-set 3DOR task to expand the applications of the traditional 3DOR task. Then, we propose the Hypergraph-Based Multi-Modal Representation (HGM 2 R) framework to learn 3D object embeddings from multi-modal representations under the open-set setting. The proposed framework is composed of two modules, i.e., the Multi-Modal 3D Object Embedding (MM3DOE) module and the Structure-Aware and Invariant Knowledge Learning (SAIKL) module. By utilizing the collaborative information of modalities derived from the same 3D object, the MM3DOE module is able to overcome the distinction across different modality representations and generate unified 3D object embeddings. Then, the SAIKL module utilizes the constructed hypergraph structure to model the high-order correlation among 3D objects from both seen and unseen categories. The SAIKL module also includes a memory bank that stores typical representations of 3D objects. By aligning with those memory anchors in the memory bank, the aligned embeddings can integrate the invariant knowledge to exhibit a powerful generalized capacity toward unseen categories. We formally prove that hypergraph modeling has better representative capability on data correlation than graph modeling. We generate four multi-modal datasets for the open-set 3DOR task, i.e., OS-ESB-core, OS-NTU-core, OS-MN40-core, and OS-ABO-core, in which each 3D object contains three modality representations: multi-view, point clouds, and voxel. Experiments on these four datasets show that the proposed method can significantly outperform existing methods. In particular, the proposed method outperforms the state-of-the-art by 12.12%/12.88% in terms of mAP on the OS-MN40-core/OS-ABO-core dataset, respectively. Results and visualizations demonstrate that the proposed method can effectively extract the generalized 3D object embeddings on the open-set 3DOR task and achieve satisfactory performance.

HGAN: Holistic Generative Adversarial Networks for Two-dimensional Image-based Three-dimensional Object Retrieval

Virtual View Generation Based on 3D-Dense-attentive GAN Networks

Feature Representation for 3D Object Retrieval Based on Unconstrained Multi-View

Multi-View Saliency Guided Deep Neural Network for 3-D Object Retrieval and Classification

Group-Pair Convolutional Neural Networks for Multi-View Based 3D Object Retrieval.

Domain-Specific Alignment Network for Multi-Domain Image-Based 3D Object Retrieval

Unsupervised Cross-media Graph Convolutional Network for 2D Image-based 3D Model Retrieval

Multi-View 3d Object Retrieval with Deep Embedding Network

Matching 2D Image Patches and 3D Point Cloud Volumes by Learning Local Cross-domain Feature Descriptors

Deep Correlated Joint Network for 2-D Image-Based 3-D Model Retrieval

Joint Intermediate Domain Generation and Distribution Alignment for 2D Image-Based 3D Objects Retrieval.

M-GCN: Multi-Branch Graph Convolution Network for 2D Image-based on 3D Model Retrieval.

Do 2D GANs Know 3D Shape? Unsupervised 3D shape reconstruction from 2D Image GANs

Cross-Domain 3D Model Retrieval Via Visual Domain Adaption

Hierarchical Set-to-Set Representation for 3-D Cross-Modal Retrieval

Convolutional Deep Learning for 3D Object Retrieval

Hypergraph-Based Multi-Modal Representation for Open-Set 3D Object Retrieval

3D-Mask-GAN:Unsupervised Single-View 3D Object Reconstruction

Domain-Adversarial-Guided Siamese Network for Unsupervised Cross-Domain 3-D Object Retrieval

Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval.

Non-rigid 3D Shape Retrieval Based on Multi-Scale Graphical Image and Joint Bayesian