Abstract:In order to effectively enhance the practicality of 3D model retrieval, we adopt a single real image as the query sample for retrieving 3D models. However, the significant differences between 2D images and 3D models in terms of lighting conditions, textures and backgrounds, posing a great challenge for accurate retrieval. Existing work on 3D model retrieval mainly focuses on closed-domain research, while the open-domain condition where the category relationship between the query image and the 3D model is unknown is more in line with the needs of real scenarios. CLIP shows significant promise in comprehending open-world visual concepts, facilitating effective zero-shot image recognition. Based on this multimodal pre-training large language model, we introduce Adaptive Open-domain Semantic Nearest-neighbor Contrast (AOSNC), a method for learning and aligning multi-modal text, image, and 3D model. In order to solve the issue of inconsistent cross-domain categories and difficult sample correlation in open-domain, we construct a cross-modal bridge using CLIP. This model utilizes textual features to bridge the gap between 2D images and 3D model views. Additionally, we design an adaptive network layer to address the limitations of the pre-training model for 3D model views and enhance cross-modal alignment. We propose a mutual nearest-neighbor semantic alignment loss to address the challenge of aligning features from disparate modalities (text, images, and 3D models). This loss function enhances cross-modal learning by effectively associating and distinguishing features, improving retrieval accuracy. We conducted comprehensive experiments using the image-based 3D model retrieval dataset MI3DOR and the cross-domain 3D model retrieval dataset NTU-PSB to validate the superiority of the proposed method. Our results show significant improvements in several evaluation metrics, underscoring the efficacy of our method in augmenting cross-modal feature alignment and retrieval performance.

Domain-Specific Modeling and Semantic Alignment for Image-Based 3d Model Retrieval

Source-enhanced Prototypical Alignment for Single Image 3D Model Retrieval

An Overview of Semantics Processing in Content-Based 3D Model Retrieval

An Ontology-Based Semantic Retrieval Approach for Heterogeneous 3D CAD Models

Domain-Specific Alignment Network for Multi-Domain Image-Based 3D Object Retrieval

Image-based 3D Model Retrieval Via Disentangled Feature Learning and Enhanced Semantic Alignment

Semantic-oriented 3D model classification and retrieval using Gaussian processes

Self-supervised Image-based 3D Model Retrieval

Cross-Domain 3D Model Retrieval Via Visual Domain Adaption

Consistent Domain Structure Learning and Domain Alignment for 2D Image-Based 3D Objects Retrieval.

Modeling Image Data for Effective Indexing and Retrieval in Large General Image Databases.

Learning Transferable and Discriminative Representations for 2D Image-Based 3D Model Retrieval

Joint Intermediate Domain Generation and Distribution Alignment for 2D Image-Based 3D Objects Retrieval.

Adaptive CLIP for open-domain 3D model retrieval

Image-based 3D Model Retrieval Using Manifold Learning

Semantic-oriented 3D Model Retrieval Using Visual Vocabulary Labelling

Combining Topological and View-Based Features for 3D Model Retrieval

Survey of 3D model retrieval and mining system

3D Model Retrieval Via Single Image Based on Feature Mapping.

ModelSeek: an Effective 3D Model Retrieval System

Monocular Image-Based 3-D Model Retrieval: A Benchmark