Abstract:In order to effectively enhance the practicality of 3D model retrieval, we adopt a single real image as the query sample for retrieving 3D models. However, the significant differences between 2D images and 3D models in terms of lighting conditions, textures and backgrounds, posing a great challenge for accurate retrieval. Existing work on 3D model retrieval mainly focuses on closed-domain research, while the open-domain condition where the category relationship between the query image and the 3D model is unknown is more in line with the needs of real scenarios. CLIP shows significant promise in comprehending open-world visual concepts, facilitating effective zero-shot image recognition. Based on this multimodal pre-training large language model, we introduce Adaptive Open-domain Semantic Nearest-neighbor Contrast (AOSNC), a method for learning and aligning multi-modal text, image, and 3D model. In order to solve the issue of inconsistent cross-domain categories and difficult sample correlation in open-domain, we construct a cross-modal bridge using CLIP. This model utilizes textual features to bridge the gap between 2D images and 3D model views. Additionally, we design an adaptive network layer to address the limitations of the pre-training model for 3D model views and enhance cross-modal alignment. We propose a mutual nearest-neighbor semantic alignment loss to address the challenge of aligning features from disparate modalities (text, images, and 3D models). This loss function enhances cross-modal learning by effectively associating and distinguishing features, improving retrieval accuracy. We conducted comprehensive experiments using the image-based 3D model retrieval dataset MI3DOR and the cross-domain 3D model retrieval dataset NTU-PSB to validate the superiority of the proposed method. Our results show significant improvements in several evaluation metrics, underscoring the efficacy of our method in augmenting cross-modal feature alignment and retrieval performance.

Self-supervised Image-based 3D Model Retrieval

Domain-Specific Modeling and Semantic Alignment for Image-Based 3d Model Retrieval

Source-enhanced Prototypical Alignment for Single Image 3D Model Retrieval

Image retrieval based on incremental subspace learning

Cross-Domain 3D Model Retrieval Via Visual Domain Adaption

Image-based 3D Model Retrieval Via Disentangled Feature Learning and Enhanced Semantic Alignment

Domain-Specific Alignment Network for Multi-Domain Image-Based 3D Object Retrieval

Learning Transferable and Discriminative Representations for 2D Image-Based 3D Model Retrieval

Self-supervised adversarial learning for cross-modal retrieval

Monocular Image-Based 3-D Model Retrieval: A Benchmark

Domain-Adversarial-Guided Siamese Network for Unsupervised Cross-Domain 3-D Object Retrieval

Unsupervised Cross-Domain Image Retrieval with Semantic-Attended Mixture-of-Experts

Focus on Hard Samples: Hierarchical Unbiased Constraints for Cross-Domain 3D Model Retrieval

Consistent Domain Structure Learning and Domain Alignment for 2D Image-Based 3D Objects Retrieval.

Unsupervised self-training correction learning for 2D image-based 3D model retrieval

Joint deep feature learning and unsupervised visual domain adaptation for cross-domain 3D object retrieval

Adaptive CLIP for open-domain 3D model retrieval

Image-based 3D Model Retrieval Using Manifold Learning

Joint Intermediate Domain Generation and Distribution Alignment for 2D Image-Based 3D Objects Retrieval.

Fully Unsupervised Domain-Agnostic Image Retrieval

Self-supervised Exclusive Learning for 3D Segmentation with Cross-modal Unsupervised Domain Adaptation