Abstract:In order to effectively enhance the practicality of 3D model retrieval, we adopt a single real image as the query sample for retrieving 3D models. However, the significant differences between 2D images and 3D models in terms of lighting conditions, textures and backgrounds, posing a great challenge for accurate retrieval. Existing work on 3D model retrieval mainly focuses on closed-domain research, while the open-domain condition where the category relationship between the query image and the 3D model is unknown is more in line with the needs of real scenarios. CLIP shows significant promise in comprehending open-world visual concepts, facilitating effective zero-shot image recognition. Based on this multimodal pre-training large language model, we introduce Adaptive Open-domain Semantic Nearest-neighbor Contrast (AOSNC), a method for learning and aligning multi-modal text, image, and 3D model. In order to solve the issue of inconsistent cross-domain categories and difficult sample correlation in open-domain, we construct a cross-modal bridge using CLIP. This model utilizes textual features to bridge the gap between 2D images and 3D model views. Additionally, we design an adaptive network layer to address the limitations of the pre-training model for 3D model views and enhance cross-modal alignment. We propose a mutual nearest-neighbor semantic alignment loss to address the challenge of aligning features from disparate modalities (text, images, and 3D models). This loss function enhances cross-modal learning by effectively associating and distinguishing features, improving retrieval accuracy. We conducted comprehensive experiments using the image-based 3D model retrieval dataset MI3DOR and the cross-domain 3D model retrieval dataset NTU-PSB to validate the superiority of the proposed method. Our results show significant improvements in several evaluation metrics, underscoring the efficacy of our method in augmenting cross-modal feature alignment and retrieval performance.

Source-enhanced Prototypical Alignment for Single Image 3D Model Retrieval

Domain-Specific Modeling and Semantic Alignment for Image-Based 3d Model Retrieval

Image-based 3D Model Retrieval Via Disentangled Feature Learning and Enhanced Semantic Alignment

Domain-Specific Alignment Network for Multi-Domain Image-Based 3D Object Retrieval

Self-supervised Image-based 3D Model Retrieval

Consistent Domain Structure Learning and Domain Alignment for 2D Image-Based 3D Objects Retrieval.

Focus on Hard Samples: Hierarchical Unbiased Constraints for Cross-Domain 3D Model Retrieval

Adaptive CLIP for open-domain 3D model retrieval

Learning Transferable and Discriminative Representations for 2D Image-Based 3D Model Retrieval

Cross-Domain Image-Object Retrieval Based on Weighted Optimal Transport

Joint Intermediate Domain Generation and Distribution Alignment for 2D Image-Based 3D Objects Retrieval.

Cross-Domain 3D Model Retrieval Via Visual Domain Adaption

Progressive Contrastive Label Optimization for Source-Free Universal 3D Model Retrieval

CMFAN: Cross-Modal Feature Alignment Network for Few-Shot Single-View 3D Reconstruction

Category Alignment Adversarial Learning for Cross-modal Retrieval

Universal unsupervised cross-domain 3D shape retrieval

Alignment efficient image-sentence retrieval considering transferable cross-modal representation learning

Category Alignment Mechanism for Few-Shot Image Classification

Combining Topological and View-Based Features for 3D Model Retrieval

Fine-grained Prototypical Voting with Heterogeneous Mixup for Semi-supervised 2D-3D Cross-modal Retrieval

Correspondence-Free Domain Alignment for Unsupervised Cross-Domain Image Retrieval.