Abstract:In order to effectively enhance the practicality of 3D model retrieval, we adopt a single real image as the query sample for retrieving 3D models. However, the significant differences between 2D images and 3D models in terms of lighting conditions, textures and backgrounds, posing a great challenge for accurate retrieval. Existing work on 3D model retrieval mainly focuses on closed-domain research, while the open-domain condition where the category relationship between the query image and the 3D model is unknown is more in line with the needs of real scenarios. CLIP shows significant promise in comprehending open-world visual concepts, facilitating effective zero-shot image recognition. Based on this multimodal pre-training large language model, we introduce Adaptive Open-domain Semantic Nearest-neighbor Contrast (AOSNC), a method for learning and aligning multi-modal text, image, and 3D model. In order to solve the issue of inconsistent cross-domain categories and difficult sample correlation in open-domain, we construct a cross-modal bridge using CLIP. This model utilizes textual features to bridge the gap between 2D images and 3D model views. Additionally, we design an adaptive network layer to address the limitations of the pre-training model for 3D model views and enhance cross-modal alignment. We propose a mutual nearest-neighbor semantic alignment loss to address the challenge of aligning features from disparate modalities (text, images, and 3D models). This loss function enhances cross-modal learning by effectively associating and distinguishing features, improving retrieval accuracy. We conducted comprehensive experiments using the image-based 3D model retrieval dataset MI3DOR and the cross-domain 3D model retrieval dataset NTU-PSB to validate the superiority of the proposed method. Our results show significant improvements in several evaluation metrics, underscoring the efficacy of our method in augmenting cross-modal feature alignment and retrieval performance.

Enhancing Image Retrieval : A Comprehensive Study on Photo Search using the CLIP Mode

InterCLIP: Adapting CLIP To Interactive Image Retrieval with Triplet Similarity

ENCLIP: Ensembling and Clustering-Based Contrastive Language-Image Pretraining for Fashion Multimodal Search with Limited Data and Low-Quality Images

CLIP-Branches: Interactive Fine-Tuning for Text-Image Retrieval

Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval

CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioning

Optimizing CLIP Models for Image Retrieval with Maintained Joint-Embedding Alignment

Enhancing Multimodal Understanding with CLIP-Based Image-to-Text Transformation

Iclip: Bridging Image Classification and Contrastive Language-Image Pre-Training for Visual Recognition

CLIP-Based Composed Image Retrieval with Comprehensive Fusion and Data Augmentation.

Adaptive CLIP for open-domain 3D model retrieval

Dr. CLIP: CLIP-Driven Universal Framework for Zero-Shot Sketch Image Retrieval

Extending CLIP for Category-to-image Retrieval in E-commerce

Composed Image Retrieval using Contrastive Learning and Task-oriented CLIP-based Features

Iterative Uni-modal and Cross-modal Clustered Contrastive Learning for Image-text Retrieval

Exploiting CLIP-based Multi-modal Approach for Artwork Classification and Retrieval

Demystifying CLIP Data

DiffCLIP: Few-shot Language-driven Multimodal Classifier

An Empirical Study of CLIP for Text-based Person Search

RadCLIP: Enhancing Radiologic Image Analysis through Contrastive Language-Image Pre-training