Abstract:We investigate the problem of multimodal search of target modality, where the task involves enhancing a query in a specific target modality by integrating information from auxiliary modalities. The goal is to retrieve relevant objects whose contents in the target modality match the specified multimodal query. The paper first introduces two baseline approaches that integrate techniques from the Database, Information Retrieval, and Computer Vision communities. These baselines either merge the results of separate vector searches for each modality or perform a single-channel vector search by fusing all modalities. However, both baselines have limitations in terms of efficiency and accuracy as they fail to adequately consider the varying importance of fusing information across modalities. To overcome these limitations, the paper proposes a novel framework, called MUST. Our framework employs a hybrid fusion mechanism, combining different modalities at multiple stages. Notably, we leverage vector weight learning to determine the importance of each modality, thereby enhancing the accuracy of joint similarity measurement. Additionally, the proposed framework utilizes a fused proximity graph index, enabling efficient joint search for multimodal queries. MUST offers several other advantageous properties, including pluggable design to integrate any advanced embedding techniques, user flexibility to customize weight preferences, and modularized index construction. Extensive experiments on real-world datasets demonstrate the superiority of MUST over the baselines in terms of both search accuracy and efficiency. Our framework achieves over 10x faster search times while attaining an average of 93 datasets containing more than 10 million data elements.

Enhancing Unimodal Features Matters: A Multimodal Framework for Building Extraction.

Enhancing Multimodal Fusion with Only Unimodal Data

SemFE:A Feature Matching Method for Learnable Local Semantic Feature Enhancement in Multimodal Images

A multimodal fusion framework for urban scene understanding and functional identification using geospatial data

MEFusion: Unsupervised Mutual Enhancement for Multimodal Image Fusion

U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation

On Uni-Modal Feature Learning in Supervised Multi-Modal Learning

Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion

On Uni-modal Feature Learning in Multi-modal Learning

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

MUST: an Effective and Scalable Framework for Multimodal Search of Target Modality

Progressive fusion learning: A multimodal joint segmentation framework for building extraction from optical and SAR images

A multimodal framework for unsupervised feature fusion

Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment

GME: Improving Universal Multimodal Retrieval by Multimodal LLMs

ACMFNet: Attention-Based Cross-Modal Fusion Network for Building Extraction of Remote Sensing Images

Large Margin Multi-modal Multi-task Feature Extraction for Image Classification

Multi-Modal Entity Alignment Method Based on Feature Enhancement

Enhancing Separate Encoding with Multi-layer Feature Alignment for Image-Text Matching

A Multi-Feature Fusion Framework Based on D-S Theory for Automatic Building Extraction from High-Resolution Remote Sensing Imagery

Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval