Abstract:Cross-modal retrieval has become popular in recent years, particularly with the rise of multimedia. Generally, the information from each modality exhibits distinct representations and semantic information, which makes feature tends to be in separate latent spaces encoded with dual-tower architecture and makes it difficult to establish semantic relationships between modalities, resulting in poor retrieval performance. To address this issue, we propose a novel framework for cross-modal retrieval which consists of a cross-modal mixer, a masked autoencoder for pre-training, and a cross-modal retriever for downstream <a class="link-external link-http" href="http://tasks.In" rel="external noopener nofollow">this http URL</a> specific, we first adopt cross-modal mixer and mask modeling to fuse the original modality and eliminate redundancy. Then, an encoder-decoder architecture is applied to achieve a fuse-then-separate task in the pre-training phase.We feed masked fused representations into the encoder and reconstruct them with the decoder, ultimately separating the original data of two modalities. In downstream tasks, we use the pre-trained encoder to build the cross-modal retrieval method. Extensive experiments on 2 real-world datasets show that our approach outperforms previous state-of-the-art methods in video-audio matching tasks, improving retrieval accuracy by up to 2 times. Furthermore, we prove our model performance by transferring it to other downstream tasks as a universal model.

A Method Based On General Model Template For Audio Retrieval

Audio Analysis Approach based on General Model

A Method Based on General Model and Rough Set for Audio Classification

Hybrid Support Vector Machine and General Model Approach for Audio Classification

An Approach Based on Reverse Hidden Markov Model for Audio Classification

Multiple Kernel Visual-Auditory Representation Learning for Retrieval

Audio Retrieval: Based On Unsupervised Learning Approach

Audio Similarity Measure by Graph Modeling and Matching

Hybrid Svm/Hmm Approach For Audio Classification

Introducing Auxiliary Text Query-modifier to Content-based Audio Retrieval

Content-based Multi-media Retrieval Technology

Audio retrieval based on perceptual similarity

Hybrid Independent Component Analysis And Support Vector Machine Approach For Audio Classification

Two-Stage Method for Specific Audio Retrieval.

Video and Audio are Images: A Cross-Modal Mixer for Original Data on Video-Audio Retrieval

Cross-Media Retrieval Method Based on Content Correlations

Hybrid Independent Component Analysis and Rough Set Approach for Audio Feature Extraction

Audio Retrieval with Fast Relevance Feedback Based on Constrained Fuzzy Clustering and Stored Index Table.

Apply Semantic Template to Support Content-Based Image Retrieval

Multimedia Analysis and Retrieval System

Anime Audio Retrieval Based on Audio Separation and Feature Recognition