Abstract:This article considers the problem of cross-modal retrieval, such as using a text query to search for images and vice-versa. Based on different autoencoders, several novel models are proposed here for solving this problem. These models are constructed by correlating hidden representations of a pair of autoencoders. A novel optimal objective, which minimizes a linear combination of the representation learning errors for each modality and the correlation learning error between hidden representations of two modalities, is used to train the model as a whole. Minimizing the correlation learning error forces the model to learn hidden representations with only common information in different modalities, while minimizing the representation learning error makes hidden representations good enough to reconstruct inputs of each modality. To balance the two kind of errors induced by representation learning and correlation learning, we set a specific parameter in our models. Furthermore, according to the modalities the models attempt to reconstruct they are divided into two groups. One group including three models is named multimodal reconstruction correspondence autoencoder since it reconstructs both modalities. The other group including two models is named unimodal reconstruction correspondence autoencoder since it reconstructs a single modality. The proposed models are evaluated on three publicly available datasets. And our experiments demonstrate that our proposed correspondence autoencoders perform significantly better than three canonical correlation analysis based models and two popular multimodal deep models on cross-modal retrieval tasks.

Cross-modal Image-Text Retrieval with Multitask Learning

Hybrid DAER Based Cross-modal Retrieval Exploiting Deep Representation Learning

Cross-Modality Retrieval by Joint Correlation Learning

Hybrid Deep Neural Network-Based Cross-Modal Image and Text Retrieval Method for Large-Scale Data.

Multi-Modal Retrieval Via Deep Textual-Visual Correlation Learning

Cross-Modal Learning Based on Semantic Correlation and Multi-Task Learning for Text-Video Retrieval

A Cross-Modal Image and Text Retrieval Method Based on Efficient Feature Extraction and Interactive Learning CAE

Cross-modal Retrieval with Correspondence Autoencoder

Correspondence Autoencoders for Cross-Modal Retrieval

Incomplete Cross-modal Retrieval with Dual-Aligned Variational Autoencoders.

Deep Multi-Modal Metric Learning with Multi-Scale Correlation for Image-Text Retrieval

Improving Cross-Modal Image-Text Retrieval With Teacher-Student Learning

Cross‐modal Semantic Correlation Learning by Bi‐CNN Network

Cross‐modal retrieval with dual multi‐angle self‐attention

Cross-modal Retrieval Based on Deep Correlated Network

Multi-Task Visual Semantic Embedding Network for Image-Text Retrieval

Cross-Graph Attention Enhanced Multi-Modal Correlation Learning for Fine-Grained Image-Text Retrieval

Cross-modal Deep Metric Learning with Multi-Task Regularization

Learning Joint Embedding for Cross-Modal Retrieval

Cross-modal Contrastive Learning for Generalizable and Efficient Image-text Retrieval

Two-stage deep learning for supervised cross-modal retrieval