Abstract:Zero-Shot Cross-Modal Retrieval (ZS-CMR) is an emerging research hotspot that aims to retrieve data of new classes across different modality data. It is challenging for not only the heterogeneous distributions across different modalities, but also the inconsistent semantics across seen and unseen classes. A handful of recently proposed methods typically borrow the idea from zero-shot learning, i.e., exploiting word embeddings of class labels (i.e., class-embeddings) as common semantic space, and using generative adversarial network (GAN) to capture the underlying multimodal data structures, as well as strengthen relations between input data and semantic space to generalize across seen and unseen classes. In this paper, we propose a novel method termed Learning Cross-Aligned Latent Embeddings (LCALE) as an alternative to these GAN based methods for ZS-CMR. Unlike using the class-embeddings as the semantic space, our method seeks for a shared low-dimensional latent space of input multimodal features and class-embeddings by modality-specific variational autoencoders. Notably, we align the distributions learned from multimodal input features and from class-embeddings to construct latent embeddings that contain the essential cross-modal correlation associated with unseen classes. Effective cross-reconstruction and cross-alignment criterions are further developed to preserve class-discriminative information in latent space, which benefits the efficiency for retrieval and enable the knowledge transfer to unseen classes. We evaluate our model using four benchmark datasets on image-text retrieval tasks and one largescale dataset on image-sketch retrieval tasks. The experimental results show that our method establishes the new state-of-the-art performance for both tasks on all datasets.

A Low Rank Structural Large Margin Method for Cross-Modal Ranking

Cross-Modal Learning to Rank Via Latent Joint Representation

Online Low-Rank Similarity Function Learning with Adaptive Relative Margin for Cross-Modal Retrieval.

Cross-media semantic representation via bi-directional learning to rank.

Zero-shot Learning with Regularized Cross-Modality Ranking.

PL-ranking

Deep Compositional Cross-modal Learning to Rank via Local-Global Alignment

Learning Multimodal Neural Network with Ranking Examples

Semantic Ranking Structure Preserving for Cross-Modal Retrieval

Simple to Complex Cross-modal Learning to Rank

Cross-modal Deep Metric Learning with Multi-Task Regularization

Cross-Modal Retrieval Using Multiordered Discriminative Structured Subspace Learning.

Multimedia Retrieval Via Deep Learning to Rank

Deep Supervised Cross-Modal Retrieval

Learning Cross-Aligned Latent Embeddings for Zero-Shot Cross-Modal Retrieval

Ranking-Based Deep Cross-Modal Hashing

Common Structured Low-Rank Matrix Recovery for Cross-View Classification

Semantic Pre-alignment and Ranking Learning with Unified Framework for Cross-modal Retrieval

Multi-Networks Joint Learning for Large-Scale Cross-Modal Retrieval

A Lightweight Multi-Scale Crossmodal Text-Image Retrieval Method in Remote Sensing

Reconstruction Regularized Low-Rank Subspace Learning for Cross-Modal Retrieval