Abstract:As a bridge between language and vision domains, cross-modal retrieval between images and texts is a hot research topic in recent years. It remains challenging because the current image representations usually lack semantic concepts in the corresponding sentence captions. To address this issue, we introduce an intuitive and interpretable model to learn a common embedding space for alignments between images and text descriptions. Specifically, our model first incorporates the semantic relationship information into visual and textual features by performing region or word relationship reasoning. Then it utilizes the gate and memory mechanism to perform global semantic reasoning on these relationship-enhanced features, select the discriminative information and gradually grow representations for the whole scene. Through the alignment learning, the learned visual representations capture key objects and semantic concepts of a scene as in the corresponding text caption. Experiments on MS-COCO [1] and Flickr30K [2] datasets validate that our method surpasses many recent state-of-the-arts with a clear margin. In addition to the effectiveness, our methods are also very efficient at the inference stage. Thanks to the effective overall representation learning with visual semantic reasoning, our methods can already achieve very strong performance by only relying on the simple inner-product to obtain similarity scores between images and captions. Experiments validate the proposed methods are more than 30-75 times faster than many recent methods with code public available. Instead of following the recent trend of using complex local matching strategies [3], [4], [5], [6] to pursue good performance while sacrificing efficiency, we show that the simple global matching strategy can still be very effective, efficient and achieve even better performance based on our framework.

A Reconstruction-based Visual-Acoustic-Semantic Embedding Method for Speech-Image Retrieval

Multi-view visual semantic embedding for cross-modal image–text retrieval

Correlating subword articulation with lip shapes for embedding aware audio-visual speech enhancement

Dynamic Visual Semantic Sub-Embeddings and Fast Re-Ranking

Image-Text Embedding Learning Via Visual and Textual Semantic Reasoning.

Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues

Direction-Oriented Visual-semantic Embedding Model for Remote Sensing Image-text Retrieval

Coarse-to-fine Alignment Makes Better Speech-image Retrieval

SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Model

Rebuilding Visual Vocabulary via Spatial-temporal Context Similarity for Video Retrieval

Audio-Visual Speech Enhancement Based on Multiscale Features and Parallel Attention

Incorporating Visual Information Reconstruction into Progressive Learning for Optimizing audio-visual Speech Enhancement

Catch Missing Details: Image Reconstruction with Frequency Augmented Variational Autoencoder

ViLEM: Visual-Language Error Modeling for Image-Text Retrieval

Semantic Retrieval Of Personal Photos Using A Deep Autoencoder Fusing Visual Features With Speech Annotations Represented As Word/Paragraph Vectors

ViSTA: Vision and Scene Text Aggregation for Cross-Modal Retrieval

VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval

Unified Visual-Semantic Embeddings: Bridging Vision and Language with Structured Meaning Representations

Super Visual Semantic Embedding for Cross-Modal Image-Text Retrieval.

Deep Visual Semantic Embedding with Text Data Augmentation and Word Embedding Initialization

Rethinking the visual cues in audio-visual speaker extraction