Abstract:Multimedia data with various modalities, such as image and text, are huge in quantity but have inconsistent distribution and representation. Many works have been done to break the boundary between image and text to measure their correlation. However, they focus on either the transformation to common subspace or the unidirectional generation from one to another individually, which cannot fully explore their interactions. It is noted that the bidirectional generation between image and text not only can provide complementary hints and mutually boost to learn cross-modal correlation but also cross-modal correlation learning can feed back to give comprehensive clues for promoting the cross-modal generation process. Therefore, we have the motivation that information transmission between image and text should be treated as a circular process, which aims to fully understand their latent correlation, and further realize cross-modal generation to produce both realistic images and text descriptions in a unified framework. In this paper, we propose the cross-modal circular correlation learning approach to perform both cross-modal correlation learning and generation simultaneously through an efficient circular learning training procedure. First, we propose the cross-modal circular learning model to perform an image-to-text caption and text-to-image synthesis circularly and learn common representation as a round-trip bridge, which can realize efficient interactions to fully exploit latent cross-modal correlations. Second, a unified bidirectional framework is proposed to conduct cross-modal mutual generation and is trained in an efficient circular process to enhance the generative ability of common representation, which can feed back circularly to further promote cross-modal correlation learning. In summary, we simultaneously perform cross-modal retrieval, image-to-text caption, and text-to-image synthesis in a unified framework with the circular learning process, which has high scalability and generality to realize universal cognition on the cross-modal data. We conduct extensive experiments to not only evaluate the correlation performance by cross-modal retrieval but also to show the generation effectiveness of both image caption and synthesis on the MS-COCO dataset.

MiC: Image-text Matching in Circles with cross-modal generative knowledge enhancement

Diversified text-to-image generation via deep mutual information estimation

Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models

Cross-modal Graph Matching Network for Image-text Retrieval

TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization

Conceptual and Syntactical Cross-modal Alignment with Cross-level Consistency for Image-Text Matching.

Contextual Relation Guided Image-Text Matching

Cross-modal Multi-Relationship Aware Reasoning for Image-Text Matching

Globally Guided Confidence Enhancement Network for Image-Text Matching

Show and Tell in the Loop: Cross-Modal Circular Correlation Learning

CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval

Improving Cross-Modal Image-Text Retrieval With Teacher-Student Learning

DSGSR - Dynamic Semantic Generation and Similarity Reasoning for Image-Text Matching.

Multimodal Knowledge Graph-Guided Cross-Modal Graph Network for Image-Text Retrieval

A Multiview Text Imagination Network Based on Latent Alignment for Image-Text Matching

Cross-modal Semantic Interference Suppression for image-text matching

Advanced Multimodal Deep Learning Architecture for Image-Text Matching

Improved Probabilistic Image-Text Representations

Multi-Head Attention Driven Dynamic Visual-Semantic Embedding for Enhanced Image-Text Matching

Multi-Modal Memory Enhancement Attention Network for Image-Text Matching

Show Your Faith: Cross-Modal Confidence-Aware Network for Image-Text Matching.