Abstract:In recent years, large scale datasets of paired images and sentences have enabled the remarkable success in automatically generating descriptions for images, namely image captioning. However, it is labour-intensive and time-consuming to collect a sufficient number of paired images and sentences in each domain. It may be beneficial to transfer the image captioning model trained in an existing domain with pairs of images and sentences (i.e., source domain) to a new domain with only unpaired data (i.e., target domain). In this paper, we propose a cross-modal retrieval aided approach to cross-domain image captioning that leverages a cross-modal retrieval model to generate pseudo pairs of images and sentences in the target domain to facilitate the adaptation of the captioning model. To learn the correlation between images and sentences in the target domain, we propose an iterative cross-modal retrieval process where a cross-modal retrieval model is first pre-trained using the source domain data and then applied to the target domain data to acquire an initial set of pseudo image-sentence pairs. The pseudo image-sentence pairs are further refined by iteratively fine-tuning the retrieval model with the pseudo image-sentence pairs and updating the pseudo image-sentence pairs using the retrieval model. To make the linguistic patterns of the sentences learned in the source domain adapt well to the target domain, we propose an adaptive image captioning model with a self-attention mechanism fine-tuned using the refined pseudo image-sentence pairs. Experimental results on several settings where MSCOCO is used as the source domain and five different datasets (Flickr30k, TGIF, CUB-200, Oxford-102 and Conceptual) are used as the target domains demonstrate that our method achieves mostly better or comparable performance against the state-of-the-art methods. We also extend our method to cross-domain video captioning where MSR-VTT is used as the source domain and two other datasets (MSVD and Charades Captions) are used as the target domains to further demonstrate the effectiveness of our method.

Cross-Modality Bridging and Knowledge Transferring for Image Understanding

Towards Cross-Category Knowledge Propagation for Learning Visual Concepts

Bridging Asymmetry Between Image and Video: Cross-modality Knowledge Transfer Based on Learning from Video

Joint Inference of Objects and Scenes with Efficient Learning of Text-Object-Scene Relations

Web Multimedia Object Classification Using Cross-Domain Correlation Knowledge

Knowledge Transferring for Image Classification

Attention Bridging Network For Knowledge Transfer

Cross-Modal Knowledge Transfer Without Task-Relevant Source Data

Heterogeneous Transfer Learning for Image Classification.

Dynamic Knowledge Distillation with Cross-Modality Knowledge Transfer

Area-keywords Cross-Modal Alignment for Referring Image Segmentation

Cross-Modality Retrieval by Joint Correlation Learning

Semantic Consistency Knowledge Transfer for Unsupervised Cross Domain Object Detection

Cross-Domain Traffic Scene Understanding: A Dense Correspondence-Based Transfer Learning Approach

Cross-modal and Cross-domain Knowledge Transfer for Label-free 3D Segmentation

Cross-Domain Image Classification with Graph Regularization Transfer Sparse Concept Coding

Image-Text Embedding Learning Via Visual and Textual Semantic Reasoning.

Cross-Modal Learning With Images, Texts And Their Semantics

Cross-Domain Image Captioning Via Cross-Modal Retrieval and Model Adaptation

Cross-modal Common Representation Learning by Hybrid Transfer Network

Cross-Modal Image-Tag Relevance Learning for Social Images