Abstract:Recent artificial intelligence research has witnessed great interest in automatically generating text descriptions of images, which are known as the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">image captioning</italic> task. Remarkable success has been achieved on domains where a large number of paired data in multimedia are available. Nevertheless, annotating sufficient data is labor-intensive and time-consuming, establishing significant barriers for adapting the image captioning systems to new domains. In this study, we introduc a novel Multitask Learning Algorithm for cross-Domain Image Captioning (MLADIC). MLADIC is a multitask system that simultaneously optimizes two coupled objectives via a dual learning mechanism: image captioning and text-to-image synthesis, with the hope that by leveraging the correlation of the two dual tasks, we are able to enhance the image captioning performance in the target domain. Concretely, the image captioning task is trained with an encoder–decoder model (i.e., CNN-LSTM) to generate textual descriptions of the input images. The image synthesis task employs the conditional generative adversarial network (C-GAN) to synthesize plausible images based on text descriptions. In C-GAN, a generative model <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$G$</tex-math></inline-formula> synthesizes plausible images given text descriptions, and a discriminative model <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$D$</tex-math></inline-formula> tries to distinguish the images in training data from the generated images by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$G$</tex-math></inline-formula> . The adversarial process can eventually guide <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$G$</tex-math></inline-formula> to generate plausible and high-quality images. To bridge the gap between different domains, a two-step strategy is adopted in order to transfer knowledge from the source domains to the target domains. First, we pre-train the model to learn the alignment between the neural representations of images and that of text data with the sufficient labeled source domain data. Second, we fine-tune the learned model by leveraging the limited image–text pairs and unpaired data in the target domain. We conduct extensive experiments to evaluate the performance of MLADIC by using the MSCOCO as the source domain data, and using Flickr30k and Oxford-102 as the target domain data. The results demonstrate that MLADIC achieves substantially better performance than the strong competitors for the cross-domain image captioning task.

Cross-Lingual Text Image Recognition Via Multi-Task Sequence to Sequence Learning.

Chinese Image Text Recognition with BLSTM-CTC: A Segmentation-Free Method.

Multi-Task Cross-Lingual Sequence Tagging from Scratch

Cross-Lingual Learning in Multilingual Scene Text Recognition

Mlts: A Multi-Language Scene Text Spotter

MTSTR: Multi-task learning for low-resolution scene text recognition via dual attention mechanism and its application in logistics industry

mCL-NER: Cross-Lingual Named Entity Recognition via Multi-view Contrastive Learning

TCT: A Cross-supervised Learning Method for Multimodal Sequence Representation

Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval

CLIP-Llama: A New Approach for Scene Text Recognition with a Pre-Trained Vision-Language Model and a Pre-Trained Language Model

Multi-Source Cross-Lingual Model Transfer: Learning What to Share

Chinese Text Recognition with A Pre-Trained CLIP-Like Model Through Image-IDS Aligning

Multi-Task Joint Learning Model for Segmenting and Classifying Tongue Images Using a Deep Neural Network

Enhancing Multimodal Understanding with CLIP-Based Image-to-Text Transformation

Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training

FC-MTLF: A Fine- and Coarse-grained Multi-Task Learning Framework for Cross-Lingual Spoken Language Understanding.

Multi-task Recurrent Model for True Multilingual Speech Recognition

Mmt: A Multimodal Translator For Image Captioning

Towards Pure End-to-End Learning for Recognizing Multiple Text Sequences from an Image

Multitask Learning for Cross-Domain Image Captioning

Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning