Abstract:Recent artificial intelligence research has witnessed great interest in automatically generating text descriptions of images, which are known as the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">image captioning</italic> task. Remarkable success has been achieved on domains where a large number of paired data in multimedia are available. Nevertheless, annotating sufficient data is labor-intensive and time-consuming, establishing significant barriers for adapting the image captioning systems to new domains. In this study, we introduc a novel Multitask Learning Algorithm for cross-Domain Image Captioning (MLADIC). MLADIC is a multitask system that simultaneously optimizes two coupled objectives via a dual learning mechanism: image captioning and text-to-image synthesis, with the hope that by leveraging the correlation of the two dual tasks, we are able to enhance the image captioning performance in the target domain. Concretely, the image captioning task is trained with an encoder–decoder model (i.e., CNN-LSTM) to generate textual descriptions of the input images. The image synthesis task employs the conditional generative adversarial network (C-GAN) to synthesize plausible images based on text descriptions. In C-GAN, a generative model <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$G$</tex-math></inline-formula> synthesizes plausible images given text descriptions, and a discriminative model <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$D$</tex-math></inline-formula> tries to distinguish the images in training data from the generated images by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$G$</tex-math></inline-formula> . The adversarial process can eventually guide <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$G$</tex-math></inline-formula> to generate plausible and high-quality images. To bridge the gap between different domains, a two-step strategy is adopted in order to transfer knowledge from the source domains to the target domains. First, we pre-train the model to learn the alignment between the neural representations of images and that of text data with the sufficient labeled source domain data. Second, we fine-tune the learned model by leveraging the limited image–text pairs and unpaired data in the target domain. We conduct extensive experiments to evaluate the performance of MLADIC by using the MSCOCO as the source domain data, and using Flickr30k and Oxford-102 as the target domain data. The results demonstrate that MLADIC achieves substantially better performance than the strong competitors for the cross-domain image captioning task.

The Power of Many: Multi-Agent Multimodal Models for Cultural Image Captioning

Cultural Concept Adaptation on Multimodal Reasoning

How Culturally Aware are Vision-Language Models?

Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models

CompCap: Improving Multimodal Large Language Models with Composite Captions

MIBench: Evaluating Multimodal Large Language Models over Multiple Images

Exploring Overall Contextual Information for Image Captioning in Human-Like Cognitive Style

A Multi-task Learning Approach for Image Captioning.

Instruction-guided Multi-Granularity Segmentation and Captioning with Large Multimodal Model

Multitask Learning for Cross-Domain Image Captioning

MOSABench: Multi-Object Sentiment Analysis Benchmark for Evaluating Multimodal Large Language Models Understanding of Complex Image

Improving Multimodal Datasets with Image Captioning

CulturePark: Boosting Cross-cultural Understanding in Large Language Models

Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models

CIC: A framework for Culturally-aware Image Captioning

MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms

Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs

Personalizing Multimodal Large Language Models for Image Captioning: An Experimental Analysis

A Multi-Modal ELMo Model for Image Sentiment Recognition of Consumer Data

MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts