Abstract:Image captioning has been an emerging and fast-developing research topic. Nevertheless, most existing works heavily rely on large amounts of image-sentence pairs and therefore hinder the practical applications of captioning in the wild. In this paper, we present a novel Semantic-Constrained Self-learning (SCS) framework that explores an iterative self-learning strategy to learn an image captioner with only unpaired image and text data. Technically, SCS consists of two stages, i.e., pseudo pair generation and captioner re-training, iteratively producing "pseudo" image-sentence pairs via a pre-trained captioner and re-training the captioner with the pseudo pairs, respectively. Particularly, both stages are guided by the recognized objects in the image, that act as semantic constraint to strengthen the semantic alignment between the input image and the output sentence. We leverage a semantic-constrained beam search for pseudo pair generation to regularize the decoding process with the recognized objects via forcing the inclusion/exclusion of the recognized/irrelevant objects in output sentence. For captioner re-training, a self-supervised triplet loss is utilized to preserve the relative semantic similarity ordering among generated sentences with regard to the input image triplets. Moreover, an object inclusion reward and an adversarial reward are adopted to encourage the inclusion of the predicted objects in the output sentence and pursue the generation of more realistic sentences during self-critical training, respectively. Experiments conducted on both dependent and independent unpaired data validate the superiority of SCS. More remarkably, we obtain the best published CIDEr score to-date of 74.7% on COCO Karpathy test split for unpaired image captioning.

Image Captioning with Inherent Sentiment

Image Captioning at Will: A Versatile Scheme for Effectively Injecting Sentiments into Image Descriptions

SentiCap: Generating Image Descriptions with Sentiments

Senti-Attend: Image Captioning using Sentiment and Attention

Automatic Image Description Generation with Emotional Classifiers

Visual sentiment analysis based on image caption and adjective–noun–pair description

"Factual" or "Emotional": Stylized Image Captioning with Adaptive Learning and Attention

Sentiment Commonsense Induced Sequential Neural Networks for Sentiment Classification.

Cross-modal image sentiment analysis via deep correlation of textual semantic

Image Captioning using Facial Expression and Attention

Similar Scenes Arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning

Emotional Video Captioning With Vision-Based Emotion Interpretation Network

Improving Image Captioning with Better Use of Caption

Evaluating authenticity and quality of image captions via sentiment and semantic analyses

Improving Image Captioning with Better Use of Captions

Image captioning with weakly-supervised attention penalty

A Semi-supervised Framework for Image Captioning

Unpaired Image Captioning With semantic-Constrained Self-Learning

Image Captioning with Multi-Context Synthetic Data

Stylized Image Captioning Model Based on Disentangle-Retrieve-Generate

Recurrent Image Captioner: Describing Images with Spatial-Invariant Transformation and Attention Filtering