Abstract:Diverse image captioning has achieved substantial progress in recent years. However, the discriminability of generative models and the limitation of cross entropy loss are generally overlooked in the traditional diverse image captioning models, which seriously hurts both the diversity and accuracy of image captioning. In this paper, aiming to improve diversity and accuracy simultaneously, we propose a novel Conditional Variational Autoencoder (DCL-CVAE) framework for diverse image captioning by seamlessly integrating sequential variational autoencoder with contrastive learning. In the encoding stage, we first build conditional variational autoencoders to separately learn the sequential latent spaces for a pair of captions. Then, we introduce contrastive learning in the sequential latent spaces to enhance the discriminability of latent representations for both image-caption pairs and mismatched pairs. In the decoding stage, we leverage the captions sampled from the pre-trained Long Short-Term Memory (LSTM), LSTM decoder as the negative examples and perform contrastive learning with the greedily sampled positive examples, which can restrain the generation of common words and phrases induced by the cross entropy loss. By virtue of dual constrastive learning, DCL-CVAE is capable of encouraging the discriminability and facilitating the diversity, while promoting the accuracy of the generated captions. Extensive experiments are conducted on the challenging MSCOCO dataset, showing that our proposed methods can achieve a better balance between accuracy and diversity compared to the state-of-the-art diverse image captioning models.

Boosting Variational Generative Model Via Condition Enhancing and Lexical-Editing

Poetry Generation Algorithm with Automatic Expansion of Keyword Semantic Information

Improve Diverse Text Generation by Self Labeling Conditional Variational Auto Encoder

Condition-Transforming Variational Autoencoder for Generating Diverse Short Text Conversations.

Condition-Transforming Variational Autoencoder for Generating Diverse Short Text Conversations

Retrieval Bias Aware Ensemble Model for Conditional Sentence Generation.

Condition-transforming Variational Autoencoder for Conversation Response Generation.

Plan-CVAE - A Planning-Based Conditional Variational Autoencoder for Story Generation.

Advanced Conditional Variational Autoencoders (A-CVAE): Towards interpreting open-domain conversation generation via disentangling latent feature representation

Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck

Transformer-Based Conditioned Variational Autoencoder for Dialogue Generation

Improved Variational Autoencoders for Text Modeling using Dilated Convolutions

On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond

Pre-train and Plug-in: Flexible Conditional Text Generation with Variational Auto-Encoders

Diverse and Accurate Image Description Using a Variational Auto-Encoder with an Additive Gaussian Encoding Space

Cross-Utterance Conditioned VAE for Speech Generation

Diverse Image Captioning via Conditional Variational Autoencoder and Dual Contrastive Learning

Draft and Edit: Automatic Storytelling Through Multi-Pass Hierarchical Conditional Variational Autoencoder

RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modeling

A Temporal Variational Model for Story Generation

Improving Variational Autoencoders with Density Gap-based Regularization