Abstract:Visual storytelling aims at producing a narrative paragraph for a given photo album automatically. It introduces more new challenges than individual image paragraph descriptions, mainly due to the difficulty in preserving coherent topics and in generating diverse phrases to depict the rich content of a photo album. Existing attention-based models that lack higher-level guiding information always result in a deviation between the generated sentence and the topic expressed by the image. In addition, these widely applied language generation approaches employing standard beam search tend to produce monotonous descriptions. In this work, a coherent visual storytelling (CoVS) framework is designed to address the above-mentioned problems. Specifically, in the encoding phase, an image sequence encoder is designed to efficiently extract visual features of the input photo album. Then, the novel parallel top-down visual and topic attention (PTDVTA) decoder is constructed via a topic-aware neural network, a parallel top-down attention model, and a coherent language generator. Concretely, visual attention focuses on the attributes and the relationships of the objects, while topic attention integrating a topic-aware neural network could improve the coherence of generated sentences. Eventually, a phrase beam search algorithm with $n$ -gram hamming diversity is further designed to optimize the expression diversity of the generated story. To justify the proposed CoVS framework, extensive experiments are conducted on the VIST dataset, which shows that CoVS can automatically generate coherent and diverse stories in a more natural way. Moreover, CoVS obtains better performance than state-of-the-art baselines on BLEU-4 and METEOR scores, while maintaining good CIDEr and ROUGH_L scores. The source code of this work can be found in https://mic.tongji.edu.cn .

Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling

Knowledgeable Storyteller: A Commonsense-Driven Generative Model for Visual Storytelling

Knowledge-enriched Attention Network with Group-wise Semantic for Visual Storytelling

Knowledge-Enriched Visual Storytelling

SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling

Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational Reasoning.

AutoStory: Generating Diverse Storytelling Images with Minimal Human Effort

Towards Confidence-Aware Commonsense Knowledge Integration for Scene Graph Generation

Glocal Cascading Network for Topic Enhanced Visual Storytelling

Informative Visual Storytelling with Cross-modal Rules

Bridging Visual and Textual Semantics: Towards Consistency for Unbiased Scene Graph Generation

A Knowledge-Enhanced Pretraining Model for Commonsense Story Generation

Retrieval, Selection and Writing: A Three-Stage Knowledge Grounded Storytelling Model

Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning

Automatic Suggestion Of Presentation Image For Storytelling

DIVE: Towards Descriptive and Diverse Visual Commonsense Generation

Efficient Large-Scale Image Data Set Exploration: Visual Concept Network and Image Summarization

SentiStory: A Multi-Layered Sentiment-Aware Generative Model for Visual Storytelling

Neural Storyboard Artist: Visualizing Stories with Coherent Image Sequences

Coherent Visual Storytelling Via Parallel Top-Down Visual and Topic Attention

Beware of Overcorrection: Scene-induced Commonsense Graph for Scene Graph Generation