Abstract:Recent generative models have demonstrated impressive capabilities in generating realistic and visually pleasing images grounded on textual prompts. Nevertheless, a significant challenge remains in applying these models for the more intricate task of story visualization. Since it requires resolving pronouns (he, she, they) in the frame descriptions, i.e., anaphora resolution, and ensuring consistent characters and background synthesis across frames. Yet, the emerging Large Language Model (LLM) showcases robust reasoning abilities to navigate through ambiguous references and process extensive sequences. Therefore, we introduce \textbf{StoryGPT-V}, which leverages the merits of the latent diffusion (LDM) and LLM to produce images with consistent and high-quality characters grounded on given story descriptions. First, we train a character-aware LDM, which takes character-augmented semantic embedding as input and includes the supervision of the cross-attention map using character segmentation masks, aiming to enhance character generation accuracy and faithfulness. In the second stage, we enable an alignment between the output of LLM and the character-augmented embedding residing in the input space of the first-stage model. This harnesses the reasoning ability of LLM to address ambiguous references and the comprehension capability to memorize the context. We conduct comprehensive experiments on two visual story visualization benchmarks. Our model reports superior quantitative results and consistently generates accurate characters of remarkable quality with low memory consumption. Our code will be made publicly available.

Latent Memory-Augmented Graph Transformer for Visual Storytelling

Storytelling from an Image Stream Using Scene Graphs

Vision Transformer Based Model for Describing a Set of Images as a Story

StoryGPT-V: Large Language Models as Consistent Story Visualizers

Generalizable Memory-driven Transformer for Multivariate Long Sequence Time-series Forecasting

Integrating Visuospatial, Linguistic and Commonsense Structure into Story Visualization

Spatial-Temporal Knowledge-Embedded Transformer for Video Scene Graph Generation

Long Short-Term Relation Transformer With Global Gating for Video Captioning

Transforming Visual Scene Graphs to Image Captions

Transformer-based Image Generation from Scene Graphs

Mart: Memory-Augmented Recurrent Transformer For Coherent Video Paragraph Captioning

TemporalStory: Enhancing Consistency in Story Visualization Using Spatial-Temporal Attention

Learning to Imagine: Visually-Augmented Natural Language Generation

A Novel End-to-End Transformer for Scene Graph Generation

Knowledgeable Storyteller: A Commonsense-Driven Generative Model for Visual Storytelling

Graph Transformer for Graph-to-Sequence Learning

StoryImager: A Unified and Efficient Framework for Coherent Story Visualization and Completion

AOG-LSTM: An Adaptive Attention Neural Network for Visual Storytelling

Emotion Reinforced Visual Storytelling.

VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers

From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models