Abstract:Accurate Story visualization requires several necessary elements, such as identity consistency across frames, the alignment between plain text and visual content, and a reasonable layout of objects in images. Most previous works endeavor to meet these requirements by fitting a text-to-image (T2I) model on a set of videos in the same style and with the same characters, e.g., the FlintstonesSV dataset. However, the learned T2I models typically struggle to adapt to new characters, scenes, and styles, and often lack the flexibility to revise the layout of the synthesized images. This paper proposes a system for generic interactive story visualization, capable of handling multiple novel characters and supporting the editing of layout and local structure. It is developed by leveraging the prior knowledge of large language and T2I models, trained on massive corpora. The system comprises four interconnected components: story-to-prompt generation (S2P), text-to-layout generation (T2L), controllable text-to-image generation (C-T2I), and image-to-video animation (I2V). First, the S2P module converts concise story information into detailed prompts required for subsequent stages. Next, T2L generates diverse and reasonable layouts based on the prompts, offering users the ability to adjust and refine the layout to their preference. The core component, C-T2I, enables the creation of images guided by layouts, sketches, and actor-specific identifiers to maintain consistency and detail across visualizations. Finally, I2V enriches the visualization process by animating the generated images. Extensive experiments and a user study are conducted to validate the effectiveness and flexibility of interactive editing of the proposed system.

TaleStream: Supporting Story Ideation with Trope Knowledge

Story Designer: Towards a Mixed-Initiative Tool to Create Narrative Structures

TropeTwist: Trope-based Narrative Structure Generation

StoryExplorer: A Visualization Framework for Storyline Generation of Textual Narratives

Tropes in films: an initial analysis

Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopses

Annotating High-Level Structures of Short Stories and Personal Anecdotes

TaleCrafter: Interactive Story Visualization with Multiple Characters

Narrative Variations in a Virtual Storyteller

Conveying the Predicted Future to Users: A Case Study of Story Plot Prediction

Imagining from Images with an AI Storytelling Tool

TaleBrush: Sketching Stories with Generative Pretrained Language Models

CodeToon: Story Ideation, Auto Comic Generation, and Structure Mapping for Code-Driven Storytelling

Toward Enabling Reproducibility for Data-Intensive Research using the Whole Tale Platform

Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoning

StoryCraft AI: Exploring Generative Approaches to Story Narration through AI

Generating Different Story Tellings from Semantic Representations of Narrative

Finding Trolls Under Bridges: Preliminary Work on a Motif Detector

Physiologically Driven Storytelling: Concept and Software Tool

A Pipeline for Creative Visual Storytelling

The Stories We Tell About Data: Media Types for Data-Driven Storytelling