Abstract:Story visualization aims to create visually compelling images or videos corresponding to textual narratives. Despite recent advances in diffusion models yielding promising results, existing methods still struggle to create a coherent sequence of subject-consistent frames based solely on a story. To this end, we propose DreamStory, an automatic open-domain story visualization framework by leveraging the LLMs and a novel multi-subject consistent diffusion model. DreamStory consists of (1) an LLM acting as a story director and (2) an innovative Multi-Subject consistent Diffusion model (MSD) for generating consistent multi-subject across the images. First, DreamStory employs the LLM to generate descriptive prompts for subjects and scenes aligned with the story, annotating each scene's subjects for subsequent subject-consistent generation. Second, DreamStory utilizes these detailed subject descriptions to create portraits of the subjects, with these portraits and their corresponding textual information serving as multimodal anchors (guidance). Finally, the MSD uses these multimodal anchors to generate story scenes with consistent multi-subject. Specifically, the MSD includes Masked Mutual Self-Attention (MMSA) and Masked Mutual Cross-Attention (MMCA) modules. MMSA and MMCA modules ensure appearance and semantic consistency with reference images and text, respectively. Both modules employ masking mechanisms to prevent subject blending. To validate our approach and promote progress in story visualization, we established a benchmark, DS-500, which can assess the overall performance of the story visualization framework, subject-identification accuracy, and the consistency of the generation model. Extensive experiments validate the effectiveness of DreamStory in both subjective and objective evaluations. Please visit our project homepage at <a class="link-external link-https" href="https://dream-xyz.github.io/dreamstory" rel="external noopener nofollow">this https URL</a>.

Connecting Dreams with Visual Brainstorming Instruction

BrainDreamer: Reasoning-Coherent and Controllable Image Generation from EEG Brain Signals via Language Guidance

DreamDiffusion: Generating High-Quality Images from Brain EEG Signals

A Novel Approach to Image EEG Sleep Data for Improving Quality of Life in Patients Suffering From Brain Injuries Using DreamDiffusion

DreamDiffusion: High-Quality EEG-to-Image Generation with Temporal Masked Signal Modeling and CLIP Alignment

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction

Brain Dialogue Interface (BDI): A User-Friendly fMRI Model for Interactive Brain Decoding

SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation

Dynamic Neural Communication: Convergence of Computer Vision and Brain-Computer Interface

DreamStory: Open-Domain Story Visualization by LLM-Guided Multi-Subject Consistent Diffusion

MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion

MindSpeech: Continuous Imagined Speech Decoding using High-Density fNIRS and Prompt Tuning for Advanced Human-AI Interaction

Unidirectional brain-computer interface: Artificial neural network encoding natural images to fMRI response in the visual cortex

Neurodm: Decoding and Visualizing Human Brain Activity with Eeg-Guided Diffusion Model

Towards a Passive BCI to Induce Lucid Dream

EEG Based Artistic Visualization of Dreams.

Dream Painter: An Interactive Art Installation Bridging Audience Interaction, Robotics, and Creative AI

Decoding human brain activity with deep learning

Smart City Smart Hospital Smart Home Cloud Server InternetInternet EEG FNIRS MEG Brain Activity IoT Scenarios DL-based Framework

DREAM: Visual Decoding from Reversing Human Visual System

Dream Recording Through Non-invasive Brain-Machine Interfaces and Generative AI-assisted Multimodal Software