Abstract:Story visualization, the task of creating visual narratives from textual descriptions, has seen progress with text-to-image generation models. However, these models often lack effective control over character appearances and interactions, particularly in multi-character scenes. To address these limitations, we propose a new task: \textbf{customized manga generation} and introduce \textbf{DiffSensei}, an innovative framework specifically designed for generating manga with dynamic multi-character control. DiffSensei integrates a diffusion-based image generator with a multimodal large language model (MLLM) that acts as a text-compatible identity adapter. Our approach employs masked cross-attention to seamlessly incorporate character features, enabling precise layout control without direct pixel transfer. Additionally, the MLLM-based adapter adjusts character features to align with panel-specific text cues, allowing flexible adjustments in character expressions, poses, and actions. We also introduce \textbf{MangaZero}, a large-scale dataset tailored to this task, containing 43,264 manga pages and 427,147 annotated panels, supporting the visualization of varied character interactions and movements across sequential frames. Extensive experiments demonstrate that DiffSensei outperforms existing models, marking a significant advancement in manga generation by enabling text-adaptable character customization. The project page is <a class="link-external link-https" href="https://jianzongwu.github.io/projects/diffsensei/" rel="external noopener nofollow">this https URL</a>.

Progressive Deep Feature Learning for Manga Character Recognition Via Unlabeled Training Data

Unsupervised Manga Character Re-identification via Face-body and Spatial-temporal Associated Clustering

Comic Text Detection and Recognition Based on Deep Learning

Facial Landmark Detection for Manga Images

Automating Manga Character Analysis: A Robust Deep Vision-Transformer Approach to Facial Landmark Detection

Dual Loss for Manga Character Recognition with Imbalanced Training Data

Anime Character Identification and Tag Prediction by Multimodality Modeling: Dataset and Model

Building a Manga Dataset "Manga109" with Annotations for Multimedia Applications

MARVEL: Raster Manga Vectorization via Primitive-wise Deep Reinforcement Learning

DiffSensei: Bridging Multi-Modal LLMs and Diffusion Models for Customized Manga Generation

Screentone-Aware Manga Super-Resolution Using DeepLearning

A Multilevel Approach For Recognizing Characters In Engineering Drawings

Zero-Shot Character Identification and Speaker Prediction in Comics via Iterative Multimodal Fusion

Dense Multitask Learning to Reconfigure Comics

Retrieval Based Cartoon Synthesis Via Heterogeneous Features Learning

Learning Gaussian Data Augmentation in Feature Space for One-shot Object Detection in Manga

ACFD: Asymmetric Cartoon Face Detector

Feature Extraction of Ancient Chinese Characters Based on Deep Convolution Neural Network and Big Data Analysis

Unconstrained Text Detection in Manga: a New Dataset and Baseline

Sketch-based manga retrieval using manga109 dataset

Ancient Asian Character Recognition for Literature Preservation and Understanding