Abstract:We present JanusFlow, a powerful framework that unifies image understanding and generation in a single model. JanusFlow introduces a minimalist architecture that integrates autoregressive language models with rectified flow, a state-of-the-art method in generative modeling. Our key finding demonstrates that rectified flow can be straightforwardly trained within the large language model framework, eliminating the need for complex architectural modifications. To further improve the performance of our unified model, we adopt two key strategies: (i) decoupling the understanding and generation encoders, and (ii) aligning their representations during unified training. Extensive experiments show that JanusFlow achieves comparable or superior performance to specialized models in their respective domains, while significantly outperforming existing unified approaches across standard benchmarks. This work represents a step toward more efficient and versatile vision-language models.

What problem does this paper attempt to address?

The paper attempts to address the problem of constructing a unified multimodal model capable of handling both image understanding and generation tasks simultaneously. Specifically, the authors propose the JanusFlow framework, which combines an autoregressive language model with the latest generative modeling method—rectified flow, aiming to achieve efficient and versatile vision-language understanding and generation capabilities within a single model. ### Main Problems and Solutions 1. **Problem**: Existing multimodal models typically require complex architectural designs to handle both image understanding and generation tasks, leading to performance limitations. - **Solution**: JanusFlow simplifies the model architecture by combining an autoregressive language model with rectified flow, eliminating the need for complex design. Additionally, by decoupling the encoders for understanding and generation and aligning their representations during training, the model's performance is further enhanced. 2. **Problem**: Existing unified models often perform worse on specific tasks compared to specialized models. - **Solution**: JanusFlow optimizes the model architecture and training strategy, achieving performance comparable to or better than specialized models on multiple standard benchmarks, particularly excelling in image generation tasks. 3. **Problem**: How to improve the performance of the model on multimodal understanding and generation tasks while keeping it compact. - **Solution**: JanusFlow adopts a lightweight encoder and decoder design, resulting in a model with only 1.3B parameters, yet it still achieves excellent performance across various tasks. ### Key Contributions - **Unified Framework**: JanusFlow is a powerful unified framework capable of handling both image understanding and generation tasks within a single model. - **Simplified Architecture**: By combining an autoregressive language model with rectified flow, JanusFlow simplifies the model architecture, reducing complexity. - **Performance Optimization**: By decoupling the encoders for understanding and generation and aligning their representations during training, the model's performance is significantly improved. - **Experimental Validation**: Extensive experiments demonstrate that JanusFlow achieves performance comparable to or better than specialized models on multimodal understanding and image generation tasks, particularly excelling in image generation. In summary, the paper proposes an innovative multimodal model, JanusFlow, which combines an autoregressive language model and rectified flow to achieve efficient and versatile vision-language understanding and generation capabilities.

JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generation

Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows

TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation

Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flows

TalkingFlow: Talking Facial Landmark Generation with Multi-Scale Normalizing Flow Network

Scaling Rectified Flow Transformers for High-Resolution Image Synthesis

Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

First Multi-Dimensional Evaluation of Flowchart Comprehension for Multimodal Large Language Models

Unifying Flow, Stereo and Depth Estimation.

FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow

Guided Flows for Generative Modeling and Decision Making

SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flow

VectorFlow: Combining Images and Vectors for Traffic Occupancy and Flow Prediction

ContextFlow++: Generalist-Specialist Flow-based Generative Models with Mixed-Variable Context Encoding

VFlow: More Expressive Generative Flows with Variational Data Augmentation

VectorFlow: Combining Images and Vectors for Traffic Occupancy and Flow Prediction

FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding

Flow-Based Unconstrained Lip to Speech Generation

Flow Matching in Latent Space

Nonparametric Generative Modeling with Conditional Sliced-Wasserstein Flows