Abstract:Multi-modal generative AI has received increasing attention in both academia and industry. Particularly, two dominant families of techniques are: i) The multi-modal large language model (MLLM) such as GPT-4V, which shows impressive ability for multi-modal understanding; ii) The diffusion model such as Sora, which exhibits remarkable multi-modal powers, especially with respect to visual generation. As such, one natural question arises: Is it possible to have a unified model for both understanding and generation? To answer this question, in this paper, we first provide a detailed review of both MLLM and diffusion models, including their probabilistic modeling procedure, multi-modal architecture design, and advanced applications to image/video large language models as well as text-to-image/video generation. Then, we discuss the two important questions on the unified model: i) whether the unified model should adopt the auto-regressive or diffusion probabilistic modeling, and ii) whether the model should utilize a dense architecture or the Mixture of Experts(MoE) architectures to better support generation and understanding, two objectives. We further provide several possible strategies for building a unified model and analyze their potential advantages and disadvantages. We also summarize existing large-scale multi-modal datasets for better model pretraining in the future. To conclude the paper, we present several challenging future directions, which we believe can contribute to the ongoing advancement of multi-modal generative AI.

A survey of multimodal deep generative models

Joint Multimodal Learning with Deep Generative Models

Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend

Improving Bi-directional Generation between Different Modalities with Variational Autoencoders

Multimodal Generative Models for Compositional Representation Learning

New Ideas and Trends in Deep Multimodal Content Understanding: A Review

Multimodal Generative Models for Scalable Weakly-Supervised Learning

Deep Multimodal Learning with Missing Modality: A Survey

Generalizing Multimodal Variational Methods to Sets

Multi-Modal Generative AI: Multi-modal LLM, Diffusion and Beyond

Multimodal Large Language Models: A Survey

Multimodal Image Synthesis and Editing: The Generative AI Era

Retrieving Multimodal Information for Augmented Generation: A Survey

Variational methods for Conditional Multimodal Deep Learning

MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning

LLMs Meet Multimodal Generation and Editing: A Survey

A Survey on Deep Learning for Multimodal Data Fusion

Multimodal Adversarially Learned Inference with Factorized Discriminators

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets

Emergent communication of multimodal deep generative models based on Metropolis-Hastings naming game

Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrations