Abstract:The human ability to easily solve multimodal tasks in context (i.e., with only a few demonstrations or simple instructions), is what current multimodal systems have largely struggled to imitate. In this work, we demonstrate that the task-agnostic in-context learning capabilities of large multimodal models can be significantly enhanced by effective scaling-up. We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences with a unified autoregressive objective. Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning, such as visual prompting and object-grounded generation. The model sets a new record on multiple multimodal understanding tasks in few-shot settings. When instruction-tuned to follow specific instructions, Emu2 further achieves new state-of-the-art on challenging tasks such as question answering benchmarks for large multimodal models and open-ended subject-driven generation. These achievements demonstrate that Emu2 can serve as a base model and general-purpose interface for a wide range of multimodal tasks. Code and models are publicly available to facilitate future research.

What problem does this paper attempt to address?

The paper primarily discusses the significant improvements in in-context learning capabilities of large-scale multimodal pre-trained models, especially through effectively expanding the model size to achieve this goal. The research team introduced Emu2, a generative multimodal model with 37 billion parameters, which is trained on a unified autoregressive objective over large-scale multimodal sequences. Emu2 demonstrated strong multimodal in-context learning abilities, capable of solving tasks that require on-the-fly reasoning, such as visual prompts and object-based generation. It set new records in few-shot settings of various multimodal understanding tasks and achieved new state-of-the-art levels on challenging tasks such as large-scale multimodal model question answering benchmarks and open-ended theme-driven generation after being fine-tuned following specific instructions. The architecture of Emu2 includes three components: a visual encoder, multimodal modeling, and a visual decoder. Images in each input multimodal sequence are transformed into continuous embeddings by the visual encoder, then interleaved with text tokens for autoregressive multimodal modeling. The degraded visual embeddings are subsequently decoded into images or videos by the visual decoder. During the pre-training phase, Emu2 utilized a variety of publicly accessible datasets, including image-text pairs, video-text pairs, interleaved image-text data, interleaved video-text data, location-based image-text pairs, and language-only data, to maintain textual reasoning capabilities. Emu2 showcased its strong potential as a foundational model and universal interface for a wide range of multimodal tasks, particularly in understanding and generation. Moreover, the study delved into the potential societal impacts of Emu2, discussed the possible issues of misuse, and emphasized the importance of continuously enhancing the model and deploying Emu2 responsibly.

Generative Multimodal Models are In-Context Learners

Emu: Generative Pretraining in Multimodality

Multi-Modal Generative Embedding Model

Multimodal Instruction Tuning with Hybrid State Space Models

Model Composition for Multimodal Large Language Models

What Makes Multimodal In-Context Learning Work?

CaMML: Context-Aware Multimodal Learner for Large Models

SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation

Multi-Modal Generative AI: Multi-modal LLM, Diffusion and Beyond

Generative Cross-Modal Retrieval: Memorizing Images in Multimodal Language Models for Retrieval and Beyond

Achieving Cross Modal Generalization with Multimodal Unified Representation.

Emu3: Next-Token Prediction is All You Need

Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages

Unified Generative and Discriminative Training for Multi-modal Large Language Models

Multimodal Sequential Generative Models for Semi-Supervised Language Instruction Following

Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

Scaling autoregressive multi-modal models: Pretraining and instruction tuning

Multimodal Infusion Tuning for Large Models

Image Anything: Towards Reasoning-coherent and Training-free Multi-modal Image Generation

Empowering MultiModal Models' In-Context Learning Ability through Large Language Models.