Abstract:The development of large language models (LLMs) has expanded to multi-modal systems capable of processing text, images, and speech within a unified framework. Training these models demands significantly larger datasets and computational resources compared to text-only LLMs. To address the scaling challenges, we introduce Mixture-of-Transformers (MoT), a sparse multi-modal transformer architecture that significantly reduces pretraining computational costs. MoT decouples non-embedding parameters of the model by modality -- including feed-forward networks, attention matrices, and layer normalization -- enabling modality-specific processing with global self-attention over the full input sequence. We evaluate MoT across multiple settings and model scales. In the Chameleon 7B setting (autoregressive text-and-image generation), MoT matches the dense baseline's performance using only 55.8\% of the FLOPs. When extended to include speech, MoT reaches speech performance comparable to the dense baseline with only 37.2\% of the FLOPs. In the Transfusion setting, where text and image are trained with different objectives, a 7B MoT model matches the image modality performance of the dense baseline with one third of the FLOPs, and a 760M MoT model outperforms a 1.4B dense baseline across key image generation metrics. System profiling further highlights MoT's practical benefits, achieving dense baseline image quality in 47.2\% of the wall-clock time and text quality in 75.6\% of the wall-clock time (measured on AWS p4de.24xlarge instances with NVIDIA A100 GPUs).

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the challenges of computational resources and dataset scale faced by multimodal large language models (LLMs) during training. Specifically: 1. **Complexity of Multimodal Processing**: - Multimodal LLMs need to handle various data types such as text, images, and speech, which significantly increases the complexity and difficulty of training the model. - Each modality (e.g., text, image, speech) exhibits different characteristics in the feature space, requiring effective processing within a unified framework. 2. **Demand for Computational Resources**: - Training multimodal LLMs requires larger datasets and higher computational resources. For example, the Chameleon model requires 9.2 trillion training tokens (including image tokens), while the text-only LLaMA2 requires only 2 trillion training tokens. - The optimization challenges for different modalities vary, necessitating the simultaneous resolution of these challenges within a unified model, increasing the complexity and computational load of optimization. 3. **Model Sparsity**: - To address the above challenges, the paper proposes a sparse multimodal transformer architecture—Mixture-of-Transformers (MoT). MoT decouples the non-embedding parameters of different modalities (such as feedforward networks, attention matrices, and layer normalization), enabling modality-specific processing while maintaining a global self-attention mechanism. - This design reduces the computational cost of pre-training and improves the efficiency and performance of the model. ### Solution Overview - **Mixture-of-Transformers (MoT) Architecture**: - MoT decouples the non-embedding parameters of different modalities, allowing each modality to be processed using specialized parameters while maintaining a global self-attention mechanism. - This design enables MoT to significantly reduce the consumption of computational resources when processing multimodal data, improving the efficiency of training and inference. - **Experimental Validation**: - The paper validates the performance of MoT on different modalities and tasks through multiple experimental setups. The experimental results show that MoT can achieve performance comparable to or better than dense models with significantly fewer computational resources across various model scales. - Specific experiments include: - **Chameleon Setup**: Generating text and images under an autoregressive objective. - **Chameleon + Speech Setup**: Introducing speech as the third modality. - **Transfusion Setup**: Generating text using an autoregressive objective and generating images using a diffusion model. ### Main Contributions - **Improved Computational Efficiency**: - MoT significantly reduces the required FLOPs (floating-point operations) in multiple experimental setups, thereby improving the efficiency of training and inference. - For example, at the 7B parameter scale, MoT achieves performance comparable to dense models in the Chameleon setup using only 55.8% of the FLOPs. - **Performance Advantages**: - MoT performs excellently across multiple modalities and tasks, particularly outperforming traditional dense models and Mixture-of-Experts (MoE) models in handling images and speech. - Experimental results also show that MoT has a significant advantage in wall-clock time (actual training time). - **Flexibility and Scalability**: - The design of the MoT architecture allows it to flexibly handle various modalities and can be easily scaled to larger model sizes. Through these innovations, the paper provides new solutions for the efficient training and application of multimodal LLMs.

Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models

p-MoD: Building Mixture-of-Depths MLLMs via Progressive Ratio Decay

Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules

MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

$γ-$MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models

Cross-token Modeling with Conditional Computation

Mixture of Tokens: Continuous MoE through Cross-Example Aggregation

MoDeGPT: Modular Decomposition for Large Language Model Compression

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing

MoEfication: Conditional Computation of Transformer Models for Efficient Inference

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More

Matryoshka Query Transformer for Large Vision-Language Models

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

MoDification: Mixture of Depths Made Easy

TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones

Multimodal Infusion Tuning for Large Models

Llama 3 Meets MoE: Efficient Upcycling