Abstract:Multimodal Large Language Models (MLLMs), by expanding the model's capabilities to perceive and interact through multi-modalities, have significantly enhanced performance across various tasks. The perception of vision is an important modality developed into LLM, enabling research in vision-language to continuously lead the cutting-edge advancements in the MLLM community. However, the standard pre-training pipeline on image-text pairs results in a limited model understanding of relationships between multiple images and texts, as well as visual details. Additionally, the setting of fine-tuning with a frozen visual backbone hinders the enhancement of visual representations on new data. These two issues lead to suboptimal performance in models for demonstrative instruction following about multiple images. This work introduces a novel framework called MLoEM, which first converts long multimodal data into an interleaved image-instruction format, and then adopts a fully autoregressive architecture model, allowing for more robust and coherent learning from naturally occurring multimodal documents than pair-based pipeline. Additionally, we incorporate the Low-Rank Adaptation (LoRA) fine-tuning method, enhancing visual representations while maintaining the stability of previously learned knowledge. Finally, we utilize ensemble methods to enhance model performance on tasks. To alleviate the storage overhead issue of parallel ensembles with large models, we design an ensemble approach that shares the MLLM while only switching the LoRA matrices. In the experiments, the proposed MLoEM shows superior performance on the testing set.

Multimodal Instruction Tuning with Conditional Mixture of LoRA

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning

CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models

MultiLoRA: Democratizing LoRA for Better Multi-Task Learning

A Framework to Implement 1+N Multi-task Fine-tuning Pattern in LLMs Using the CGC-LORA Algorithm

Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning

MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning

Demystifying Instruction Mixing for Fine-tuning Large Language Models

MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

Visual Cue Enhancement and Dual Low-Rank Adaptation for Efficient Visual Instruction Fine-Tuning

MoR: Mixture of Ranks for Low-Rank Adaptation Tuning

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs

Demonstrative Instruction Following in Multimodal LLMs Via Integrating Low-Rank Adaptation with Ensemble Learning

M^2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

Chain-of-LoRA: Enhancing the Instruction Fine-Tuning Performance of Low-Rank Adaptation on Diverse Instruction Set

M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

Lateralization LoRA: Interleaved Instruction Tuning with Modality-Specialized Adaptations