Abstract:Recently, growing interest has been aroused in extending the multimodal capability of large language models (LLMs), e.g., vision-language (VL) learning, which is regarded as the next milestone of artificial general intelligence. However, existing solutions are prohibitively expensive, which not only need to optimize excessive parameters, but also require another large-scale pre-training before VL instruction tuning. In this paper, we propose a novel and affordable solution for the effective VL adaption of LLMs, called Mixture-of-Modality Adaptation (MMA). Instead of using large neural networks to connect the image encoder and LLM, MMA adopts lightweight modules, i.e., adapters, to bridge the gap between LLMs and VL tasks, which also enables the joint optimization of the image and language models. Meanwhile, MMA is also equipped with a routing algorithm to help LLMs achieve an automatic shift between single- and multi-modal instructions without compromising their ability of natural language understanding. To validate MMA, we apply it to a recent LLM called LLaMA and term this formed large vision-language instructed model as LaVIN. To validate MMA and LaVIN, we conduct extensive experiments under two setups, namely multimodal science question answering and multimodal dialogue. The experimental results not only demonstrate the competitive performance and the superior training efficiency of LaVIN than existing multimodal LLMs, but also confirm its great potential as a general-purpose chatbot. More importantly, the actual expenditure of LaVIN is extremely cheap, e.g., only 1.4 training hours with 3.8M trainable parameters, greatly confirming the effectiveness of MMA. Our project is released at https://luogen1996.github.io/lavin.

LaVy: Vietnamese Multimodal Large Language Model

ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models

Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models

Towards Comprehensive Vietnamese Retrieval-Augmented Generation and Large Language Models

Vi-Mistral-X: Building a Vietnamese Language Model with Advanced Continual Pre-training

Vintern-1B: An Efficient Multimodal Large Language Model for Vietnamese

VinaLLaMA: LLaMA-based Vietnamese Foundation Model

Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

LLaVA-SLT: Visual Language Tuning for Sign Language Translation

Evaluating Large Language Model Capability in Vietnamese Fact-Checking Data Generation

Evaluating the Symbol Binding Ability of Large Language Models for Multiple-Choice Questions in Vietnamese General Education

On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study

LLaVaOLMoBitnet1B: Ternary LLM goes Multimodal!

LLaVA-KD: A Framework of Distilling Multimodal Large Language Models

VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models

LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model

VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks

A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks

VNHSGE: VietNamese High School Graduation Examination Dataset for Large Language Models