X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment

Dongjae Shin,Hyeonseok Lim,Inho Won,Changsu Choi,Minjun Kim,Seungwoo Song,Hangyeol Yoo,Sangmin Kim,Kyungtae Lim

2024-04-01

Abstract:The impressive development of large language models (LLMs) is expanding into the realm of large multimodal models (LMMs), which incorporate multiple types of data beyond text. However, the nature of multimodal models leads to significant expenses in the creation of training data. Furthermore, constructing multilingual data for LMMs presents its own set of challenges due to language diversity and complexity. Therefore, in this study, we propose two cost-effective methods to solve this problem: (1) vocabulary expansion and pretraining of multilingual LLM for specific languages, and (2) automatic and elaborate construction of multimodal datasets using GPT4-V. Based on015 these methods, we constructed a 91K English-Korean-Chinese multilingual, multimodal training dataset. Additionally, we developed a bilingual multimodal model that exhibits excellent performance in both Korean and English, surpassing existing approaches.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address the costly issue faced by large multimodal models (LMMs) during the creation of training data, especially the challenges of linguistic diversity and complexity when dealing with multilingual data. Specifically, the paper proposes two cost-effective methods to tackle these problems: 1. **Vocabulary Expansion and Language-Specific Pretraining**: Expanding the vocabulary of the target language and conducting pretraining that connects knowledge from multiple languages. 2. **Automatic Construction of Multimodal Datasets**: Utilizing GPT4-V to automatically and comprehensively construct multimodal datasets. Based on these two methods, the researchers constructed a trilingual multimodal training dataset in English, Korean, and Chinese, containing 91,000 samples, and developed a bilingual multimodal model X-LLaV A that performs excellently in both Korean and English, surpassing existing methods. Experimental results show that the X-LLaV A model achieves an average improvement of approximately 5.2% in three Korean quantitative evaluations and also achieves the highest performance in two English quantitative evaluations. Qualitative evaluations indicate that the responses generated by this model are superior in 19%-93% of cases compared to existing models. Additionally, the study emphasizes that the constructed dataset and model can be realized with relatively small resources, costing approximately $3,200 and using an A6000 GPU.

X-LLaVA: Optimizing Bilingual Large Vision-Language Alignment

Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean

Xmodel-VLM: A Simple Baseline for Multimodal Vision Language Model

Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners

u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model

BayLing 2: A Multilingual Large Language Model with Efficient Language Alignment

Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

X-VILA: Cross-Modality Alignment for Large Language Model

EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model

Large-scale Bilingual Language-Image Contrastive Learning

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

Lens: Rethinking Multilingual Enhancement for Large Language Models

SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecture

Exploring Large Language Models on Cross-Cultural Values in Connection with Training Methodology

Extrapolating Large Language Models to Non-English by Aligning Languages

LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models

Unified Generative and Discriminative Training for Multi-modal Large Language Models

AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models

Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training