Abstract:Large language models have exhibited significant proficiency in languages endowed with extensive linguistic resources, such as English and Chinese. Nevertheless, their effectiveness notably diminishes when applied to languages characterized by limited linguistic resources, particularly within the Southeast Asian linguistic landscape, such as Indonesian. The scarcity of linguistic resources for these languages presents challenges associated with inadequate training, restricted vocabulary coverage, and challenging evaluation processes. In response to these exigencies, we have introduced CompassLLM, a large multilingual model specifically tailored for Southeast Asian languages, with the primary aim of supporting the developmental requirements of Shopee. Our methodology encompasses several key strategies. To progressively enhance multilingual proficiencies, we implemented a multi-stage pre-training strategy integrated with curriculum learning, gradually intensifying the focus on low-resource languages. Concurrently, to better accommodate low-resource human instructions, we curated and generated a repository of high-quality multilingual human instructions, culminating the CompassLLM-SFT model through supervised instruction fine-tuning. Finally, to reinforce the model's alignment with human preference behaviors, we have embraced the principle of Direct Preference Optimization (DPO) to obtain CompassLLM-DPO model. Preliminary evaluation of the CompassLLM model yields promising results, with our model surpassing benchmark models like Vicuna-7b-v1.5, Sealion, Falcon and SeaLLM, across diverse evaluation tasks, as verified through both automated and human-driven assessments. Notably, our model exhibits its superior performance in South-east Asia languages, such as Indonesian language.

COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances

IndoCulture: Exploring Geographically-Influenced Cultural Commonsense Reasoning Across Eleven Indonesian Provinces

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

Constructing and Expanding Low-Resource and Underrepresented Parallel Datasets for Indonesian Local Languages

IndoNLI: A Natural Language Inference Dataset for Indonesian

NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages

NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages

Analysis of language identification algorithms for regional Indonesian languages

Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese

Komodo: A Linguistic Expedition into Indonesia's Regional Languages

One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia

Multi-lingual and Multi-cultural Figurative Language Understanding

IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP

IndoToD: A Multi-Domain Indonesian Benchmark For End-to-End Task-Oriented Dialogue Systems

IndoRobusta: Towards Robustness Against Diverse Code-Mixed Indonesian Local Languages

Building Dialogue Understanding Models for Low-resource Language Indonesian from Scratch

IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding

Compass: Large Multilingual Language Model for South-east Asia

SNLI Indo: A recognizing textual entailment dataset in Indonesian derived from the Stanford Natural Language Inference dataset

Exploring Reasoning Biases in Large Language Models Through Syllogism: Insights from the NeuBAROCO Dataset

Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia