ALLaM: Large Language Models for Arabic and English
M Saiful Bari,Yazeed Alnumay,Norah A. Alzahrani,Nouf M. Alotaibi,Hisham A. Alyahya,Sultan AlRashed,Faisal A. Mirza,Shaykhah Z. Alsubaie,Hassan A. Alahmed,Ghadah Alabduljabbar,Raghad Alkhathran,Yousef Almushayqih,Raneem Alnajim,Salman Alsubaihi,Maryam Al Mansour,Majed Alrubaian,Ali Alammari,Zaki Alawami,Abdulmohsen Al-Thubaity,Ahmed Abdelali,Jeril Kuriakose,Abdalghani Abujabal,Nora Al-Twairesh,Areeb Alowisheq,Haidar Khan
DOI: https://doi.org/10.48550/arxiv.2407.15390
2024-07-22
Computation and Language
Abstract:We present ALLaM: Arabic Large Language Model, a series of large language models to support the ecosystem of Arabic Language Technologies (ALT). ALLaM is carefully trained considering the values of language alignment and knowledge transfer at scale. Our autoregressive decoder-only architecture models demonstrate how second-language acquisition via vocabulary expansion and pretraining on a mixture of Arabic and English text can steer a model towards a new language (Arabic) without any catastrophic forgetting in the original language (English). Furthermore, we highlight the effectiveness of using parallel/translated data to aid the process of knowledge alignment between languages. Finally, we show that extensive alignment with human preferences can significantly enhance the performance of a language model compared to models of a larger scale with lower quality alignment. ALLaM achieves state-of-the-art performance in various Arabic benchmarks, including MMLU Arabic, ACVA, and Arabic Exams. Our aligned models improve both in Arabic and English from their base aligned models.