Abstract:Sámi, an indigenous language group comprising multiple languages, faces digital marginalization due to the limited availability of data and sophisticated language models designed for its linguistic intricacies. This work focuses on increasing technological participation for the Sámi language. We draw the attention of the ML community towards the language modeling problem of Ultra Low Resource (ULR) languages. ULR languages are those for which the amount of available textual resources is very low, and the speaker count for them is also very low. ULRLs are also not supported by mainstream Large Language Models (LLMs) like ChatGPT, due to which gathering artificial training data for them becomes even more challenging. Mainstream AI foundational model development has given less attention to this category of languages. Generally, these languages have very few speakers, making it hard to find them. However, it is important to develop foundational models for these ULR languages to promote inclusion and the tangible abilities and impact of LLMs. To this end, we have compiled the available Sámi language resources from the web to create a clean dataset for training language models. In order to study the behavior of modern LLM models with ULR languages (Sámi), we have experimented with different kinds of LLMs, mainly at the order of $\sim$ seven billion parameters. We have also explored the effect of multilingual LLM training for ULRLs. We found that the decoder-only models under a sequential multilingual training scenario perform better than joint multilingual training, whereas multilingual training with high semantic overlap, in general, performs better than training from scratch.This is the first study on the Sámi language for adapting non-statistical language models that use the latest developments in the field of natural language processing (NLP).

LLMs for Extremely Low-Resource Finno-Ugric Languages

Do Large Language Models Speak All Languages Equally? A Comparative Study in Low-Resource Settings

Teaching Unseen Low-resource Languages to Large Translation Models

Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts

Too Late to Train, Too Early To Use? A Study on Necessity and Viability of Low-Resource Bengali LLMs

Beyond Metrics: Evaluating LLMs' Effectiveness in Culturally Nuanced, Low-Resource Real-World Scenarios

Challenges in Adapting Multilingual LLMs to Low-Resource Languages using LoRA PEFT Tuning

Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?

Targeted Multilingual Adaptation for Low-resource Language Families

LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages

Shortcomings of LLMs for Low-Resource Translation: Retrieval and Understanding are Both the Problem

Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions

Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings

Improving Multilingual Instruction Finetuning via Linguistically Natural and Diverse Datasets

Towards Multilingual LLM Evaluation for European Languages

Towards a More Inclusive AI: Progress and Perspectives in Large Language Model Training for the Sámi Language

How Vocabulary Sharing Facilitates Multilingualism in LLaMA?

Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer

LLM for Everyone: Representing the Underrepresented in Large Language Models

Vikhr: Constructing a State-of-the-art Bilingual Open-Source Instruction-Following Large Language Model for Russia