Abstract:Sámi, an indigenous language group comprising multiple languages, faces digital marginalization due to the limited availability of data and sophisticated language models designed for its linguistic intricacies. This work focuses on increasing technological participation for the Sámi language. We draw the attention of the ML community towards the language modeling problem of Ultra Low Resource (ULR) languages. ULR languages are those for which the amount of available textual resources is very low, and the speaker count for them is also very low. ULRLs are also not supported by mainstream Large Language Models (LLMs) like ChatGPT, due to which gathering artificial training data for them becomes even more challenging. Mainstream AI foundational model development has given less attention to this category of languages. Generally, these languages have very few speakers, making it hard to find them. However, it is important to develop foundational models for these ULR languages to promote inclusion and the tangible abilities and impact of LLMs. To this end, we have compiled the available Sámi language resources from the web to create a clean dataset for training language models. In order to study the behavior of modern LLM models with ULR languages (Sámi), we have experimented with different kinds of LLMs, mainly at the order of $\sim$ seven billion parameters. We have also explored the effect of multilingual LLM training for ULRLs. We found that the decoder-only models under a sequential multilingual training scenario perform better than joint multilingual training, whereas multilingual training with high semantic overlap, in general, performs better than training from scratch.This is the first study on the Sámi language for adapting non-statistical language models that use the latest developments in the field of natural language processing (NLP).

Small Languages, Big Models: A Study of Continual Training on Languages of Norway

Large-Scale Contextualised Language Modelling for Norwegian

Towards a More Inclusive AI: Progress and Perspectives in Large Language Model Training for the Sámi Language

Continual Learning Under Language Shift

Danish Foundation Models

NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian

GPT-SW3: An Autoregressive Language Model for the Nordic Languages

Language Resources for Dutch Large Language Modelling

CodeGen2: Lessons for Training LLMs on Programming and Natural Languages

Continual Learning for Large Language Models: A Survey

Simple and Scalable Strategies to Continually Pre-train Large Language Models

Poro 34B and the Blessing of Multilinguality

Large Language Models Demonstrate the Potential of Statistical Learning in Language

Danoliteracy of Generative, Large Language Models

The Nordic Pile: A 1.2TB Nordic Dataset for Language Modeling

Large Language Models aren't all that you need

Large Language Models Meet NL2Code: A Survey

The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective

Larger-Scale Transformers for Multilingual Masked Language Modeling

SambaLingo: Teaching Large Language Models New Languages