Deploying Technology to Save Endangered Languages

Hilaria Cruz,Joseph Waring

DOI: https://doi.org/10.48550/arXiv.1908.08971

2019-08-23

Computation and Language

Abstract:Computer scientists working on natural language processing, native speakers of endangered languages, and field linguists to discuss ways to harness Automatic Speech Recognition, especially neural networks, to automate annotation, speech tagging, and text parsing on endangered languages.

What problem does this paper attempt to address?

Harnessing the Power of Artificial Intelligence to Vitalize Endangered Indigenous Languages: Technologies and Experiences

Claudio Pinhanez,Paulo Cavalin,Luciana Storto,Thomas Finbow,Alexander Cobbinah,Julio Nogima,Marisa Vasconcelos,Pedro Domingues,Priscila de Souza Mizukami,Nicole Grell,Majoí Gongora,Isabel Gonçalves

2024-07-30

Abstract:Since 2022 we have been exploring application areas and technologies in which Artificial Intelligence (AI) and modern Natural Language Processing (NLP), such as Large Language Models (LLMs), can be employed to foster the usage and facilitate the documentation of Indigenous languages which are in danger of disappearing. We start by discussing the decreasing diversity of languages in the world and how working with Indigenous languages poses unique ethical challenges for AI and NLP. To address those challenges, we propose an alternative development AI cycle based on community engagement and usage. Then, we report encouraging results in the development of high-quality machine learning translators for Indigenous languages by fine-tuning state-of-the-art (SOTA) translators with tiny amounts of data and discuss how to avoid some common pitfalls in the process. We also present prototypes we have built in projects done in 2023 and 2024 with Indigenous communities in Brazil, aimed at facilitating writing, and discuss the development of Indigenous Language Models (ILMs) as a replicable and scalable way to create spell-checkers, next-word predictors, and similar tools. Finally, we discuss how we envision a future for language documentation where dying languages are preserved as interactive language models.

Computation and Language,Artificial Intelligence
Not always about you: Prioritizing community needs when developing endangered language technology

Zoey Liu,Crystal Richardson,Emily Prud'hommeaux,Richard Hatcher Jr

DOI: https://doi.org/10.48550/arXiv.2204.05541

2022-04-12

Computation and Language

Abstract:Languages are classified as low-resource when they lack the quantity of data necessary for training statistical and machine learning tools and models. Causes of resource scarcity vary but can include poor access to technology for developing these resources, a relatively small population of speakers, or a lack of urgency for collecting such resources in bilingual populations where the second language is high-resource. As a result, the languages described as low-resource in the literature are as different as Finnish on the one hand, with millions of speakers using it in every imaginable domain, and Seneca, with only a small-handful of fluent speakers using the language primarily in a restricted domain. While issues stemming from the lack of resources necessary to train models unite this disparate group of languages, many other issues cut across the divide between widely-spoken low resource languages and endangered languages. In this position paper, we discuss the unique technological, cultural, practical, and ethical challenges that researchers and indigenous speech community members face when working together to develop language technology to support endangered language documentation and revitalization. We report the perspectives of language teachers, Master Speakers and elders from indigenous communities, as well as the point of view of academics. We describe an ongoing fruitful collaboration and make recommendations for future partnerships between academic researchers and language community stakeholders.
Revitalizing Endangered Languages: AI-powered language learning as a catalyst for language appreciation

Dinesh Kumar Nanduri,Elizabeth M. Bonsignore

DOI: https://doi.org/10.48550/arXiv.2304.09394

IF: 6.4588

2023-04-19

Human-Computer Interaction

Abstract:According to UNESCO, there are nearly 7,000 languages spoken worldwide, of which around 3,000 languages are in danger of disappearing before the end of the century. With roughly 230 languages having already become extinct between the years 1950-2010, collectively this represents a significant loss of linguistic and cultural diversity. This position paper aims to explore the potential of AI-based language learning approaches that promote early exposure and appreciation of languages to ultimately contribute to the preservation of endangered languages, thereby addressing the urgent need to protect linguistic and cultural diversity.
Endangered Languages: A Sociocognitive Approach to Language Death, Identity Loss, and Preservation in the Age of Artificial Intelligence

Dylan Scott Low,Isaac Mcneill,Michael James Day

DOI: https://doi.org/10.2478/sm-2022-0011

2022-12-01

Sustainable Multilingualism

Abstract:Summary Sociolinguists suggest language death entails significant cultural, personal, and ecological loss. Socio-cultural and socio-political factors exacerbate language erosion and encourage supplantation by another more dominant language. Hence, we ask: what are the sociocognitive principles which make language death hurtful and symbolic? Within this article, we attempt to outline a sociocognitive account of language death, situating the Hallidayan perspective of language as a “social-semiotic” system alongside a Cognitive Linguistic approach. We further contextualise language as inseparable from culture, drawing insight from the sociological thought of Bourdieu. We contend that language death entails psychological trauma, representing the destruction of cultural genealogy and the loss of knowledge intrinsic to personal self-imagery and identity. To this end, we present a case study of the Māori languaculture in Aotearoa (New Zealand), tracing the impact of colonialism and marginalisation to current efforts and ambitions to ensure the languacultural survival of Māori and reclaim space in Aotearoa as a respected knowledge system and means of expression, particularly in the socio-technical age of artificial intelligence (AI) and the Web. We argue that our analysis bodes practical implications for language maintenance and revitalisation, concluding that sociolinguistic practitioners should consider a socio-cognitivist as well as socio-technical paradigm for language intervention. In closing, we discuss leveraging AI technologies towards language heritage, archival, and preservation to limit the destructive impact of the death of a language.
NLP Progress in Indigenous Latin American Languages

Atnafu Lambebo Tonja,Fazlourrahman Balouchzahi,Sabur Butt,Olga Kolesnikova,Hector Ceballos,Alexander Gelbukh,Thamar Solorio

2024-05-12

Abstract:The paper focuses on the marginalization of indigenous language communities in the face of rapid technological advancements. We highlight the cultural richness of these languages and the risk they face of being overlooked in the realm of Natural Language Processing (NLP). We aim to bridge the gap between these communities and researchers, emphasizing the need for inclusive technological advancements that respect indigenous community perspectives. We show the NLP progress of indigenous Latin American languages and the survey that covers the status of indigenous languages in Latin America, their representation in NLP, and the challenges and innovations required for their preservation and development. The paper contributes to the current literature in understanding the need and progress of NLP for indigenous communities of Latin America, specifically low-resource and indigenous communities in general.

Computation and Language
"It's how you do things that matters": Attending to Process to Better Serve Indigenous Communities with Language Technologies

Ned Cooper,Courtney Heldreth,Ben Hutchinson

2024-02-06

Abstract:Indigenous languages are historically under-served by Natural Language Processing (NLP) technologies, but this is changing for some languages with the recent scaling of large multilingual models and an increased focus by the NLP community on endangered languages. This position paper explores ethical considerations in building NLP technologies for Indigenous languages, based on the premise that such projects should primarily serve Indigenous communities. We report on interviews with 17 researchers working in or with Aboriginal and/or Torres Strait Islander communities on language technology projects in Australia. Drawing on insights from the interviews, we recommend practices for NLP researchers to increase attention to the process of engagements with Indigenous communities, rather than focusing only on decontextualised artefacts.

Computation and Language
Living Dictionaries: A Platform for Indigenous and Under-Resourced Languages

Anna Luisa Daigneault,Gregory D. S. Anderson

DOI: https://doi.org/10.1353/dic.2023.a915065

2023-01-01

Abstract:ABSTRACT: Due to globalization, cultural assimilation, the long-term impacts of colonization, and official (or de facto ) policies hostile to linguistic diversity, many languages of the world are threatened or endangered. Free online technological tools can assist in documentation efforts and revitalization programs, while also providing safe online spaces in which materials can be systematically recorded and shared. Led by community activists and linguists, Living Dictionaries are collaborative multimedia projects that are editable, expandable, and searchable. Using the latest web technologies to facilitate the creation and storage of language data paired with digital images, audio and video recordings of native speakers, the platform is user-friendly and available in fourteen interface languages. Living Dictionaries promote visibility for under-resourced languages, foster connectivity over vast distances, and support an online community of language learners who wish to access a language without proximity to proficient speakers. The Living Dictionaries platform provides meaningful opportunities for Indigenous citizen science through regular training webinars.
Challenges of language technologies for the indigenous languages of the Americas

Manuel Mager,Ximena Gutierrez-Vasques,Gerardo Sierra,Ivan Meza

DOI: https://doi.org/10.48550/arXiv.1806.04291

2018-06-12

Abstract:Indigenous languages of the American continent are highly diverse. However, they have received little attention from the technological perspective. In this paper, we review the research, the digital resources and the available NLP systems that focus on these languages. We present the main challenges and research questions that arise when distant languages and low-resource scenarios are faced. We would like to encourage NLP research in linguistically rich and diverse areas like the Americas.

Computation and Language
Exploring the Role of Artificial Intelligence in Language Documentation and Endangered Language Preservation.

Soumi Ray,

DOI: https://doi.org/10.52783/tjjpt.v45.i02.6122

2024-04-12

Abstract:This paper explores the intersection of artificial intelligence (AI) and language preservation efforts, focusing on the documentation and preservation of endangered languages. With over 40% of the world's languages facing extinction, the need for innovative approaches to language documentation and preservation is urgent. Recent advancements in AI, including machine learning, natural language processing, and speech recognition, offer promising solutions to address the challenges faced by linguists and communities in preserving linguistic diversity. Through a comprehensive review of literature and case studies, this paper examines the role of AI in automating data collection, linguistic analysis, transcription, and even language revitalization efforts. Additionally, it discusses ethical considerations such as data bias, privacy, and cultural sensitivity in the application of AI technologies for language preservation. By highlighting the potential benefits and challenges of AI in this context, this paper aims to inform future research and practice in the field of endangered language preservation. DOI: https://doi.org/10.52783/tjjpt.v45.i02.6122
Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions

Kexun Zhang,Yee Man Choi,Zhenqiao Song,Taiqi He,William Yang Wang,Lei Li

2024-02-28

Abstract:How can large language models (LLMs) process and translate endangered languages? Many languages lack a large corpus to train a decent LLM; therefore existing LLMs rarely perform well in unseen, endangered languages. On the contrary, we observe that 2000 endangered languages, though without a large corpus, have a grammar book or a dictionary. We propose LINGOLLM, a training-free approach to enable an LLM to process unseen languages that hardly occur in its pre-training. Our key insight is to demonstrate linguistic knowledge of an unseen language in an LLM's prompt, including a dictionary, a grammar book, and morphologically analyzed input text. We implement LINGOLLM on top of two models, GPT-4 and Mixtral, and evaluate their performance on 5 tasks across 8 endangered or low-resource languages. Our results show that LINGOLLM elevates translation capability from GPT-4's 0 to 10.5 BLEU for 10 language directions. Our findings demonstrate the tremendous value of linguistic knowledge in the age of LLMs for endangered languages. Our data, code, and model generations can be found at

Computation and Language
Shaping the Future of Endangered and Low-Resource Languages -- Our Role in the Age of LLMs: A Keynote at ECIR 2024

Josiane Mothe

DOI: https://doi.org/10.1145/3687273.3687280

2024-09-05

Abstract:Isidore of Seville is credited with the adage that it is language that gives birth to a people, and not the other way around , underlining the profound role played by language in the formation of cultural and social identity. Today, of the more than 7100 languages listed, a significant number are endangered. Since the 1970s, linguists, information seekers and enthusiasts have helped develop digital resources and automatic tools to support a wide range of languages, including endangered ones. The advent of Large Language Model (LLM) technologies holds both promise and peril. They offer unprecedented possibilities for the translation and generation of content and resources, key elements in the preservation and revitalisation of languages. They also present threat of homogenisation, cultural oversimplification and the further marginalisation of already vulnerable languages. The talk this paper is based on has proposed an initiatory journey, exploring the potential paths and partnerships between technology and tradition, with a particular focus on the Occitan language. Occitan is a language from Southern France, parts of Spain and Italy that played a major cultural and economic role, particularly in the Middle Ages. It is now endangered according to UNESCO. The talk critically has examined how human expertise and artificial intelligence can work together to offer hope for preserving the linguistic diversity that forms the foundation of our global and especially our European heritage while addressing some of the ethical and practical challenges that accompany the use of these powerful technologies. This paper is based on the keynote I gave at the 46th European Conference on Information Retrieval (ECIR 2024). As an alternative to reading this paper, a video talk is available online. 1 Date: 26 March 2024.

Computation and Language,Information Retrieval
A Summary of the First Workshop on Language Technology for Language Documentation and Revitalization

Graham Neubig,Shruti Rijhwani,Alexis Palmer,Jordan MacKenzie,Hilaria Cruz,Xinjian Li,Matthew Lee,Aditi Chaudhary,Luke Gessler,Steven Abney,Shirley Anugrah Hayati,Antonios Anastasopoulos,Olga Zamaraeva,Emily Prud'hommeaux,Jennette Child,Sara Child,Rebecca Knowles,Sarah Moeller,Jeffrey Micher,Yiyuan Li,Sydney Zink,Mengzhou Xia,Roshan S Sharma,Patrick Littell

DOI: https://doi.org/10.48550/arXiv.2004.13203

2020-04-28

Abstract:Despite recent advances in natural language processing and other language technology, the application of such technology to language documentation and conservation has been limited. In August 2019, a workshop was held at Carnegie Mellon University in Pittsburgh to attempt to bring together language community members, documentary linguists, and technologists to discuss how to bridge this gap and create prototypes of novel and practical language revitalization technologies. This paper reports the results of this workshop, including issues discussed, and various conceived and implemented technologies for nine languages: Arapaho, Cayuga, Inuktitut, Irish Gaelic, Kidaw'ida, Kwak'wala, Ojibwe, San Juan Quiahije Chatino, and Seneca.

Computation and Language
Endangered languages, social subjects and mediatization:

Eliane Fernandes Azzari

DOI: https://doi.org/10.31686/ijier.vol7.iss8.1671

2019-08-31

International Journal for Innovation Education and Research

Abstract:This paper discusses possible roles of midia/midiatization in the face of processes of linguistic changes and/in endagered languages in contemporary society. It is grounded on a social and historical approach to language and it finds support in Bakhtinian propositions about discourse, subjects and chronotopes. It assumes a sociolinguistic perspective to the interface between linguistic landscape, digital media and discursive agency. The study adopts the digital ethnography as a methodological procedure that grants the investigation of human actions and interactions in digital contexts. In order to explore the target theme, it presents a brief analysis of the case of Wikitongues – a non-profit organization that offers several digital platforms/media for individuals to share, divulge and comment on endangered language and cultural diversity. The analysis suggests that, by resorting to free online digital spaces and their affordances, spontaneous and activist video communities were created, showing that mediatization processes supported by the internet might help promote language diversity, discursive agency and cultural awareness.
What shall we do with an hour of data? Speech recognition for the un- and under-served languages of Common Voice

Francis M. Tyers,Josh Meyer

DOI: https://doi.org/10.48550/arXiv.2105.04674

2021-05-11

Abstract:This technical report describes the methods and results of a three-week sprint to produce deployable speech recognition models for 31 under-served languages of the Common Voice project. We outline the preprocessing steps, hyperparameter selection, and resulting accuracy on official testing sets. In addition to this we evaluate the models on multiple tasks: closed-vocabulary speech recognition, pre-transcription, forced alignment, and key-word spotting. The following experiments use Coqui STT, a toolkit for training and deployment of neural Speech-to-Text models.

Computation and Language,Machine Learning,Sound,Audio and Speech Processing
Local Translation Services for Neglected Languages

David Noever,Josh Kalin,Matthew Ciolino,Dom Hambrick,Gerry Dozier

DOI: https://doi.org/10.5121/csit.2021.110110

2021-01-23

Abstract:Taking advantage of computationally lightweight, but high-quality translators prompt consideration of new applications that address neglected languages. For projects with protected or personal data, translators for less popular or low-resource languages require specific compliance checks before posting to a public translation API. In these cases, locally run translators can render reasonable, cost-effective solutions if done with an army of offline, smallscale pair translators. Like handling a specialist’s dialect, this research illustrates translating two historically interesting, but obfuscated languages: 1) hacker-speak (“l33t”) and 2) reverse (or “mirror”) writing as practiced by Leonardo da Vinci. The work generalizes a deep learning architecture to translatable variants of hacker-speak with lite, medium, and hard vocabularies. The original contribution highlights a fluent translator of hacker-speak in under 50 megabytes and demonstrates a companion text generator for augmenting future datasets with greater than a million bilingual sentence pairs. A primary motivation stems from the need to understand and archive the evolution of the international computer community, one that continuously enhances their talent for speaking openly but in hidden contexts. This training of bilingual sentences supports deep learning models using a long short-term memory, recurrent neural network (LSTM-RNN). It extends previous work demonstrating an English-to-foreign translation service built from as little as 10,000 bilingual sentence pairs. This work further solves the equivalent translation problem in twenty-six additional (non-obfuscated) languages and rank orders those models and their proficiency quantitatively with Italian as the most successful and Mandarin Chinese as the most challenging. For neglected languages, the method prototypes novel services for smaller niche translations such as Kabyle (Algerian dialect) which covers between 5-7 million speakers but one which for most enterprise translators, has not yet reached development. One anticipates the extension of this approach to other important dialects, such as translating technical (medical or legal) jargon and processing health records or handling many of the dialects collected from specialized domains (mixed languages like “Spanglish”, acronym-laden Twitter feeds, or urban slang).
Is Machine Learning Speaking my Language? A Critical Look at the NLP-Pipeline Across 8 Human Languages

Esma Wali,Yan Chen,Christopher Mahoney,Thomas Middleton,Marzieh Babaeianjelodar,Mariama Njie,Jeanna Neefe Matthews

DOI: https://doi.org/10.48550/arXiv.2007.05872

2020-07-11

Computation and Language

Abstract:Natural Language Processing (NLP) is increasingly used as a key ingredient in critical decision-making systems such as resume parsers used in sorting a list of job candidates. NLP systems often ingest large corpora of human text, attempting to learn from past human behavior and decisions in order to produce systems that will make recommendations about our future world. Over 7000 human languages are being spoken today and the typical NLP pipeline underrepresents speakers of most of them while amplifying the voices of speakers of other languages. In this paper, a team including speakers of 8 languages - English, Chinese, Urdu, Farsi, Arabic, French, Spanish, and Wolof - takes a critical look at the typical NLP pipeline and how even when a language is technically supported, substantial caveats remain to prevent full participation. Despite huge and admirable investments in multilingual support in many tools and resources, we are still making NLP-guided decisions that systematically and dramatically underrepresent the voices of much of the world.
Meta AI system is a boost to endangered languages — as long as humans aren't forgotten

DOI: https://doi.org/10.1038/d41586-024-01619-y

IF: 64.8

2024-06-06

Nature

Abstract:Automated approaches to translation could provide a lifeline to under-resourced languages, but only if companies engage with the people who speak them. Automated approaches to translation could provide a lifeline to under-resourced languages, but only if companies engage with the people who speak them.

multidisciplinary sciences
OCR Post Correction for Endangered Language Texts

Shruti Rijhwani,Antonios Anastasopoulos,Graham Neubig

DOI: https://doi.org/10.48550/arXiv.2011.05402

2020-11-11

Abstract:There is little to no data available to build natural language processing models for most endangered languages. However, textual data in these languages often exists in formats that are not machine-readable, such as paper books and scanned images. In this work, we address the task of extracting text from these resources. We create a benchmark dataset of transcriptions for scanned books in three critically endangered languages and present a systematic analysis of how general-purpose OCR tools are not robust to the data-scarce setting of endangered languages. We develop an OCR post-correction method tailored to ease training in this data-scarce setting, reducing the recognition error rate by 34% on average across the three languages.

Computation and Language
Towards a General-Purpose Linguistic Annotation Backend

Graham Neubig,Patrick Littell,Chian-Yu Chen,Jean Lee,Zirui Li,Yu-Hsiang Lin,Yuyan Zhang

DOI: https://doi.org/10.48550/arXiv.1812.05272

2018-12-13

Abstract:Language documentation is inherently a time-intensive process; transcription, glossing, and corpus management consume a significant portion of documentary linguists' work. Advances in natural language processing can help to accelerate this work, using the linguists' past decisions as training material, but questions remain about how to prioritize human involvement. In this extended abstract, we describe the beginnings of a new project that will attempt to ease this language documentation process through the use of natural language processing (NLP) technology. It is based on (1) methods to adapt NLP tools to new languages, based on recent advances in massively multilingual neural networks, and (2) backend APIs and interfaces that allow linguists to upload their data. We then describe our current progress on two fronts: automatic phoneme transcription, and glossing. Finally, we briefly describe our future directions.

Computation and Language
Automatic Speech Recognition Advancements for Indigenous Languages of the Americas

Monica Romero,Sandra Gomez,Ivan G. Torre

2024-09-22

Abstract:Indigenous languages are a fundamental legacy in the development of human communication, embodying the unique identity and culture of local communities in America. The Second AmericasNLP (Americas Natural Language Processing) Competition Track 1 of NeurIPS (Neural Information Processing Systems) 2022 proposed the task of training automatic speech recognition (ASR) systems for five Indigenous languages: Quechua, Guarani, Bribri, Kotiria, and Wa'ikhana. In this paper, we describe the fine-tuning of a state-of-the-art ASR model for each target language, using approximately 36.65 h of transcribed speech data from diverse sources enriched with data augmentation methods. We systematically investigate, using a Bayesian search, the impact of the different hyperparameters on the Wav2vec2.0 XLS-R (Cross-Lingual Speech Representations) variants of 300 M and 1 B parameters. Our findings indicate that data and detailed hyperparameter tuning significantly affect ASR accuracy, but language complexity determines the final result. The Quechua model achieved the lowest character error rate (CER) (12.14), while the Kotiria model, despite having the most extensive dataset during the fine-tuning phase, showed the highest CER (36.59). Conversely, with the smallest dataset, the Guarani model achieved a CER of 15.59, while Bribri and Wa'ikhana obtained, respectively, CERs of 34.70 and 35.23. Additionally, Sobol' sensitivity analysis highlighted the crucial roles of freeze fine-tuning updates and dropout rates. We release our best models for each language, marking the first open ASR models for Wa'ikhana and Kotiria. This work opens avenues for future research to advance ASR techniques in preserving minority Indigenous languages

Computation and Language

Deploying Technology to Save Endangered Languages

Harnessing the Power of Artificial Intelligence to Vitalize Endangered Indigenous Languages: Technologies and Experiences

Not always about you: Prioritizing community needs when developing endangered language technology

Revitalizing Endangered Languages: AI-powered language learning as a catalyst for language appreciation

Endangered Languages: A Sociocognitive Approach to Language Death, Identity Loss, and Preservation in the Age of Artificial Intelligence

NLP Progress in Indigenous Latin American Languages

"It's how you do things that matters": Attending to Process to Better Serve Indigenous Communities with Language Technologies

Living Dictionaries: A Platform for Indigenous and Under-Resourced Languages

Challenges of language technologies for the indigenous languages of the Americas

Exploring the Role of Artificial Intelligence in Language Documentation and Endangered Language Preservation.

Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions

Shaping the Future of Endangered and Low-Resource Languages -- Our Role in the Age of LLMs: A Keynote at ECIR 2024

A Summary of the First Workshop on Language Technology for Language Documentation and Revitalization

Endangered languages, social subjects and mediatization:

What shall we do with an hour of data? Speech recognition for the un- and under-served languages of Common Voice

Local Translation Services for Neglected Languages

Is Machine Learning Speaking my Language? A Critical Look at the NLP-Pipeline Across 8 Human Languages

Meta AI system is a boost to endangered languages — as long as humans aren't forgotten

OCR Post Correction for Endangered Language Texts

Towards a General-Purpose Linguistic Annotation Backend

Automatic Speech Recognition Advancements for Indigenous Languages of the Americas