Abstract:The development of monolingual language models for low and mid-resource languages continues to be hindered by the difficulty in sourcing high-quality training data. In this study, we present a novel cross-lingual vocabulary transfer strategy, trans-tokenization, designed to tackle this challenge and enable more efficient language adaptation. Our approach focuses on adapting a high-resource monolingual LLM to an unseen target language by initializing the token embeddings of the target language using a weighted average of semantically similar token embeddings from the source language. For this, we leverage a translation resource covering both the source and target languages. We validate our method with the Tweeties, a series of trans-tokenized LLMs, and demonstrate their competitive performance on various downstream tasks across a small but diverse set of languages. Additionally, we introduce Hydra LLMs, models with multiple swappable language modeling heads and embedding tables, which further extend the capabilities of our trans-tokenization strategy. By designing a Hydra LLM based on the multilingual model TowerInstruct, we developed a state-of-the-art machine translation model for Tatar, in a zero-shot manner, completely bypassing the need for high-quality parallel data. This breakthrough is particularly significant for low-resource languages like Tatar, where high-quality parallel data is hard to come by. By lowering the data and time requirements for training high-quality models, our trans-tokenization strategy allows for the development of LLMs for a wider range of languages, especially those with limited resources. We hope that our work will inspire further research and collaboration in the field of cross-lingual vocabulary transfer and contribute to the empowerment of languages on a global scale.

How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?

An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference

An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models

From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages

Exploring Design Choices for Building Language-Specific LLMs

Xlm-v: Overcoming the vocabulary bottleneck in multilingual masked language models

Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

The Ups and Downs of Large Language Model Inference with Vocabulary Trimming by Language Heuristics

How Vocabulary Sharing Facilitates Multilingualism in LLaMA?

From Tokens to Words: On the Inner Lexicon of LLMs

Large Vocabulary Size Improves Large Language Models

Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages

Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models

Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP

Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?

Improving Pre-Trained Multilingual Model with Vocabulary Expansion

Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners

LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders

Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments