Abstract:Large Language Models (LLMs) demonstrate strong machine translation capabilities on languages they are trained on. However, the impact of factors beyond training data size on translation performance remains a topic of debate, especially concerning languages not directly encountered during training. Our study delves into Llama2's translation capabilities. By modeling a linear relationship between linguistic feature distances and machine translation scores, we ask ourselves if there are potentially better central languages for LLMs other than English. Our experiments show that the 7B Llama2 model yields above 10 BLEU when translating into all languages it has seen, which rarely happens for languages it has not seen. Most translation improvements into unseen languages come from scaling up the model size rather than instruction tuning or increasing shot count. Furthermore, our correlation analysis reveals that syntactic similarity is not the only linguistic factor that strongly correlates with machine translation scores. Interestingly, we discovered that under specific circumstances, some languages (e.g. Swedish, Catalan), despite having significantly less training data, exhibit comparable correlation levels to English. These insights challenge the prevailing landscape of LLMs, suggesting that models centered around languages other than English could provide a more efficient foundation for multilingual applications.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper explores whether we can have better multilingual large language models (LLMs) if English is not the core language. Specifically, the researchers aim to answer the following questions by analyzing the machine translation capabilities of the Llama2 model in different languages: 1. **Translation Performance in Unseen Languages**: How do existing large language models perform in translating languages they have not encountered before? 2. **Impact of Model Size**: Does increasing the number of model parameters improve translation performance more than instruction tuning or increasing the number of samples? 3. **Impact of Language Similarity**: Besides grammatical similarity, are there other language features that strongly correlate with machine translation performance? 4. **Choice of Core Language**: Are there other languages that could serve as the core language for multilingual models to enhance their multilingual capabilities? ### Research Methods 1. **Machine Translation Evaluation**: The researchers evaluated the Llama2 model's machine translation capabilities in 41 languages, 15 of which were not present in the training data. Evaluation metrics included BLEU scores and COMET-22 scores. 2. **Correlation Analysis**: By calculating various distances between languages (such as genetic, geographic, phonological, syntactic, etc.), the researchers analyzed the correlation between these distances and machine translation performance. 3. **Model Variant Experiments**: The researchers also tested different scales of the Llama2 model (7B, 13B), chat versions, and the effect of increasing the number of samples. ### Key Findings 1. **Translation Performance**: Llama2 performed well in languages seen in the training data, with BLEU scores generally above 10. However, performance was poor in unseen languages, with many languages having BLEU scores below 10. 2. **Impact of Model Size**: Increasing the number of model parameters had the most significant effect on improving translation performance, while the effects of instruction tuning and increasing the number of samples varied by language. 3. **Impact of Language Similarity**: Syntactic similarity is not the only language feature strongly correlated with translation performance. For example, despite having less training data, Swedish and Catalan showed a correlation between language distance and translation performance comparable to English. 4. **Choice of Core Language**: The results suggest that although English has the highest proportion in the training data, other languages (such as Swedish and Catalan) could also serve as core languages for multilingual models, thereby enhancing the model's multilingual capabilities. ### Conclusion This study provides a comprehensive evaluation of the machine translation capabilities of multilingual large language models and reveals the impact of language similarity and model size on translation performance. The findings challenge the current English-centric design of multilingual models and propose new directions for more efficient and sustainable development of future multilingual systems.

Could We Have Had Better Multilingual LLMs If English Was Not the Central Language?

Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners

The Reasonableness Behind Unreasonable Translation Capability of Large Language Model

Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models

Beyond Data Quantity: Key Factors Driving Performance in Multilingual Language Models

Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?

RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs

Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis

Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?

Unraveling the Potential of Large Language Models in Code Translation: How Far Are We?

Improving Bilingual Capabilities of Language Models to Support Diverse Linguistic Practices in Education

What do Large Language Models Need for Machine Translation Evaluation?

A Novel Paradigm Boosting Translation Capabilities of Large Language Models

Enhancing Document-level Translation of Large Language Model via Translation Mixed-instructions

Bridging the Gap: Enhancing LLM Performance for Low-Resource African Languages with New Benchmarks, Fine-Tuning, and Cultural Adjustments

How Multilingual Are Large Language Models Fine-Tuned for Translation?

A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models

Extrapolating Large Language Models to Non-English by Aligning Languages

Assessing Translation capabilities of Large Language Models involving English and Indian Languages

LLM for Everyone: Representing the Underrepresented in Large Language Models

Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?