Abstract:Recent studies have highlighted the potential of exploiting parallel corpora to enhance multilingual large language models, improving performance in both bilingual tasks, e.g., machine translation, and general-purpose tasks, e.g., text classification. Building upon these findings, our comprehensive study aims to identify the most effective strategies for leveraging parallel corpora. We investigate the impact of parallel corpora quality and quantity, training objectives, and model size on the performance of multilingual large language models enhanced with parallel corpora across diverse languages and tasks. Our analysis reveals several key insights: (i) filtering noisy translations is essential for effectively exploiting parallel corpora, while language identification and short sentence filtering have little effect; (ii) even a corpus containing just 10K parallel sentences can yield results comparable to those obtained from much larger datasets; (iii) employing only the machine translation objective yields the best results among various training objectives and their combinations; (iv) larger multilingual language models benefit more from parallel corpora than smaller models due to their stronger capacity for cross-task transfer. Our study offers valuable insights into the optimal utilization of parallel corpora to enhance multilingual large language models, extending the generalizability of previous findings from limited languages and tasks to a broader range of scenarios.

Iterative Learning of Parallel Lexicons and Phrases from Non-Parallel Corpora

Agreement-based Learning of Parallel Lexicons and Phrases from Non-Parallel Corpora

Iterative Task-adaptive Pretraining for Unsupervised Word Alignment

A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models

Machine Translation Model based on Non-parallel Corpus and Semi-supervised Transductive Learning

Bilingual lexicon induction from non-parallel corpora

Extracting parallel phrases from comparable corpora

Exploring Bilingual Parallel Corpora for Syntactically Controllable Paraphrase Generation

Phrase-Based Chinese-Vietnamese Pseudo-Parallel Sentence Pair Generation

Improved Chinese-Japanese phrase-based MT quality using an extended quasi-parallel corpus

Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora

Extracting paraphrases from a parallel corpus

Neural Machine Translation Research Based on the Semantic Vector of the Tri-Lingual Parallel Corpus

Extending Multilingual Machine Translation through Imitation Learning

Bilingual Lexicon Induction from Non-Parallel Data with Minimal Supervision.

Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data

Syntactically-Informed Unsupervised Paraphrasing with Non-Parallel Data.

Revealing the Parallel Multilingual Learning within Large Language Models

Bilingual Dictionary-based Language Model Pretraining for Neural Machine Translation

Inflating a Small Parallel Corpus into a Large Quasi-parallel Corpus Using Monolingual Data for Chinese-Japanese Machine Translation.

Learning to Decode Collaboratively with Multiple Language Models