Abstract:Recent interest has surged in employing Large Language Models (LLMs) for machine translation (MT) via in-context learning (ICL) (Vilar et al., 2023). Most prior studies primarily focus on optimizing translation quality, with limited attention to understanding the specific aspects of ICL that influence the said quality. To this end, we perform the first of its kind, an exhaustive study of in-context learning for machine translation. We first establish that ICL is primarily example-driven and not instruction-driven. Following this, we conduct an extensive exploration of various aspects of the examples to understand their influence on downstream performance. Our analysis includes factors such as quality and quantity of demonstrations, spatial proximity, and source versus target originality. Further, we also investigate challenging scenarios involving indirectness and misalignment of examples to understand the limits of ICL. While we establish the significance of the quality of the target distribution over the source distribution of demonstrations, we further observe that perturbations sometimes act as regularizers, resulting in performance improvements. Surprisingly, ICL does not necessitate examples from the same task, and a related task with the same target distribution proves sufficient. We hope that our study acts as a guiding resource for considerations in utilizing ICL for MT. Our code is available on <a class="link-external link-https" href="https://github.com/PranjalChitale/in-context-mt-analysis" rel="external noopener nofollow">this https URL</a>.

Going Beyond Word Matching: Syntax Improves In-context Example Selection for Machine Translation

SCOI: Syntax-augmented Coverage-based In-context Example Selection for Machine Translation

Ungrammatical-syntax-based In-context Example Selection for Grammatical Error Correction

In-Context Example Selection via Similarity Search Improves Low-Resource Machine Translation

In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax

Syntax-Aware Complex-Valued Neural Machine Translation

Syntax-Enhanced Neural Machine Translation with Syntax-Aware Word Representations

CTQScorer: Combining Multiple Features for In-context Example Selection for Machine Translation

Syntax-Based Context Representation for Statistical Machine Translation.

Leveraging Code to Improve In-context Learning for Semantic Parsing

In-context Examples Selection for Machine Translation

Syntax-aware Transformer Encoder for Neural Machine Translation

Effective In-Context Example Selection through Data Compression

Revisiting Demonstration Selection Strategies in In-Context Learning

Instruction Tuning Vs. In-Context Learning: Revisiting Large Language Models in Few-Shot Computational Social Science

An Empirical Study of In-context Learning in LLMs for Machine Translation

Syntax-aware Neural Machine Translation Directed by Syntactic Dependency Degree

Compositional Exemplars for In-context Learning

A Survey on In-context Learning

Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning

In-context Learning as Maintaining Coherency: A Study of On-the-fly Machine Translation Using Large Language Models