Abstract:The large language model (LLM) has garnered significant attention due to its in-context learning mechanisms and emergent capabilities. The research community has conducted several pilot studies to apply LLMs to machine translation tasks and evaluate their performance from diverse perspectives. However, previous research has primarily focused on the LLM itself and has not explored human intervention in the inference process of LLM. The characteristics of LLM, such as in-context learning and prompt engineering, closely mirror human cognitive abilities in language tasks, offering an intuitive solution for human-in-the-loop generation. In this study, we propose a human-in-the-loop pipeline that guides LLMs to produce customized outputs with revision instructions. The pipeline initiates by prompting the LLM to produce a draft translation, followed by the utilization of automatic retrieval or human feedback as supervision signals to enhance the LLM's translation through in-context learning. The human-machine interactions generated in this pipeline are also stored in an external database to expand the in-context retrieval database, enabling us to leverage human supervision in an offline setting. We evaluate the proposed pipeline using GPT-3.5-turbo API on five domain-specific benchmarks for German-English translation. The results demonstrate the effectiveness of the pipeline in tailoring in-domain translations and improving translation performance compared to direct translation. Additionally, we discuss the results from the following perspectives: 1) the effectiveness of different in-context retrieval methods; 2) the construction of a retrieval database under low-resource scenarios; 3) the observed domains differences; 4) the quantitative analysis of linguistic statistics; and 5) the qualitative analysis of translation cases. The code and data are available at <a class="link-external link-https" href="https://github.com/NLP2CT/HIL-MT/" rel="external noopener nofollow">this https URL</a>.

Can we teach language models to gloss endangered languages?

Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions

GlossLM: A Massively Multilingual Corpus and Pretrained Model for Interlinear Glossed Text

SIGMORPHON 2023 Shared Task of Interlinear Glossing: Baseline Model

Wav2Gloss: Generating Interlinear Glossed Text from Speech

GrammaMT: Improving Machine Translation with Grammar-Informed In-Context Learning

Embedded Translations for Low-resource Automated Glossing

Can LLMs Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challenges

LLMs in the Loop: Leveraging Large Language Model Annotations for Active Learning in Low-Resource Languages

Robust Generalization Strategies for Morpheme Glossing in an Endangered Language Documentation Context

Interpretable Language Modeling via Induction-head Ngram Models

Can LLMs facilitate interpretation of pre-trained language models?

LLMs Are In-Context Reinforcement Learners

In-Context Language Learning: Architectures and Algorithms

Boosting the Capabilities of Compact Models in Low-Data Contexts with Large Language Models and Retrieval-Augmented Generation

Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Tasks

Human-in-the-loop Machine Translation with Large Language Model

Supervised Knowledge Makes Large Language Models Better In-context Learners

Augmenting interpretable models with large language models during training

MAGNIFICo: Evaluating the In-Context Learning Ability of Large Language Models to Generalize to Novel Interpretations

An Empirical Study of In-context Learning in LLMs for Machine Translation