Abstract:The large language model (LLM) has garnered significant attention due to its in-context learning mechanisms and emergent capabilities. The research community has conducted several pilot studies to apply LLMs to machine translation tasks and evaluate their performance from diverse perspectives. However, previous research has primarily focused on the LLM itself and has not explored human intervention in the inference process of LLM. The characteristics of LLM, such as in-context learning and prompt engineering, closely mirror human cognitive abilities in language tasks, offering an intuitive solution for human-in-the-loop generation. In this study, we propose a human-in-the-loop pipeline that guides LLMs to produce customized outputs with revision instructions. The pipeline initiates by prompting the LLM to produce a draft translation, followed by the utilization of automatic retrieval or human feedback as supervision signals to enhance the LLM's translation through in-context learning. The human-machine interactions generated in this pipeline are also stored in an external database to expand the in-context retrieval database, enabling us to leverage human supervision in an offline setting. We evaluate the proposed pipeline using GPT-3.5-turbo API on five domain-specific benchmarks for German-English translation. The results demonstrate the effectiveness of the pipeline in tailoring in-domain translations and improving translation performance compared to direct translation. Additionally, we discuss the results from the following perspectives: 1) the effectiveness of different in-context retrieval methods; 2) the construction of a retrieval database under low-resource scenarios; 3) the observed domains differences; 4) the quantitative analysis of linguistic statistics; and 5) the qualitative analysis of translation cases. The code and data are available at <a class="link-external link-https" href="https://github.com/NLP2CT/HIL-MT/" rel="external noopener nofollow">this https URL</a>.

Human Language Modeling

Large Human Language Models: A Need and the Challenges

Unified Language Model Pre-training for Natural Language Understanding and Generation

HLB: Benchmarking LLMs' Humanlikeness in Language Use

Large Language Models as Zero-Shot Human Models for Human-Robot Interaction

Human Simulacra: Benchmarking the Personification of Large Language Models

Evaluating Human-Language Model Interaction

Are Human Conversations Special? A Large Language Model Perspective

Social-LLM: Modeling User Behavior at Scale using Language Models and Social Network Data

HumanVLM: Foundation for Human-Scene Vision-Language Model

Modeling Human Subjectivity in LLMs Using Explicit and Implicit Human Factors in Personas

Extending Machine Language Models toward Human-Level Language Understanding

Large Language Models Humanize Technology

Efficient Models for the Detection of Hate, Abuse and Profanity

Cognitive Modeling of Semantic Fluency Using Transformers

Human-in-the-loop Machine Translation with Large Language Model

Anatomy of Neural Language Models

Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?

Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication

Multilevel Large Language Models for Everyone