Decomposed Prompting: Unveiling Multilingual Linguistic Structure Knowledge in English-Centric Large Language Models

Ercong Nie,Shuzhou Yuan,Bolei Ma,Helmut Schmid,Michael Färber,Frauke Kreuter,Hinrich Schütze

2024-02-28

Abstract:Despite the predominance of English in their training data, English-centric Large Language Models (LLMs) like GPT-3 and LLaMA display a remarkable ability to perform multilingual tasks, raising questions about the depth and nature of their cross-lingual capabilities. This paper introduces the decomposed prompting approach to probe the linguistic structure understanding of these LLMs in sequence labeling tasks. Diverging from the single text-to-text prompt, our method generates for each token of the input sentence an individual prompt which asks for its linguistic label. We assess our method on the Universal Dependencies part-of-speech tagging dataset for 38 languages, utilizing both English-centric and multilingual LLMs. Our findings show that decomposed prompting surpasses the iterative prompting baseline in efficacy and efficiency under zero- and few-shot settings. Further analysis reveals the influence of evaluation methods and the use of instructions in prompts. Our multilingual investigation shows that English-centric language models perform better on average than multilingual models. Our study offers insights into the multilingual transferability of English-centric LLMs, contributing to the understanding of their multilingual linguistic knowledge.

Computation and Language

What problem does this paper attempt to address?

The problem this paper attempts to address is the exploration of the performance of English-centric large language models (LLMs) in multilingual tasks and the nature of their cross-linguistic capabilities. Specifically, despite being primarily trained on English data, these models still perform well in multilingual tasks. The core question of the research is whether these models possess deep, generalizable multilingual understanding or merely rely on surface-level lexical pattern alignment across different languages. To answer this question, the researchers propose a new method called "decomposed prompting" to investigate the multilingual structural understanding capabilities of these models in sequence labeling tasks. They also evaluate the effectiveness and efficiency of this method through experiments, particularly comparing its performance with existing methods in zero-shot and few-shot settings. Additionally, the paper discusses the performance differences between English-centric LLMs and multilingual LLMs, especially in terms of knowledge transfer when dealing with non-English languages.