Abstract:Language Confusion is a phenomenon where Large Language Models (LLMs) generate text that is neither in the desired language, nor in a contextually appropriate language. This phenomenon presents a critical challenge in text generation by LLMs, often appearing as erratic and unpredictable behavior. We hypothesize that there are linguistic regularities to this inherent vulnerability in LLMs and shed light on patterns of language confusion across LLMs. We introduce a novel metric, Language Confusion Entropy, designed to directly measure and quantify this confusion, based on language distributions informed by linguistic typology and lexical variation. Comprehensive comparisons with the Language Confusion Benchmark (Marchisio et al., 2024) confirm the effectiveness of our metric, revealing patterns of language confusion across LLMs. We further link language confusion to LLM security, and find patterns in the case of multilingual embedding inversion attacks. Our analysis demonstrates that linguistic typology offers theoretically grounded interpretation, and valuable insights into leveraging language similarities as a prior for LLM alignment and security.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the language confusion phenomenon that occurs when multilingual large language models (LLMs) generate text. Specifically, this phenomenon is manifested as the text generated by LLMs being neither the expected language nor a context - appropriate language, which brings unpredictable behavior to text generation. The paper aims to quantify and analyze this phenomenon by introducing a new metric - language confusion entropy - and explore its impact on the security of LLMs. ### Main Research Questions 1. **What are the measurable patterns? How can these patterns be effectively quantified?** - The researchers proposed a new metric named "language confusion entropy" to quantify the language confusion phenomenon in multilingual large language models. This metric is based on language distribution and takes into account the influence of language typology and lexical variation. 2. **How does language similarity affect language confusion? How can this knowledge be applied to enhance the alignment and security of LLMs?** - By constructing a language atlas based on language typology, the researchers found a strong correlation between language similarity and language confusion. Low - resource languages show less confusion, and training across different scripts and language families can more effectively reduce language confusion. ### Background and Motivation - **Challenges of Multilingual Large Language Models**: Although multilingual large language models have made significant progress in natural language processing (NLP), they are prone to language confusion when generating text, especially more obvious in multilingual embedding inversion attacks. - **Limitations of Existing Methods**: Existing metrics such as the index proposed by Marchisio et al., although they can measure the percentage of non - expected languages in the model response, fail to capture the subtle differences in language distribution. ### Methods and Contributions - **Language Confusion Entropy**: A new metric - language confusion entropy - was proposed to quantify the language confusion phenomenon in multilingual large language models. This metric can more effectively capture language confusion by re - weighting the language distribution, especially emphasizing the long - tail distribution. - **Language Atlas**: A language atlas was constructed based on language typology, revealing a strong correlation between language confusion and semantic similarity. - **Modified KL Divergence Algorithm**: A modified KL divergence algorithm was proposed to determine the correlation between language similarity (defined by the language atlas) and language confusion. - **Extensive Analysis**: Through extensive analysis, statistically significant language confusion patterns were revealed, providing new insights for the security research of LLMs. ### Conclusion The paper systematically analyzed the language confusion phenomenon in multilingual large language models by introducing language confusion entropy and constructing a language atlas, and explored its impact on model security. The research results show that language similarity plays an important role in language confusion, low - resource languages show less confusion, and training across different scripts and language families can more effectively reduce language confusion. These findings provide a theoretical basis and practical suggestions for enhancing the alignment and security of LLMs.

Large Language Models are Easily Confused: A Quantitative Metric, Security Implications and Typological Analysis

Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks

Understanding and Mitigating Language Confusion in LLMs

Large Language Models and Security

Recent Advances in Attack and Defense Approaches of Large Language Models

A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly

Large Language Model Supply Chain: Open Problems From the Security Perspective

A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions

Against All Odds: Overcoming Typology, Script, and Language Confusion in Multilingual Embedding Inversion Attacks

Privacy in Large Language Models: Attacks, Defenses and Future Directions

Large Language Models in Cybersecurity: State-of-the-Art

Challenges and Applications of Large Language Models

Unique Security and Privacy Threats of Large Language Model: A Comprehensive Survey

Misinforming LLMs: vulnerabilities, challenges and opportunities

Perplexed: Understanding When Large Language Models are Confused

The Quo Vadis of the Relationship between Language and Large Language Models

Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks

Security and Privacy Challenges of Large Language Models: A Survey

Towards Trustable Language Models: Investigating Information Quality of Large Language Models