Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge

Li Zhou,Taelin Karidi,Wanlong Liu,Nicolas Garneau,Yong Cao,Wenyu Chen,Haizhou Li,Daniel Hershcovich

2024-10-18

Abstract:Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet often lack a robust methodology to dissect these phenomena comprehensively. Our work aims to bridge this gap by delving into the Food domain, a universally relevant yet culturally diverse aspect of human life. We introduce FmLAMA, a multilingual dataset centered on food-related cultural facts and variations in food practices. We analyze LLMs across various architectures and configurations, evaluating their performance in both monolingual and multilingual settings. By leveraging templates in six different languages, we investigate how LLMs interact with language-specific and cultural knowledge. Our findings reveal that (1) LLMs demonstrate a pronounced bias towards food knowledge prevalent in the United States; (2) Incorporating relevant cultural context significantly improves LLMs' ability to access cultural knowledge; (3) The efficacy of LLMs in capturing cultural nuances is highly dependent on the interplay between the probing language, the specific model architecture, and the cultural context in question. This research underscores the complexity of integrating cultural understanding into LLMs and emphasizes the importance of culturally diverse datasets to mitigate biases and enhance model performance across different cultural domains.

Computation and Language

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper aims to explore whether large language models (LLMs) exhibit bias in cultural knowledge within the food domain and proposes a systematic approach to evaluate these models' performance across different cultural and linguistic contexts. Specifically, the researchers focus on the following questions:

1. **Cultural Bias in LLMs**: Do LLMs show a preference for food knowledge from specific regions? For example, are they more inclined to understand food knowledge from the United States?
2. **Effective Prompting and Evaluation of LLMs**: How can we design prompt templates and evaluation metrics to better extract food knowledge specific to certain cultures from LLMs?
3. **Impact of Cultural Context and Language Specificity**: How do cultural background information and language specificity in prompts affect the performance of LLMs?

To answer these questions, the researchers constructed a multilingual, multicultural food knowledge dataset (FMLAMA) and designed a series of experiments to evaluate the performance of different LLMs in both monolingual and multilingual settings. Through these experiments, they hope to uncover the complexities and challenges LLMs face when handling cross-cultural food knowledge and propose improvements to reduce bias and enhance model performance.