Beyond Aesthetics: Cultural Competence in Text-to-Image Models

Nithish Kannen,Arif Ahmad,Marco Andreetto,Vinodkumar Prabhakaran,Utsav Prabhu,Adji Bousso Dieng,Pushpak Bhattacharyya,Shachi Dave

2024-08-04

Abstract:Text-to-Image (T2I) models are being increasingly adopted in diverse global communities where they create visual representations of their unique cultures. Current T2I benchmarks primarily focus on faithfulness, aesthetics, and realism of generated images, overlooking the critical dimension of cultural competence. In this work, we introduce a framework to evaluate cultural competence of T2I models along two crucial dimensions: cultural awareness and cultural diversity, and present a scalable approach using a combination of structured knowledge bases and large language models to build a large dataset of cultural artifacts to enable this evaluation. In particular, we apply this approach to build CUBE (CUltural BEnchmark for Text-to-Image models), a first-of-its-kind benchmark to evaluate cultural competence of T2I models. CUBE covers cultural artifacts associated with 8 countries across different geo-cultural regions and along 3 concepts: cuisine, landmarks, and art. CUBE consists of 1) CUBE-1K, a set of high-quality prompts that enable the evaluation of cultural awareness, and 2) CUBE-CSpace, a larger dataset of cultural artifacts that serves as grounding to evaluate cultural diversity. We also introduce cultural diversity as a novel T2I evaluation component, leveraging quality-weighted Vendi score. Our evaluations reveal significant gaps in the cultural awareness of existing models across countries and provide valuable insights into the cultural diversity of T2I outputs for under-specified prompts. Our methodology is extendable to other cultural regions and concepts, and can facilitate the development of T2I models that better cater to the global population.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the lack of cultural diversity and cultural awareness in Text-to-Image (T2I) models when generating images. Specifically:

1. **Cultural Awareness**: Current T2I models often fail to accurately recognize or generate concepts or items related to specific cultures. For example, Figures 1(c) and (d) show inaccurately generated images of an Indian temple and Japanese grilled eel.

2. **Cultural Diversity**: T2I models tend to adopt a simplified, homogenized view of cultures, generating only a narrow set of concepts or items from a particular culture. For instance, Figure 1(b) shows a lack of diversity in the generated images of Nigerian dishes.

To address these issues, the authors introduce a new evaluation framework—CUBE (CUltural BEnchmark), to assess the cultural capabilities of T2I models. CUBE covers two key dimensions:
- **Cultural Awareness**: Whether the model can reliably generate correct images related to specific cultures.
- **Cultural Diversity**: Whether the model can generate images that reflect cultural diversity.

Additionally, the paper constructs a large-scale dataset containing 300,000 cultural items and proposes a new evaluation component—the Cultural Diversity Score, to better assess the cultural diversity performance of T2I models. Through this series of work, the paper reveals the cultural awareness gaps of existing models across different countries and regions and provides valuable insights.