Dialogue with the Machine and Dialogue with the Art World: Evaluating Generative AI for Culturally-Situated Creativity

Rida Qadri,Piotr Mirowski,Aroussiak Gabriellan,Farbod Mehr,Huma Gupta,Pamela Karimi,Remi Denton
2024-12-19
Abstract:This paper proposes dialogue as a method for evaluating generative AI tools for culturally-situated creative practice, that recognizes the socially situated nature of art. Drawing on sociologist Howard Becker's concept of Art Worlds, this method expands the scope of traditional AI and creativity evaluations beyond benchmarks, user studies with crowd-workers, or focus groups conducted with artists. Our method involves two mutually informed dialogues: 1) 'dialogues with art worlds' placing artists in conversation with experts such as art historians, curators, and archivists, and 2)'dialogues with the machine,' facilitated through structured artist- and critic-led experimentation with state-of-the-art generative AI tools. We demonstrate the value of this method through a case study with artists and experts steeped in non-western art worlds, specifically the Persian Gulf. We trace how these dialogues help create culturally rich and situated forms of evaluation for representational possibilities of generative AI that mimic the reception of generative artwork in the broader art ecosystem. Putting artists in conversation with commentators also allow artists to shift their use of the tools to respond to their cultural and creative context. Our study can provide generative AI researchers an understanding of the complex dynamics of technology, human creativity and the socio-politics of art worlds, to build more inclusive machines for diverse art worlds.
Computers and Society,Artificial Intelligence
What problem does this paper attempt to address?
The core problem that this paper attempts to solve is: How to evaluate generative AI tools in a specific cultural context to ensure that they can better reflect and integrate into the local cultural and social context. Specifically, the paper proposes a new qualitative method - the dialogue method, which evaluates the performance of generative AI tools in culturally - located creative practices through two ways: "dialogue with the art world" and "dialogue with the machine". ### Main problems of the paper 1. **Limitations of existing evaluation methods**: - Existing generative AI evaluation methods mainly rely on quantitative indicators (such as image quality, diversity, etc.), and these methods ignore the influence of social and cultural contexts on artistic creation. - Traditional user studies and focus groups mostly focus on the individual use experiences of artists, while ignoring the broader art ecosystem. 2. **Insufficient cultural representation**: - Existing generative AI tools have the problem of insufficient representation when dealing with non - Western cultures, and are unable to fully capture and express the uniqueness and complexity of these cultures. - It is necessary to explore how to make generative AI tools better serve artists in different cultural contexts and promote more culturally - sensitive creation. 3. **Social and technological interaction**: - How to expand the evaluation of generative AI tools to include multi - perspective expert groups such as critics, curators, historians, etc., in order to form a more comprehensive understanding. - Explore the application of generative AI tools in the actual creation process and how it affects the creation methods and thinking patterns of artists. ### Solutions proposed in the paper - **Dialogue method**: Through the two ways of "dialogue with the art world" and "dialogue with the machine", combined with the actual creation experiences of artists and the opinions of experts in multiple fields, a more comprehensive and in - depth evaluation method is formed. - **Dialogue with the art world**: Invite artists to communicate with experts such as art historians, curators, and archivists to explore the possibilities of generative AI in cultural expression and its social impacts. - **Dialogue with the machine**: Through structured experiments, let artists directly use generative AI tools for creation, and record their creation processes and feedback. ### Case study The paper shows the application of this method through a specific case study, that is, exploring the possibilities of generative AI as a cultural expression tool among artist groups in the Persian Gulf region. Through this case, the researchers hope to provide a more inclusive and culturally - sensitive direction for the research and development of generative AI. ### Conclusion Through the dialogue method, researchers can not only better understand the performance of generative AI in a specific cultural context, but also provide valuable improvement suggestions for future tool design, thereby promoting the application and development of generative AI in different cultural contexts.