Public Aversion Against ChatGPT in Creative Fields?
Yueying Chu,Peng Liu
DOI: https://doi.org/10.1016/j.xinn.2023.100449
2023-01-01
Abstract:Dear Editor, Machines, powered by artificial intelligence (AI) and other innovations, are expanding the scope of tasks that they can do.1Rahwan I. Cebrian M. Obradovich N. et al.Machine behaviour.Nature. 2019; 568: 477-486Crossref PubMed Scopus (422) Google Scholar,2Innovation Editorial Team, ChinaInnovation focus in 2022.Innovation. 2023; 4100371Google Scholar They have outperformed human professionals in objective tasks such as weather forecasting and image-based disease diagnosis. They are even reaching or surpassing humans in subjective and creative tasks. In late November 2022, OpenAI released an AI chatbot named ChatGPT-3.5 (hereafter, ChatGPT) and an image generator named DALL-E2, which took the internet by storm with their excellent capabilities in understanding human intentions and performing various human tasks.2Innovation Editorial Team, ChinaInnovation focus in 2022.Innovation. 2023; 4100371Google Scholar They represent monumental leaps in large language models and image generation. Everyone can co-work with them via specific prompts to generate creative content (e.g., poetry, painting, and fiction) that can reach the proficiency level of human masters. Even before their release, a non-artist teaming with another AI beat dozens of amateur artists and won a digital arts competition.3Harwell D. He used AI to win a fine-arts competition. Was it cheating?.https://shorturl.at/knpHQDate: 2022Google Scholar Although benefiting human productivity and efficiency, creative machines would yield profound and subversive influences on humanity.4Harari Y.N. Yuval Noah Harari argues that AI has hacked the operating system of human civilisation.https://shorturl.at/vBJQUDate: 2023Google Scholar They challenge our old belief that art—a last frontier of human prerogative—is unique to humans5Ornes S. Computers take art in new directions, challenging the meaning of "creativity".Proc. Natl. Acad. Sci. USA. 2019; 116: 4760-4763Crossref PubMed Scopus (10) Google Scholar and raise substantial concerns about their threat to human distinctiveness and identity. Humans are standing at a crossroad. Someone calls for defending humankind as creative machines rise. Interestingly, people would be negatively biased toward them accomplishing specific human tasks. In algorithmic decision-making, researchers reported that people have harsher responses or more discomfort with algorithms and their decisions, called “algorithm aversion.”6Dietvorst B.J. Simmons J.P. Massey C. Algorithm aversion: people erroneously avoid algorithms after seeing them err.J. Exp. Psychol. Gen. 2015; 144: 114-126Crossref PubMed Scopus (698) Google Scholar Furthermore, it shows that algorithm aversion might be domain specific: it may not exist in objective fields with low identity relevance and clear evaluative criteria (e.g., weather forecasting) but emerges in subjective and creative fields with high identity relevance and ambiguous criteria.7Morewedge C.K. Preference for human, not algorithm aversion.Trends Cognit. Sci. 2022; 26: 824-826Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar For instance, machine authorship of artworks lowers people’s evaluations of their quality and artistic value.8Shank D.B., Stefanik C., Stuhlsatz C., et al. AI composer bias: listeners like music less when they think it was composed by an AI. J. Exp. Psychol. Appl. in press.Google Scholar Therefore, people’s negative bias would play a psychological defender for humankind in creative fields. Unlike other AI machines, ChatGPT is an “all-around player”2Innovation Editorial Team, ChinaInnovation focus in 2022.Innovation. 2023; 4100371Google Scholar and impresses the world by generating high-quality content. But how laypeople respond to it and its creative content remains unknown. Therefore, we are interested in two questions: (1) do laypeople still have an aversion against ChatGPT, and (2) does it meet or even outperform human creativity in their eyes? According to the “algorithm aversion” phenomenon in creative fields,7Morewedge C.K. Preference for human, not algorithm aversion.Trends Cognit. Sci. 2022; 26: 824-826Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar we submitted two hypotheses for the former question. First, artworks declared from ChatGPT receive fewer positive evaluations than those being anonymous (H1) and those declared from human artists (H2). We conducted a pre-registered online experiment (https://osf.io/352pq) involving two surveys (modern Chinese poetry and classical Chinese poetry). The Ethics Review Board of the Center for Psychological Sciences at Zhejiang University, China, approved the experiment. Its materials, data, code, results, and the prompts for content generation are available at https://osf.io/3ntbw/?view_only=e0c904b91f9f452bba8041e576950818. In each poetry condition, we first selected a poet-written poem that most people would not know (to reduce the influence of familiarity). Then, we asked ChatGPT to translate it into English and generate a new one with similar genres. Finally, after several simple prompts by the authors who are not good at poetry writing, we determined the final ChatGPT-generated poetry (see https://osf.io/3ntbw/?view_only=e0c904b91f9f452bba8041e576950818). In each survey, we adopted a three (declared identity: human vs. ChatGPT vs. anonymous) by two (real identity: human vs. ChatGPT) between-subjects design and assigned participants randomly to one of the six conditions. We recruited Chinese participants (final n = 1,034, with 524 for modern poetry, and 510 for classical poetry; 589 females; mean age = 31.5 years) through the sample service of an online survey platform. We showed them a poem—by ChatGPT or a poet—and informed them that it was from a poet, ChatGPT, or did not disclose its authorship. Participants gave their responses to the poem through 13 items for their aesthetic emotion (e.g., “The poem fascinated me,” Cronbach’s α = 0.90), comprehensibility (e.g., “The poem is obscure,” α = 0.79), perceived quality (e.g., “The poem has a clear theme,” α = 0.75), and behavioral intention (e.g., “I am willing to recommend the poem to others,” α = 0.90) on a seven-point scale (1 = totally disagree to 7 = totally agree). Participants then answered an attention-check question about the poem’s declared author and indicated whether they read this poem before. We excluded participants failing the attention check or those who had read the poem previously. In addition, participants assessed their ability to appreciate the poetry, identified the real author, whether they had heard of or used ChatGPT, and their general attitude toward AI (these responses did not moderate our major findings, and we ignored them to keep within the space limit). Finally, participants answered demographic questions, and we found they did not differ in gender, age, education, and prior information about ChatGPT across the six conditions in each survey (ps > 0.05). Confirmatory factor analysis showed that the four-factor model exhibited better fitness (χ2(59) = 173.55, p < 0.001) than the two-factor or three-factor models. We performed a full factorial analysis of variance (ANOVA) for each factor. The main effect of declared identity was significant on aesthetic emotion (F(2, 518) = 3.12, p = 0.045, ηp2 = 0.012) and perceived quality (F(2, 518) = 3.29, p = 0.038, ηp2 = 0.012) in the modern poetry survey, and on aesthetic emotion (F(2, 504) = 6.59, p = 0.001, ηp2 = 0.025) and behavioral intention (F(2, 504) = 3.50, p = 0.031, ηp2 = 0.014) in the classical poetry survey. Their interaction effects with real identity were non-significant (ps > 0.05). Then we conducted pre-planned contrasts between (1) ChatGPT versus human and (2) ChatGPT versus anonymous. Figure 1 shows the significant differences revealed by pre-planned contrasts. Across the two artworks, we only found six significant differences out of 32 pre-planned contrasts. For instance, the human-created modern poem attributed to ChatGPT had lower ratings on aesthetic emotion (ΔM = −0.53, t(518) = −2.53, p = 0.012, Cohen’s d = −0.39) and perceived quality (ΔM = −0.42, t(518) = −2.52, p = 0.012, d = −0.39) than the same one attributed to a poet (see Figure 1A). Among the six differences, only one was between ChatGPT versus anonymous: the ChatGPT-generated classical poem had a lower aesthetic emotion rating when attributed to ChatGPT than when we did not disclose its authorship (ΔM = −0.46, t(504) = −2.47, p = 0.014, d = −0.36). Therefore, our data overall did not support the “algorithm aversion” phenomenon in creative fields.7Morewedge C.K. Preference for human, not algorithm aversion.Trends Cognit. Sci. 2022; 26: 824-826Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar,8Shank D.B., Stefanik C., Stuhlsatz C., et al. AI composer bias: listeners like music less when they think it was composed by an AI. J. Exp. Psychol. Appl. in press.Google Scholar Although Chinese people have an “aversion” against ChatGPT in certain contrasts, they seemingly are insensitive to the creator’s identity. Therefore, public aversion against ChatGPT may not be general, at least in the surveyed country. A potential reason is that China usually more positively frames AI-powered machines versus developed countries; consequently, Chinese people may have more positive attitudes toward them. Hence, they might not have a negative bias against using machines in art and other creative fields. Furthermore, among the six pieces of “algorithm aversion” evidence, we found only one between ChatGPT and anonymous, seemingly supporting Morewedge’s7Morewedge C.K. Preference for human, not algorithm aversion.Trends Cognit. Sci. 2022; 26: 824-826Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar argument that when “algorithm aversion” exists, it might reflect people’s appreciation of human authorship rather than their true aversion against machine authorship. In the modern poetry survey, real identity (ChatGPT-generated versus human-created) had significant main effects on aesthetic emotion (F(1, 518) = 14.64, p < 0.001, ηp2 = 0.027), comprehensibility (F(1, 518) = 42.36, p < 0.001, ηp2 = 0.075), and perceived quality (F(1, 518) = 35.38, p < 0.001, ηp2 = 0.063). The ChatGPT-generated poem received more positive evaluations in these factors, further supported by pre-planned contrasts (see Figure 1A). In the classical poetry survey, participants had non-different responses to the ChatGPT-generated and human-created poetry (ps > 0.05). The human-created classical poem had a higher aesthetic emotion rating in a single contrast (see Figure 1B). Therefore, our data tentatively indicated that ChatGPT meets human creativity in Chinese poetry writing. Previous research9Hitsuwari J. Ueda Y. Yun W. et al.Does human–AI collaboration lead to more creative art? Aesthetic evaluation of human-made and AI-generated haiku poetry.Comput. Hum. Behav. 2023; 139107502Crossref Scopus (4) Google Scholar also supports that laypeople cannot reliably distinguish machine-generated artworks from human-created ones. To conclude, Chinese participants’ reactions show that ChatGPT is crossing art, the pinnacle of human creativity. More notably, our participants did not become more negative toward the involved artworks attributed to ChatGPT, contrary to specific research7Morewedge C.K. Preference for human, not algorithm aversion.Trends Cognit. Sci. 2022; 26: 824-826Abstract Full Text Full Text PDF PubMed Scopus (1) Google Scholar,8Shank D.B., Stefanik C., Stuhlsatz C., et al. AI composer bias: listeners like music less when they think it was composed by an AI. J. Exp. Psychol. Appl. in press.Google Scholar but consistent with others.10Köbis N. Mossink L.D. Artificial intelligence versus Maya Angelou: experimental evidence that people cannot differentiate AI-generated from human-written poetry.Comput. Hum. Behav. 2021; 114106553Crossref Scopus (54) Google Scholar This finding may reflect increasing human comfort with artistic and creative machines. In addition, it implies that people’s negative bias against machine identity may not be a robust psychological defender for humankind in creative fields in the coming era of ChatGPT and artificial general intelligence. We suggest more research to overcome our study’s limitations (e.g., participants from a single country) and acquire a better understanding of how people experience and evaluate machine-generated creative content; for instance, future research could collect evaluations from other nations and cultures and from professionals and investigate more kinds of creative content. Furthermore, we must explore the long-term consequences of creative machines, including whether they will devalue human creativity and lead humans to be passive consumers of creative content. The authors declare no competing interests.