How journals are fighting back against a wave of questionable images

Nicola Jones

DOI: https://doi.org/10.1038/d41586-024-00372-6

IF: 64.8

2024-02-13

Nature

Abstract:Publishers are deploying AI-based tools to detect suspicious images, but generative AI threatens their efforts. Publishers are deploying AI-based tools to detect suspicious images, but generative AI threatens their efforts.

multidisciplinary sciences

What problem does this paper attempt to address?

AI-generated images threaten science — here's how researchers hope to spot them

Diana Kwon

DOI: https://doi.org/10.1038/d41586-024-03542-8

IF: 64.8

2024-11-06

Nature

Abstract:Generative-AI technologies can create convincing scientific data with ease — publishers and integrity specialists fear a torrent of faked science. Generative-AI technologies can create convincing scientific data with ease — publishers and integrity specialists fear a torrent of faked science.

multidisciplinary sciences
AI Art and its Impact on Artists

Jessica Cheng,Deja Workman,Timnit Gebru,J. Flowers,Abhishek Gupta,Harry H. Jiang,Lauren Brown,A. Hanna,Mehtab Khan

DOI: https://doi.org/10.1145/3600211.3604681

2023-08-08

Abstract:The last 3 years have resulted in machine learning (ML)-based image generators with the ability to output consistently higher quality images based on natural language prompts as inputs. As a result, many popular commercial “generative AI Art” products have entered the market, making generative AI an estimated $48B industry [125]. However, many professional artists have spoken up about the harms they have experienced due to the proliferation of large scale image generators trained on image/text pairs from the Internet. In this paper, we review some of these harms which include reputational damage, economic loss, plagiarism and copyright infringement. To guard against these issues while reaping the potential benefits of image generators, we provide recommendations such as regulation that forces organizations to disclose their training data, and tools that help artists prevent using their content as training data without their consent.

Computer Science,Art
Why technical solutions for detecting AI-generated content in research and education are insufficient

Jahna Otterbacher

DOI: https://doi.org/10.1016/j.patter.2023.100796

2023-07-01

Patterns

Abstract:Artificial intelligence (AI)-generated content detectors are not foolproof and often introduce other problems, as shown by Desaire et al. and Liang et al. in papers published recently in <i>Patterns</i> and <i>Cell Reports Physical Science.</i> Rather than "fighting" AI with more AI, we must develop an academic culture that promotes the use of generative AI in a creative, ethical manner.
Examining the Prevalence and Dynamics of AI-Generated Media in Art Subreddits

Hana Matatov,Marianne Aubin Le Quéré,Ofra Amir,Mor Naaman

2024-10-10

Abstract:Broadly accessible generative AI models like Dall-E have made it possible for anyone to create compelling visual art. In online communities, the introduction of AI-generated content (AIGC) may impact community dynamics by shifting the kinds of content being posted or the responses to content suspected of being generated by AI. We take steps towards examining the potential impact of AIGC on art-related communities on Reddit. We distinguish between communities that disallow AI content and those without a direct policy. We look at image-based posts made to these communities that are transparently created by AI, or comments in these communities that suspect authors of using generative AI. We find that AI posts (and accusations) have played a very small part in these communities through the end of 2023, accounting for fewer than 0.2% of the image-based posts. Even as the absolute number of author-labelled AI posts dwindles over time, accusations of AI use remain more persistent. We show that AI content is more readily used by newcomers and may help increase participation if it aligns with community rules. However, the tone of comments suspecting AI use by others have become more negative over time, especially in communities that do not have explicit rules about AI. Overall, the results show the changing norms and interactions around AIGC in online communities designated for creativity.

Artificial Intelligence,Computers and Society,Social and Information Networks
AI-enabled image fraud in scientific publications

Jinjin Gu,Xinlei Wang,Chenang Li,Junhua Zhao,Weijin Fu,Gaoqi Liang,Jing Qiu

DOI: https://doi.org/10.1016/j.patter.2022.100511

2022-07-08

Abstract:Destroying image integrity in scientific papers may result in serious consequences. Inappropriate duplication and fabrication of images are two common misconducts in this aspect. The rapid development of artificial-intelligence technology has brought to us promising image-generation models that can produce realistic fake images. Here, we show that such advanced generative models threaten the publishing system in academia as they may be used to generate fake scientific images that cannot be effectively identified. We demonstrate the disturbing potential of these generative models in synthesizing fake images, plagiarizing existing images, and deliberately modifying images. It is very difficult to identify images generated by these models by visual inspection, image-forensic tools, and detection tools due to the unique paradigm of the generative models for processing images. This perspective reveals vast risks and arouses the vigilance of the scientific community on fake scientific images generated by artificial intelligence (AI) models.
Navigating the light and shadow of scientific publishing faced with machine learning and generative AI

Federico Palmisani,Daniel Segelcke,Jan Vollert

DOI: https://doi.org/10.1002/ejp.4736

2024-10-05

European Journal of Pain

Abstract:Background The public release of ChatGPT in November 2022 sparked a boom and public interest in generative artificial intelligence (AI) that has led to journals and journal families hastily releasing generative AI policies, ranging from asking authors for acknowledgement or declaration to the outright banning of use. Results Here, we briefly discuss the basics of machine learning, generative AI, and how it will affect scientific publishing. We focus especially on potential risks and benefits to the scientific community as a whole and journals specifically. Conclusion While the concerns of editors, for example about manufactured studies, are valid, some recently implemented or suggested policies will not be sustainable in the long run. The quality of generated text and code is quickly becoming so high that it will not only be impossible to detect the use of generative AI but would also mean taking a powerful tool away from researchers that can make their life easier every day. Significance We discuss the history and current state of AI and highlight its relevance for medical publishing and pain research. We provide guidance on how to act now to increase good scientific practice in the world of ChatGPT and call for a task force focusing on improving publishing pain research with use of generative AI.

neurosciences,clinical neurology,anesthesiology
Generative AI Images and Indian Media Industry: An Overview of Opportunities and Challenges

Suchivrat Arya,Gunjan Sharma

DOI: https://doi.org/10.58966/jcm2023249

2023-12-18

Abstract:Artificial Intelligence (AI) has gained initial momentum in the past few years. Another side of this is Generative AI, which is growing and has the capacity to transform journalism and media content. There are speculations about the consequences of AI from creating warfare to the making of movies. This article considers notable platforms like Shutterstock, providers of stock photographs, music and editing tools and secondly, DALL.E 2-Open AI, a generative AI platform of Chat GPT.The proposed research article aims to find out the effects of generative AI images in the media industry, the opportunities of AI generative visuals and the challenges faced by the industry due to the innovative technology. This article will also try to demonstrate the capacity and the limitations of generative AI content and reflect on the implications of generative AI for media education and journalism.
Editors' Statement on the Responsible Use of Generative AI Technologies in Scholarly Journal Publishing

Gregory E Kaebnick,David Christopher Magnus,Audiey Kao,Mohammad Hosseini,David Resnik,Veljko Dubljević,Christy Rentmeester,Bert Gordijn,Mark J Cherry,Karen J Maschke,John McMillan,Lisa M Rasmussen,Laura Haupt,Udo Schüklenk,Ruth Chadwick,Debora Diniz

DOI: https://doi.org/10.1002/eahr.500182

Abstract:Generative artificial intelligence (AI) has the potential to transform many aspects of scholarly publishing. Authors, peer reviewers, and editors might use AI in a variety of ways, and those uses might augment their existing work or might instead be intended to replace it. We are editors of bioethics and humanities journals who have been contemplating the implications of this ongoing transformation. We believe that generative AI may pose a threat to the goals that animate our work but could also be valuable for achieving those goals. In the interests of fostering a wider conversation about how generative AI may be used, we have developed a preliminary set of recommendations for its use in scholarly publishing. We hope that the recommendations and rationales set out here will help the scholarly community navigate toward a deeper understanding of the strengths, limits, and challenges of AI for responsible scholarly work.
How Spammers and Scammers Leverage AI-Generated Images on Facebook for Audience Growth

Renee DiResta,Josh A. Goldstein

2024-03-19

Abstract:Much of the research and discourse on risks from artificial intelligence (AI) image generators, such as DALL-E and Midjourney, has centered around whether they could be used to inject false information into political discourse. We show that spammers and scammers - seemingly motivated by profit or clout, not ideology - are already using AI-generated images to gain significant traction on Facebook. At times, the Facebook Feed is recommending unlabeled AI-generated images to users who neither follow the Pages posting the images nor realize that the images are AI-generated, highlighting the need for improved transparency and provenance standards as AI models proliferate.

Computers and Society
Nothing Is Harder to Resist Than the Temptation of AI

Andrew Park,Jan Kietzmann,Jayson Killoran,Yuanyuan Cui,Patrick van Esch,Amir Dabirian

DOI: https://doi.org/10.1109/mitp.2023.3340529

2024-02-03

IT Professional

Abstract:The use of generative AI has become increasingly prevalent in the business world. With the ability to create original content and automate certain tasks, businesses have been quick to adopt this technology. However, as with any emerging technology, there are potential pitfalls to be aware of. This article aims to review the current state of generative AI in business and highlight some of the potential risks associated with its use. Specifically, we examine issues such as pausing giant AI experiments, misinformation, data accuracy, process automation, shift of power, control of civilization, organizational security, and the potential for AI-generated content to deceive individuals. By bringing these concerns to the forefront, we hope to encourage a more thoughtful and cautious approach to the use of generative AI in business.

computer science, information systems,telecommunications, software engineering
AI-generated images and video are here: how could they shape research?

Carissa Wong

DOI: https://doi.org/10.1038/d41586-024-00659-8

IF: 64.8

2024-03-08

Nature

Abstract:Scientists are already using image-generating models to jazz up papers and presentations. But some say these tools could harm research. Scientists are already using image-generating models to jazz up papers and presentations. But some say these tools could harm research.

multidisciplinary sciences
Genuine images in 2024

H Holden Thorp,H. Holden Thorp

DOI: https://doi.org/10.1126/science.adn7530

IF: 56.9

2024-01-05

Science

Abstract:In recent years, the research community has become increasingly concerned with issues involving the manipulation of images in scientific papers. Some of these alterations—involving images from experimental techniques such as microscopy, flow cytometry, and western blots—are inadvertent and may not change the conclusions of papers. But in rare cases, some are done deliberately to mislead readers. Image sleuths who can detect these alterations, like the scientific integrity consultant Elisabeth Bik, have risen to prominence, as has the website PubPeer, where many of the detected flaws are posted. High-profile incidents, such as one involving the laboratory of former Stanford University President Marc Tessier-Lavigne, have eroded public confidence in science and harmed careers of investigators who missed doctored images coming from their own laboratories. To address these problems, in 2024, the Science family of journals is adopting the use of Proofig, an artificial intelligence (AI)–powered image-analysis tool, to detect altered images across all six of the journals.

multidisciplinary sciences
AI to publish knowledge: a tectonic shift

Thomas Lemberger

DOI: https://doi.org/10.1038/s44319-024-00119-4

IF: 9.071

2024-03-20

EMBO Reports

Abstract:The rise of generative AI will transform scientific publishing but it also poses risks. While AI enables the dissemination of knowledge in computable form, preserving transparency and human values will become even more important for maintaining trust in the scientific discourse.

cell biology,biochemistry & molecular biology
Image synthesis from an ethical perspective

Oliver Bendel

DOI: https://doi.org/10.1007/s00146-023-01780-4

2023-09-28

AI & Society

Abstract:Generative AI has gained a lot of attention in society, business, and science. This trend has increased since 2018, and the big breakthrough came in 2022. In particular, AI-based text and image generators are now widely used. This raises a variety of ethical issues. The present paper first gives an introduction to generative AI and then to applied ethics in this context. Three specific image generators are presented: DALL-E 2, Stable Diffusion, and Midjourney. The author goes into technical details and basic principles, and compares their similarities and differences. This is followed by an ethical discussion. The paper addresses not only risks, but opportunities for generative AI. A summary with an outlook rounds off the article.
AI Imagery and the Overton Window

Sarah K. Amer

2023-06-03

Abstract:AI-based text-to-image generation has undergone a significant leap in the production of visually comprehensive and aesthetic imagery over the past year, to the point where differentiating between a man-made piece of art and an AI-generated image is becoming more difficult. Generative Models such as Stable Diffusion, Midjourney and others are expected to affect several major industries in technological and ethical aspects. Striking the balance between raising human standard of life and work vs exploiting one group of people to enrich another is a complex and crucial part of the discussion. Due to the rapid growth of this technology, the way in which its models operate, and gray area legalities, visual and artistic domains - including the video game industry, are at risk of being taken over from creators by AI infrastructure owners. This paper is a literature review examining the concerns facing both AI developers and users today, including identity theft, data laundering and more. It discusses legalization challenges and ethical concerns, and concludes with how AI generative models can be tremendously useful in streamlining the process of visual creativity in both static and interactive media given proper regulation. Keywords: AI text-to-image generation, Midjourney, Stable Diffusion, AI Ethics, Game Design, Digital Art, Data Laundering

Computers and Society,Artificial Intelligence,Human-Computer Interaction
Organic or Diffused: Can We Distinguish Human Art from AI-generated Images?

Anna Yoo Jeong Ha,Josephine Passananti,Ronik Bhaskar,Shawn Shan,Reid Southen,Haitao Zheng,Ben Y. Zhao

2024-07-03

Abstract:The advent of generative AI images has completely disrupted the art world. Distinguishing AI generated images from human art is a challenging problem whose impact is growing over time. A failure to address this problem allows bad actors to defraud individuals paying a premium for human art and companies whose stated policies forbid AI imagery. It is also critical for content owners to establish copyright, and for model trainers interested in curating training data in order to avoid potential model collapse. There are several different approaches to distinguishing human art from AI images, including classifiers trained by supervised learning, research tools targeting diffusion models, and identification by professional artists using their knowledge of artistic techniques. In this paper, we seek to understand how well these approaches can perform against today's modern generative models in both benign and adversarial settings. We curate real human art across 7 styles, generate matching images from 5 generative models, and apply 8 detectors (5 automated detectors and 3 different human groups including 180 crowdworkers, 4000+ professional artists, and 13 expert artists experienced at detecting AI). Both Hive and expert artists do very well, but make mistakes in different ways (Hive is weaker against adversarial perturbations while Expert artists produce higher false positives). We believe these weaknesses will remain as models continue to evolve, and use our data to demonstrate why a combined team of human and automated detectors provides the best combination of accuracy and robustness.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning
Using Generative AI for Literature Searches and Scholarly Writing: Is the Integrity of the Scientific Discourse in Jeopardy?

Paul G. Schmidt,Amnon J. Meir

DOI: https://doi.org/10.48550/arXiv.2311.06981

2023-10-08

Abstract:Ever since the public release of ChatGPT in November 2022, serious concerns have been raised about the impact and potentially dire consequences of the increasingly widespread use of generative AI tools for purposes of scientific writing and publishing. We document the ongoing discussion in the science community with a review of news articles, editorials, and position statements by major scientific publishers; discuss the potential pitfalls of using generative AI tools such as ChatGPT as aids in scholarly writing; furnish evidence for the proposition that AI-induced contamination of the scientific literature is not only a threat, but already a reality; and call upon leaders in our field to develop policies and guidelines to stem the spread of such contamination. Closing on a positive note, we provide a brief overview of potentially useful capabilities and sensible applications of ChatGPT and similar AI tools for purposes of scholarly writing.

Computers and Society,General Mathematics,History and Overview
Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI

Robert Hönig,Javier Rando,Nicholas Carlini,Florian Tramèr

2024-06-18

Abstract:Artists are increasingly concerned about advancements in image generation models that can closely replicate their unique artistic styles. In response, several protection tools against style mimicry have been developed that incorporate small adversarial perturbations into artworks published online. In this work, we evaluate the effectiveness of popular protections -- with millions of downloads -- and show they only provide a false sense of security. We find that low-effort and "off-the-shelf" techniques, such as image upscaling, are sufficient to create robust mimicry methods that significantly degrade existing protections. Through a user study, we demonstrate that all existing protections can be easily bypassed, leaving artists vulnerable to style mimicry. We caution that tools based on adversarial perturbations cannot reliably protect artists from the misuse of generative AI, and urge the development of alternative non-technological solutions.

Cryptography and Security
GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Models

Emilio Ferrara

DOI: https://doi.org/10.1007/s42001-024-00250-1

2024-01-23

Abstract:Generative Artificial Intelligence (GenAI) and Large Language Models (LLMs) are marvels of technology; celebrated for their prowess in natural language processing and multimodal content generation, they promise a transformative future. But as with all powerful tools, they come with their shadows. Picture living in a world where deepfakes are indistinguishable from reality, where synthetic identities orchestrate malicious campaigns, and where targeted misinformation or scams are crafted with unparalleled precision. Welcome to the darker side of GenAI applications. This article is not just a journey through the meanders of potential misuse of GenAI and LLMs, but also a call to recognize the urgency of the challenges ahead. As we navigate the seas of misinformation campaigns, malicious content generation, and the eerie creation of sophisticated malware, we'll uncover the societal implications that ripple through the GenAI revolution we are witnessing. From AI-powered botnets on social media platforms to the unnerving potential of AI to generate fabricated identities, or alibis made of synthetic realities, the stakes have never been higher. The lines between the virtual and the real worlds are blurring, and the consequences of potential GenAI's nefarious applications impact us all. This article serves both as a synthesis of rigorous research presented on the risks of GenAI and misuse of LLMs and as a thought-provoking vision of the different types of harmful GenAI applications we might encounter in the near future, and some ways we can prepare for them.

Computers and Society,Artificial Intelligence,Computation and Language,Human-Computer Interaction
AI Art is Theft: Labour, Extraction, and Exploitation, Or, On the Dangers of Stochastic Pollocks

Trystan S. Goetze

DOI: https://doi.org/10.1145/3630106.3658898

2024-05-15

Abstract:Since the launch of applications such as DALL-E, Midjourney, and Stable Diffusion, generative artificial intelligence has been controversial as a tool for creating artwork. While some have presented longtermist worries about these technologies as harbingers of fully automated futures to come, more pressing is the impact of generative AI on creative labour in the present. Already, business leaders have begun replacing human artistic labour with AI-generated images. In response, the artistic community has launched a protest movement, which argues that AI image generation is a kind of theft. This paper analyzes, substantiates, and critiques these arguments, concluding that AI image generators involve an unethical kind of labour theft. If correct, many other AI applications also rely upon theft.

Computers and Society,Artificial Intelligence

How journals are fighting back against a wave of questionable images

AI-generated images threaten science — here's how researchers hope to spot them

AI Art and its Impact on Artists

Why technical solutions for detecting AI-generated content in research and education are insufficient

Examining the Prevalence and Dynamics of AI-Generated Media in Art Subreddits

AI-enabled image fraud in scientific publications

Navigating the light and shadow of scientific publishing faced with machine learning and generative AI

Generative AI Images and Indian Media Industry: An Overview of Opportunities and Challenges

Editors' Statement on the Responsible Use of Generative AI Technologies in Scholarly Journal Publishing

How Spammers and Scammers Leverage AI-Generated Images on Facebook for Audience Growth

Nothing Is Harder to Resist Than the Temptation of AI

AI-generated images and video are here: how could they shape research?

Genuine images in 2024

AI to publish knowledge: a tectonic shift

Image synthesis from an ethical perspective

AI Imagery and the Overton Window

Organic or Diffused: Can We Distinguish Human Art from AI-generated Images?

Using Generative AI for Literature Searches and Scholarly Writing: Is the Integrity of the Scientific Discourse in Jeopardy?

Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI

GenAI Against Humanity: Nefarious Applications of Generative Artificial Intelligence and Large Language Models

AI Art is Theft: Labour, Extraction, and Exploitation, Or, On the Dangers of Stochastic Pollocks