AI produces gibberish when trained on too much AI-generated data

Emily Wenger

DOI: https://doi.org/10.1038/d41586-024-02355-z

IF: 64.8

2024-07-25

Nature

Abstract:Generative AI models are now widely accessible, enabling everyone to create their own machine-made something. But these models can collapse if their training data sets contain too much AI-generated content. Generative-AI models collapse when trained recursively.

multidisciplinary sciences

What problem does this paper attempt to address?

AI models fed AI-generated data quickly spew nonsense

Elizabeth Gibney

DOI: https://doi.org/10.1038/d41586-024-02420-7

IF: 64.8

2024-07-25

Nature

Abstract:Researchers gave successive versions of a large language model information produced by previous generations of the AI — and observed rapid collapse. Researchers gave successive versions of a large language model information produced by previous generations of the AI — and observed rapid collapse.

multidisciplinary sciences
Nepotistically Trained Generative-AI Models Collapse

Matyas Bohacek,Hany Farid

2023-11-21

Abstract:Trained on massive amounts of human-generated content, AI (artificial intelligence) image synthesis is capable of reproducing semantically coherent images that match the visual appearance of its training data. We show that when retrained on even small amounts of their own creation, these generative-AI models produce highly distorted images. We also show that this distortion extends beyond the text prompts used in retraining, and that once poisoned, the models struggle to fully heal even after retraining on only real images.

Artificial Intelligence,Computer Vision and Pattern Recognition
AI models collapse when trained on recursively generated data

Ilia Shumailov,Zakhar Shumaylov,Yiren Zhao,Nicolas Papernot,Ross Anderson,Yarin Gal

DOI: https://doi.org/10.1038/s41586-024-07566-y

IF: 64.8

Nature

Abstract:Stable diffusion revolutionized image creation from descriptive text. GPT-2 (ref. 1), GPT-3(.5) (ref. 2) and GPT-4 (ref. 3) demonstrated high performance across a variety of language tasks. ChatGPT introduced such language models to the public. It is now clear that generative artificial intelligence (AI) such as large language models (LLMs) is here to stay and will substantially change the ecosystem of online text and images. Here we consider what may happen to GPT-{n} once LLMs contribute much of the text found online. We find that indiscriminate use of model-generated content in training causes irreversible defects in the resulting models, in which tails of the original content distribution disappear. We refer to this effect as 'model collapse' and show that it can occur in LLMs as well as in variational autoencoders (VAEs) and Gaussian mixture models (GMMs). We build theoretical intuition behind the phenomenon and portray its ubiquity among all learned generative models. We demonstrate that it must be taken seriously if we are to sustain the benefits of training from large-scale data scraped from the web. Indeed, the value of data collected about genuine human interactions with systems will be increasingly valuable in the presence of LLM-generated content in data crawled from the Internet.
Generative AI: An Existential Threat to Human Content Creators?

Jiasun Li

DOI: https://doi.org/10.2139/ssrn.4666860

2023-01-01

SSRN Electronic Journal

Abstract:With the phenomenal rise of generative AI models (e.g., Large Language Models such as GPT or text-to-image models such as Stable Diffusion), there are increasing concerns about human content creators' future. Specifically, as generative models further advance, would they eventually take over the jobs of all human content creators? We argue that the answer is no, even if generative AI models can perform at their theoretical limit one day. Our theory has a close analogy to the familiar insight in financial economics on the impossibility of a perfectly informationally efficient market as in Grossman and Stiglitz (1980): If generative AI models can provide all the content that human needs at low variable costs, then there is no incentive for human content creators to spend resources on content creation as they cannot profit from it. But if no human creates new content, then generative AI models can only rehash what they already know and be unable to provide up-to-date answers. This creates a paradox.
The Generative AI Paradox: "What It Can Create, It May Not Understand"

Peter West,Ximing Lu,Nouha Dziri,Faeze Brahman,Linjie Li,Jena D. Hwang,Liwei Jiang,Jillian Fisher,Abhilasha Ravichander,Khyathi Chandu,Benjamin Newman,Pang Wei Koh,Allyson Ettinger,Yejin Choi

2023-11-01

Abstract:The recent wave of generative AI has sparked unprecedented global attention, with both excitement and concern over potentially superhuman levels of artificial intelligence: models now take only seconds to produce outputs that would challenge or exceed the capabilities even of expert humans. At the same time, models still show basic errors in understanding that would not be expected even in non-expert humans. This presents us with an apparent paradox: how do we reconcile seemingly superhuman capabilities with the persistence of errors that few humans would make? In this work, we posit that this tension reflects a divergence in the configuration of intelligence in today's generative models relative to intelligence in humans. Specifically, we propose and test the Generative AI Paradox hypothesis: generative models, having been trained directly to reproduce expert-like outputs, acquire generative capabilities that are not contingent upon -- and can therefore exceed -- their ability to understand those same types of outputs. This contrasts with humans, for whom basic understanding almost always precedes the ability to generate expert-level outputs. We test this hypothesis through controlled experiments analyzing generation vs. understanding in generative models, across both language and image modalities. Our results show that although models can outperform humans in generation, they consistently fall short of human capabilities in measures of understanding, as well as weaker correlation between generation and understanding performance, and more brittleness to adversarial inputs. Our findings support the hypothesis that models' generative capability may not be contingent upon understanding capability, and call for caution in interpreting artificial intelligence by analogy to human intelligence.

Artificial Intelligence,Computation and Language,Computer Vision and Pattern Recognition,Machine Learning
Is Generative AI an Existential Threat to Human Creatives? Insights from Financial Economics

Jiasun Li

2024-07-29

Abstract:With the phenomenal rise of generative AI models (e.g., large language models such as GPT or large image models such as Diffusion), there are increasing concerns about human creatives' futures. Specifically, as generative models' power further increases, will they eventually replace all human creatives' jobs? We argue that the answer is "no," even if existing generative AI models' capabilities reach their theoretical limit. Our theory has a close analogy to a familiar insight in financial economics on the impossibility of an informationally efficient market [Grossman and Stiglitz (1980)]: If generative AI models can provide all the content humans need at low variable costs, then there is no incentive for humans to spend costly resources on content creation as they cannot profit from it. But if no human creates new content, then generative AI can only learn from stale information and be unable to generate up-to-date content that reflects new happenings in the physical world. This creates a paradox.

Human-Computer Interaction,Artificial Intelligence
Self-Consuming Generative Models Go MAD

Sina Alemohammad,Josue Casco-Rodriguez,Lorenzo Luzi,Ahmed Imtiaz Humayun,Hossein Babaei,Daniel LeJeune,Ali Siahkoohi,Richard G. Baraniuk

2023-07-05

Abstract:Seismic advances in generative AI algorithms for imagery, text, and other data types has led to the temptation to use synthetic data to train next-generation models. Repeating this process creates an autophagous (self-consuming) loop whose properties are poorly understood. We conduct a thorough analytical and empirical analysis using state-of-the-art generative image models of three families of autophagous loops that differ in how fixed or fresh real training data is available through the generations of training and in whether the samples from previous generation models have been biased to trade off data quality versus diversity. Our primary conclusion across all scenarios is that without enough fresh real data in each generation of an autophagous loop, future generative models are doomed to have their quality (precision) or diversity (recall) progressively decrease. We term this condition Model Autophagy Disorder (MAD), making analogy to mad cow disease.

Machine Learning,Artificial Intelligence,Computer Vision and Pattern Recognition
Generative Ai: potential and pitfalls

Iryna Yasenova,Oksana Berdnyk

DOI: https://doi.org/10.32347/tit.2023.61.0301

2023-12-31

Transfer of innovative technologies

Abstract:The explosive popularity of ChatGPT around the world gave us the first real tipping point in public acceptance of AI. Finally, everyone, everywhere can see the breakthrough potential of this technology for themselves. Large language models (LLM) and the fundamental models underlying these advances in generative artificial intelligence (GenAI) represent a significant turning point. Not only have they cracked the code of language complexity, allowing machines to learn context, infer intent, and be independent creative individuals, but they can be quickly configured to perform a wide variety of different tasks. This technology should fundamentally change everything — from science, business, health care to, in fact, society itself. The positive impact on human creativity and productivity will be enormous. Companies will use these models to rethink the way work is done. Every role in every enterprise has the potential to be reimagined, as AI people working as co-pilots become the norm, greatly expanding their capabilities. Generative AI will affect tasks, not professions. Some of these tasks will be automated, some will be transformed by artificial intelligence, and some will remain unchanged. It can also be expected that humans will face a large number of new challenges, such as ensuring the accurate and responsible use of GenAI systems. That's why organizations that invest in training people to work with generative AI will have a significant advantage.
Degenerative AI?

Steven M. Bellovin

DOI: https://doi.org/10.1109/msec.2024.3385549

IF: 3.105

2024-05-22

IEEE Security & Privacy

Abstract:It is not secret that generative AI, especially in the form of large language models (LLMs), is extremely popular today. One might go so far as to say that it's eaten the world. It may be a bubble, or it may last—though the death of cryptocurrencies has long been predicted, as I write this Bitcoin has just reached an all-time high value against the American dollar—but for now and at least the next few years, generative AI will be with us. As people who care about security and privacy, we need to understand the implications of it: is it good or bad for our field, and if the latter, what should we do about it? Ignoring it is not an option.

computer science, information systems, software engineering
Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World

Joshua Kazdan,Rylan Schaeffer,Apratim Dey,Matthias Gerstgrasser,Rafael Rafailov,David L. Donoho,Sanmi Koyejo

2024-10-22

Abstract:The increasing presence of AI-generated content on the internet raises a critical question: What happens when generative machine learning models are pretrained on web-scale datasets containing data created by earlier models? Some authors prophesy $\textit{model collapse}$ under a "$\textit{replace}$" scenario: a sequence of models, the first trained with real data and each later one trained only on synthetic data from its preceding model. In this scenario, models successively degrade. Others see collapse as easily avoidable; in an "$\textit{accumulate}$' scenario, a sequence of models is trained, but each training uses all real and synthetic data generated so far. In this work, we deepen and extend the study of these contrasting scenarios. First, collapse versus avoidance of collapse is studied by comparing the replace and accumulate scenarios on each of three prominent generative modeling settings; we find the same contrast emerges in all three settings. Second, we study a compromise scenario; the available data remains the same as in the accumulate scenario -- but unlike $\textit{accumulate}$ and like $\textit{replace}$, each model is trained using a fixed compute budget; we demonstrate that model test loss on real data is larger than in the $\textit{accumulate}$ scenario, but apparently plateaus, unlike the divergence seen with $\textit{replace}$. Third, we study the relative importance of cardinality and proportion of real data for avoiding model collapse. Surprisingly, we find a non-trivial interaction between real and synthetic data, where the value of synthetic data for reducing test loss depends on the absolute quantity of real data. Our insights are particularly important when forecasting whether future frontier generative models will collapse or thrive, and our results open avenues for empirically and mathematically studying the context-dependent value of synthetic data.

Machine Learning,Artificial Intelligence
Generative AI enhances individual creativity but reduces the collective diversity of novel content

Anil R. Doshi,Oliver P. Hauser

DOI: https://doi.org/10.1126/sciadv.adn5290

IF: 13.6

2024-07-13

Science Advances

Abstract:Creativity is core to being human. Generative artificial intelligence (AI)—including powerful large language models (LLMs)—holds promise for humans to be more creative by offering new ideas, or less creative by anchoring on generative AI ideas. We study the causal impact of generative AI ideas on the production of short stories in an online experiment where some writers obtained story ideas from an LLM. We find that access to generative AI ideas causes stories to be evaluated as more creative, better written, and more enjoyable, especially among less creative writers. However, generative AI–enabled stories are more similar to each other than stories by humans alone. These results point to an increase in individual creativity at the risk of losing collective novelty. This dynamic resembles a social dilemma: With generative AI, writers are individually better off, but collectively a narrower scope of novel content is produced. Our results have implications for researchers, policy-makers, and practitioners interested in bolstering creativity.

multidisciplinary sciences
A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'

Ali Borji

2024-10-25

Abstract:The study conducted by Shumailov et al. (2024) demonstrates that repeatedly training a generative model on synthetic data leads to model collapse. This finding has generated considerable interest and debate, particularly given that current models have nearly exhausted the available data. In this work, we investigate the effects of fitting a distribution (through Kernel Density Estimation, or KDE) or a model to the data, followed by repeated sampling from it. Our objective is to develop a theoretical understanding of the phenomenon observed by Shumailov et al. (2024). Our results indicate that the outcomes reported are a statistical phenomenon and may be unavoidable.

Machine Learning,Artificial Intelligence
AI-generated images threaten science — here's how researchers hope to spot them

Diana Kwon

DOI: https://doi.org/10.1038/d41586-024-03542-8

IF: 64.8

2024-11-06

Nature

Abstract:Generative-AI technologies can create convincing scientific data with ease — publishers and integrity specialists fear a torrent of faked science. Generative-AI technologies can create convincing scientific data with ease — publishers and integrity specialists fear a torrent of faked science.

multidisciplinary sciences
AI Art and its Impact on Artists

Jessica Cheng,Deja Workman,Timnit Gebru,J. Flowers,Abhishek Gupta,Harry H. Jiang,Lauren Brown,A. Hanna,Mehtab Khan

DOI: https://doi.org/10.1145/3600211.3604681

2023-08-08

Abstract:The last 3 years have resulted in machine learning (ML)-based image generators with the ability to output consistently higher quality images based on natural language prompts as inputs. As a result, many popular commercial “generative AI Art” products have entered the market, making generative AI an estimated $48B industry [125]. However, many professional artists have spoken up about the harms they have experienced due to the proliferation of large scale image generators trained on image/text pairs from the Internet. In this paper, we review some of these harms which include reputational damage, economic loss, plagiarism and copyright infringement. To guard against these issues while reaping the potential benefits of image generators, we provide recommendations such as regulation that forces organizations to disclose their training data, and tools that help artists prevent using their content as training data without their consent.

Computer Science,Art
Watching the Generative AI Hype Bubble Deflate

David Gray Widder,Mar Hicks

2024-08-16

Abstract:Only a few short months ago, Generative AI was sold to us as inevitable by the leadership of AI companies, those who partnered with them, and venture capitalists. As certain elements of the media promoted and amplified these claims, public discourse online buzzed with what each new beta release could be made to do with a few simple prompts. As AI became a viral sensation, every business tried to become an AI business. Some businesses added "AI" to their names to juice their stock prices, and companies talking about "AI" on their earnings calls saw similar increases. While the Generative AI hype bubble is now slowly deflating, its harmful effects will last.

Computers and Society
Generative AI Degrades Online Communities

Gordon Burtch,Dokyun Lee,Zhichen Chen

DOI: https://doi.org/10.1145/3624732

IF: 22.7

2024-02-22

Communications of the ACM

Abstract:How large language models are influencing online communities.

computer science, theory & methods, software engineering, hardware & architecture
When AI Eats Itself: On the Caveats of AI Autophagy

Xiaodan Xing,Fadong Shi,Jiahao Huang,Yinzhe Wu,Yang Nan,Sheng Zhang,Yingying Fang,Mike Roberts,Carola-Bibiane Schönlieb,Javier Del Ser,Guang Yang

2024-11-08

Abstract:Generative Artificial Intelligence (AI) technologies and large models are producing realistic outputs across various domains, such as images, text, speech, and music. Creating these advanced generative models requires significant resources, particularly large and high-quality datasets. To minimise training expenses, many algorithm developers use data created by the models themselves as a cost-effective training solution. However, not all synthetic data effectively improve model performance, necessitating a strategic balance in the use of real versus synthetic data to optimise outcomes. Currently, the previously well-controlled integration of real and synthetic data is becoming uncontrollable. The widespread and unregulated dissemination of synthetic data online leads to the contamination of datasets traditionally compiled through web scraping, now mixed with unlabeled synthetic data. This trend, known as the AI autophagy phenomenon, suggests a future where generative AI systems may increasingly consume their own outputs without discernment, raising concerns about model performance, reliability, and ethical implications. What will happen if generative AI continuously consumes itself without discernment? What measures can we take to mitigate the potential adverse effects? To address these research questions, this study examines the existing literature, delving into the consequences of AI autophagy, analyzing the associated risks, and exploring strategies to mitigate its impact. Our aim is to provide a comprehensive perspective on this phenomenon advocating for a balanced approach that promotes the sustainable development of generative AI technologies in the era of large models.

Machine Learning,Artificial Intelligence
The Real Dangers of Generative AI

Danielle Allen,E. Glen Weyl

DOI: https://doi.org/10.1353/jod.2024.a915355

2024-01-01

Journal of Democracy

Abstract:Abstract: As perhaps the most consequential technology of our time, Generative Foundation Models (GFMs) present unprecedented challenges for democratic institutions. By allowing deception and de-contextualized information sharing at a previously unimaginable scale and pace, GFMs could undermine the foundations of democracy. At the same time, the investment scale required to develop the models and the race dynamics around that development threaten to enable concentrations of democratically unaccountable power (both public and private). This essay examines the twin threats of collapse and singularity occasioned by the rise of GFMs.

political science
The current state of artificial intelligence generative language models is more creative than humans on divergent thinking tasks

Kent F. Hubert,Kim N. Awa,Darya L. Zabelina

DOI: https://doi.org/10.1038/s41598-024-53303-w

IF: 4.6

2024-02-12

Scientific Reports

Abstract:The emergence of publicly accessible artificial intelligence (AI) large language models such as ChatGPT has given rise to global conversations on the implications of AI capabilities. Emergent research on AI has challenged the assumption that creative potential is a uniquely human trait thus, there seems to be a disconnect between human perception versus what AI is objectively capable of creating. Here, we aimed to assess the creative potential of humans in comparison to AI. In the present study, human participants (N = 151) and GPT-4 provided responses for the Alternative Uses Task, Consequences Task, and Divergent Associations Task. We found that AI was robustly more creative along each divergent thinking measurement in comparison to the human counterparts. Specifically, when controlling for fluency of responses, AI was more original and elaborate. The present findings suggest that the current state of AI language models demonstrate higher creative potential than human respondents.

multidisciplinary sciences

AI produces gibberish when trained on too much AI-generated data

AI models fed AI-generated data quickly spew nonsense

Nepotistically Trained Generative-AI Models Collapse

AI models collapse when trained on recursively generated data

Generative AI: An Existential Threat to Human Content Creators?

The Generative AI Paradox: "What It Can Create, It May Not Understand"

Is Generative AI an Existential Threat to Human Creatives? Insights from Financial Economics

Self-Consuming Generative Models Go MAD

Generative Ai: potential and pitfalls

Degenerative AI?

Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World

Generative AI enhances individual creativity but reduces the collective diversity of novel content

A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'

AI-generated images threaten science — here's how researchers hope to spot them

AI Art and its Impact on Artists

Watching the Generative AI Hype Bubble Deflate

Generative AI Degrades Online Communities

When AI Eats Itself: On the Caveats of AI Autophagy

The Real Dangers of Generative AI

The current state of artificial intelligence generative language models is more creative than humans on divergent thinking tasks