Abstract:Data pruning is the problem of identifying a core subset that is most beneficial to training and discarding the remainder. While pruning strategies are well studied for discriminative models like those used in classification, little research has gone into their application to generative models. Generative models aim to estimate the underlying distribution of the data, so presumably they should benefit from larger datasets. In this work we aim to shed light on the accuracy of this statement, specifically answer the question of whether data pruning for generative diffusion models could have a positive impact. Contrary to intuition, we show that eliminating redundant or noisy data in large datasets is beneficial particularly when done strategically. We experiment with several pruning methods including recent-state-of-art methods, and evaluate over CelebA-HQ and ImageNet datasets. We demonstrate that a simple clustering method outperforms other sophisticated and computationally demanding methods. We further exhibit how we can leverage clustering to balance skewed datasets in an unsupervised manner to allow fair sampling for underrepresented populations in the data distribution, which is a crucial problem in generative models.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the effectiveness of applying data pruning techniques in generative diffusion models (DMs). Specifically, the paper explores whether model performance can be improved by identifying and removing redundant or harmful data samples when training generative diffusion models. The goal of generative diffusion models is to estimate the underlying distribution of data, so theoretically they should benefit from larger datasets. However, the paper proposes that strategically eliminating redundant or noisy data in large datasets can have a positive impact on the model. This is counter - intuitive because it is usually thought that more data is always beneficial for the learning of generative models. The paper evaluates the impact of different pruning methods (such as random pruning, loss monotonicity, GraNd, EL2N, MoSo and clustering methods) on generative diffusion models through experiments and validates them on the CelebA - HQ and ImageNet datasets. The research results show that the simple clustering method is even superior to other complex and computationally expensive methods in some cases. In addition, the clustering method can also be used to balance the skewed distribution in the dataset, thereby ensuring that under - represented groups in the data distribution can also be fairly sampled, which is especially important for generative models. In summary, this paper aims to answer the following questions: - Is data pruning beneficial for generative diffusion models? - How to distinguish redundant or harmful data samples? - What is the pruning threshold at which the model will be damaged? - How to balance the skewed data distribution? Through these studies, the paper provides a new perspective on the data efficiency of generative models, especially how to optimize model training through data pruning in the case of limited resources.

Data Pruning in Generative Diffusion Models

Pruning then Reweighting: Towards Data-Efficient Training of Diffusion Models

Structural Pruning for Diffusion Models

Dataset Pruning: Reducing Training Data by Examining Generalization Influence

DRoP: Distributionally Robust Pruning

Distilled Pruning: Using Synthetic Data to Win the Lottery

Effortless Efficiency: Low-Cost Pruning of Diffusion Models

GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning

Multi-domain clustering pruning: Exploring space and frequency similarity based on GAN

Gradient-based Intra-attention Pruning on Pre-trained Language Models

Distilling the Knowledge in Data Pruning

LD-Pruner: Efficient Pruning of Latent Diffusion Models using Task-Agnostic Insights

Not All Data Matters: An End-to-End Adaptive Dataset Pruning Framework for Enhancing Model Performance and Efficiency

Statistical Mechanical Analysis of Neural Network Pruning

DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization

FGPGAN: a finer-grained CNN pruning via generative adversarial network

Pruning of generative adversarial neural networks for medical imaging diagnostics with evolution strategy

A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations

A Survey on Generative Diffusion Model

Beyond Efficiency: Molecular Data Pruning for Enhanced Generalization

Rethinking the Value of Network Pruning