Debiasing Classifiers by Amplifying Bias with Latent Diffusion and Large Language Models

Donggeun Ko,Dongjun Lee,Namjun Park,Wonkyeong Shim,Jaekwang Kim

2024-11-25

Abstract:Neural networks struggle with image classification when biases are learned and misleads correlations, affecting their generalization and performance. Previous methods require attribute labels (e.g. background, color) or utilizes Generative Adversarial Networks (GANs) to mitigate biases. We introduce DiffuBias, a novel pipeline for text-to-image generation that enhances classifier robustness by generating bias-conflict samples, without requiring training during the generation phase. Utilizing pretrained diffusion and image captioning models, DiffuBias generates images that challenge the biases of classifiers, using the top-$K$ losses from a biased classifier ($f_B$) to create more representative data samples. This method not only debiases effectively but also boosts classifier generalization capabilities. To the best of our knowledge, DiffuBias is the first approach leveraging a stable diffusion model to generate bias-conflict samples in debiasing tasks. Our comprehensive experimental evaluations demonstrate that DiffuBias achieves state-of-the-art performance on benchmark datasets. We also conduct a comparative analysis of various generative models in terms of carbon emissions and energy consumption to highlight the significance of computational efficiency.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in image classification tasks, when neural networks learn the biases in the data set, these biases will lead to a decline in the generalization ability and performance of the model. Specifically, the paper focuses on how to reduce the over - dependence of the model on specific attributes or patterns in the data set (such as object shape, color, texture, etc.) by generating bias - conflict samples, thereby improving the robustness and generalization ability of the model.

The paper proposes a new method named DiffuBias. This method uses pre - trained diffusion models and image captioning models to generate bias - conflict samples without additional training during the generation stage. This method can not only effectively reduce biases but also enhance the generalization ability of the classifier. By using pre - trained models, DiffuBias can generate high - quality bias - conflict samples without increasing additional learning costs, thereby achieving effective de - biasing of the classifier.

In summary, the main contributions of this paper are as follows:
1. For the first time, use pre - trained latent diffusion models and image captioning models to generate synthetic bias - conflict samples to reduce the bias of the classifier.
2. This method does not require training the generation model, thus eliminating the learning cost.
3. Although there is no additional training cost, this model can still effectively amplify the bias samples, thereby processing the biased data set and reducing the bias of the classifier.