Abstract:Image segmentation is a crucial vision task that groups pixels within an image into semantically meaningful segments, which is pivotal in obtaining a fine-grained understanding of real-world scenes. However, an increasing privacy concern exists regarding training large-scale image segmentation models on unauthorized private data. In this work, we exploit the concept of unlearnable examples to make images unusable to model training by generating and adding unlearnable noise into the original images. Particularly, we propose a novel Unlearnable Segmentation (UnSeg) framework to train a universal unlearnable noise generator that is capable of transforming any downstream images into their unlearnable version. The unlearnable noise generator is finetuned from the Segment Anything Model (SAM) via bilevel optimization on an interactive segmentation dataset towards minimizing the training error of a surrogate model that shares the same architecture with SAM but is trained from scratch. We empirically verify the effectiveness of UnSeg across 6 mainstream image segmentation tasks, 10 widely used datasets, and 7 different network architectures, and show that the unlearnable images can reduce the segmentation performance by a large margin. Our work provides useful insights into how to leverage foundation models in a data-efficient and computationally affordable manner to protect images against image segmentation models.
Computer Vision and Pattern Recognition,Cryptography and Security,Machine Learning
What problem does this paper attempt to address?
The problem that this paper attempts to solve is: how to generate effective unlearnable examples (UEs) to protect images from being used for training large - scale image segmentation models, especially when these images are collected and used without permission. Specifically, the author focuses on generating unlearnable examples in the image segmentation task, which is a fine - grained visual task, different from previous research mainly focused on the image classification task. The paper proposes a new framework named UnSeg, aiming to protect image privacy by generating unlearnable noise to invalidate images for model training.
### Main Problems
1. **Data Efficiency Challenge**: An effective UE generation method should be able to learn to generate effective UEs based on a small number of images, rather than relying on existing large - scale image segmentation datasets.
2. **Generation Efficiency Challenge**: When applied to protect private images, the method should be able to directly generate UEs without optimizing each image.
3. **Transferability Challenge**: The UE generation method should remain effective when transferred to different downstream tasks and datasets.
### Solutions
To address the above challenges, the paper proposes the UnSeg framework, which fine - tunes the pre - trained Segment Anything Model (SAM) through bilevel min - min optimization, converting it into a general unlearnable noise generator. The specific steps are as follows:
1. **Threat Model**: Assume that the data owner wishes to protect the images they post on social media platforms, preventing these images from being unauthorizedly collected and used for training large - scale image segmentation models.
2. **UnSeg Framework**:
- **Unlearnable Noise Generator**: Based on the fine - tuning of the pre - trained SAM, generate error - minimizing noise in the target area through visual cues (such as points, boxes, and masks).
- **Proxy Model**: A re - initialized SAM used to simulate potential data exploitation situations.
3. **Training Process**:
- Alternately optimize the noise generator and the proxy model, and reduce the training error of the proxy model through bilevel min - min optimization.
- Introduce the ε generalization technique, use a small noise budget ε during training and a large ε during inference to stabilize the training process.
### Experimental Results
The paper verifies the effectiveness, efficiency, and transferability of UnSeg through extensive experiments. The experimental results show that the unlearnable noise generated by UnSeg can significantly reduce the performance of the image segmentation model. For example, in the COCO instance segmentation task, the performance drops by 92%.
### Main Contributions
1. **Propose a new UE generation framework UnSeg**, specifically for the image segmentation task, and fine - tune a general UE generator from the pre - trained SAM.
2. **Introduce bilevel min - min optimization based on interactive image segmentation**, and adopt the ε generalization technique in the training process to improve the stability and effectiveness of training.
3. **Verify the effectiveness of UnSeg through extensive experiments**, and it performs well on multiple image segmentation tasks, datasets, and network architectures.
### Conclusion
UnSeg provides an effective method to protect images from being used for training image segmentation models by generating unlearnable noise, solving key challenges such as data efficiency, generation efficiency, and transferability. This provides new ideas and technical means for protecting image privacy.