Abstract:The remarkable advances in deep learning have led to the emergence of many off-the-shelf classifiers, e.g., large pre-trained models. However, since they are typically trained on clean data, they remain vulnerable to adversarial attacks. Despite this vulnerability, their superior performance and transferability make off-the-shelf classifiers still valuable in practice, demanding further work to provide adversarial robustness for them in a post-hoc manner. A recently proposed method, denoised smoothing, leverages a denoiser model in front of the classifier to obtain provable robustness without additional training. However, the denoiser often creates hallucination, i.e., images that have lost the semantics of their originally assigned class, leading to a drop in robustness. Furthermore, its noise-and-denoise procedure introduces a significant distribution shift from the original distribution, causing the denoised smoothing framework to achieve sub-optimal robustness. In this paper, we introduce Fine-Tuning with Confidence-Aware Denoised Image Selection (FT-CADIS), a novel fine-tuning scheme to enhance the certified robustness of off-the-shelf classifiers. FT-CADIS is inspired by the observation that the confidence of off-the-shelf classifiers can effectively identify hallucinated images during denoised smoothing. Based on this, we develop a confidence-aware training objective to handle such hallucinated images and improve the stability of fine-tuning from denoised images. In this way, the classifier can be fine-tuned using only images that are beneficial for adversarial robustness. We also find that such a fine-tuning can be done by updating a small fraction of parameters of the classifier. Extensive experiments demonstrate that FT-CADIS has established the state-of-the-art certified robustness among denoised smoothing methods across all $\ell_2$-adversary radius in various benchmarks.

Shape-biased CNNs Are Not Always Superior in Out-of-Distribution Robustness

Informative Dropout for Robust Representation Learning: A Shape-bias Perspective

Clarifying Myths About the Relationship Between Shape Bias, Accuracy, and Robustness

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Shape-Biased Learning by Thinking Inside the Box

A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking

Increasing the Robustness of Semantic Segmentation Models with Painting-by-Numbers

Confidence-aware Denoised Fine-tuning of Off-the-shelf Models for Certified Robustness

MaxStyle: Adversarial Style Composition for Robust Medical Image Segmentation

A Bayesian Approach to OOD Robustness in Image Classification

Teaching CNNs to Mimic Human Visual Cognitive Process & Regularise Texture-Shape Bias

Shape Robustness in Style Enhanced Cross Domain Semantic Segmentation

Robustifying Deep Vision Models Through Shape Sensitization

Rethinking and Improving Robustness of Convolutional Neural Networks: a Shapley Value-based Approach in Frequency Domain

Toward Metrics for Differentiating Out-of-Distribution Sets

Effective Robustness against Natural Distribution Shifts for Models with Different Training Data

OOD-CV-v2: An extended Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Images

ESTIMATE OF THE MAXIMUM SUSTAINABLE YIELD OF SERGESTID SHRIMP IN THE WATERS OFF SOUTHWESTERN TAIWAN

Robust Image Classification in the Presence of Out-of-Distribution and Adversarial Samples Using Attractors in Neural Networks

How explainable are adversarially-robust CNNs?

Category-Sensitive Incremental Learning for Image-Based 3D Shape Reconstruction