Abstract:Highly performant large-scale pre-trained models promise to also provide a valuable foundation for learning specialized tasks, by fine-tuning the model to the desired task. By starting from a good general-purpose model, the goal is to achieve both specialization in the target task and maintain robustness. To assess the robustness of models to out-of-distribution samples after fine-tuning on downstream datasets, we introduce a new robust fine-tuning benchmark, ImageNet-RIB (Robustness Inheritance Benchmark). The benchmark consists of a set of related but distinct specialized (downstream) tasks; pre-trained models are fine-tuned on one task in the set and their robustness is assessed on the rest, iterating across all tasks for fine-tuning and assessment. We find that the continual learning methods, EWC and LwF maintain robustness after fine-tuning though fine-tuning generally does reduce performance on generalization to related downstream tasks across models. Not surprisingly, models pre-trained on large and rich datasets exhibit higher initial robustness across datasets and suffer more pronounced degradation during fine-tuning. The distance between the pre-training and downstream datasets, measured by optimal transport, predicts this performance degradation on the pre-training dataset. However, counterintuitively, model robustness after fine-tuning on related downstream tasks is the worst when the pre-training dataset is the richest and the most diverse. This suggests that starting with the strongest foundation model is not necessarily the best approach for performance on specialist tasks. The benchmark thus offers key insights for developing more resilient fine-tuning strategies and building robust machine learning models. <a class="link-external link-https" href="https://jd730.github.io/projects/ImageNet-RIB" rel="external noopener nofollow">this https URL</a>

ImageNet Pre-training Also Transfers Non-robustness

Do Adversarially Robust ImageNet Models Transfer Better?

Does Robustness on ImageNet Transfer to Downstream Tasks?

Towards Inadequately Pre-trained Models in Transfer Learning

ImageNet-RIB Benchmark: Large Pre-Training Datasets Don't Guarantee Robustness after Fine-Tuning

On Transfer of Adversarial Robustness from Pretraining to Downstream Tasks

Initialization Matters for Adversarial Transfer Learning

Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning

Adversarially-Trained Deep Nets Transfer Better: Illustration on Image Classification

Transferable Learned Image Compression-Resistant Adversarial Perturbations

An adversarial defense algorithm based on robust U-net

TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization

Improving Adversarial Transferability with Gradient Refining

Using Pre-Training Can Improve Model Robustness and Uncertainty

Competition on Robust Deep Learning

Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models

Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks

Exploring Transferable and Robust Adversarial Perturbation Generation Across Network Hierarchy

Robustness and Transferability of Adversarial Attacks on Different Image Classification Neural Networks

Inadequately Pre-trained Models are Better Feature Extractors

A Comprehensive Study on Robustness of Image Classification Models: Benchmarking and Rethinking