Abstract:Adversarial Robustness Distillation (ARD) aims to transfer the robustness of large teacher models to small student models, facilitating the attainment of robust performance on resource-limited devices. However, existing research on ARD primarily focuses on the overall robustness of student models, overlooking the crucial aspect of $\textit{robust fairness}$. Specifically, these models may demonstrate strong robustness on some classes of data while exhibiting high vulnerability on other classes. Unfortunately, the "buckets effect" implies that the robustness of the deployed model depends on the classes with the lowest level of robustness. In this paper, we first investigate the inheritance of robust fairness during ARD and reveal that student models only partially inherit robust fairness from teacher models. We further validate this issue through fine-grained experiments with various model capacities and find that it may arise due to the gap in capacity between teacher and student models, as well as the existing methods treating each class equally during distillation. Based on these observations, we propose $\textbf{Fair}$ $\textbf{A}$dversarial $\textbf{R}$obustness $\textbf{D}$istillation (Fair-ARD), a novel framework for enhancing the robust fairness of student models by increasing the weights of difficult classes, and design a geometric perspective-based method to quantify the difficulty of different classes for determining the weights. Extensive experiments show that Fair-ARD surpasses both state-of-the-art ARD methods and existing robust fairness algorithms in terms of robust fairness (e.g., the worst-class robustness under AutoAttack is improved by at most 12.3\% and 5.3\% using ResNet18 on CIFAR10, respectively), while also slightly improving overall robustness. Our code is available at: [https://github.com/NISP-official/Fair-ARD](https://github.com/NISP-official/Fair-ARD).

Adversarial Contrastive Distillation with Adaptive Denoising

DCCD: Reducing Neural Network Redundancy Via Distillation

Boosting Adversarial Robustness Distillation Via Hybrid Decomposed Knowledge.

Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation

Revisiting Adversarial Robustness Distillation: Robust Soft Labels Make Student Better

Improving Adversarial Robustness Through a Curriculum-Guided Reliable Distillation

Data-Free Adversarial Distillation

Feature Distillation With Guided Adversarial Contrastive Learning

MAE-MACD: the Masked Adversarial Contrastive Distillation Algorithm Grounded in Masked Autoencoders

LDN-RC: a Lightweight Denoising Network with Residual Connection to Improve Adversarial Robustness

Enhanced Accuracy and Robustness via Multi-teacher Adversarial Distillation

Revisiting Adversarial Robustness Distillation from the Perspective of Robust Fairness.

Adversarially Robust Distillation

Rethinking Adversarial Robustness Distillation VIA Strength-Dependent Adaptive Regularization

Mitigating Accuracy-Robustness Trade-off via Balanced Multi-Teacher Adversarial Distillation

Distilling Adversarial Robustness Using Heterogeneous Teachers

Improving Adversarial Robust Fairness via Anti-Bias Soft Label Distillation

Denoising Autoencoder-based Defensive Distillation as an Adversarial Robustness Algorithm

A Patch-wise Adversarial Denoising Could Enhance the Robustness of Adversarial Training

How and When Adversarial Robustness Transfers in Knowledge Distillation?