Abstract:Deep learning models have proven to be successful in a wide range of machine learning tasks. Yet, they are often highly sensitive to perturbations on the input data which can lead to incorrect decisions with high confidence, hampering their deployment for practical use-cases. Thus, finding architectures that are (more) robust against perturbations has received much attention in recent years. Just like the search for well-performing architectures in terms of clean accuracy, this usually involves a tedious trial-and-error process with one additional challenge: the evaluation of a network's robustness is significantly more expensive than its evaluation for clean accuracy. Thus, the aim of this paper is to facilitate better streamlined research on architectural design choices with respect to their impact on robustness as well as, for example, the evaluation of surrogate measures for robustness. We therefore borrow one of the most commonly considered search spaces for neural architecture search for image classification, NAS-Bench-201, which contains a manageable size of 6466 non-isomorphic network designs. We evaluate all these networks on a range of common adversarial attacks and corruption types and introduce a database on neural architecture design and robustness evaluations. We further present three exemplary use cases of this dataset, in which we (i) benchmark robustness measurements based on Jacobian and Hessian matrices for their robustness predictability, (ii) perform neural architecture search on robust accuracies, and (iii) provide an initial analysis of how architectural design choices affect robustness. We find that carefully crafting the topology of a network can have substantial impact on its robustness, where networks with the same parameter count range in mean adversarial robust accuracy from 20%-41%. Code and data is available at <a class="link-external link-http" href="http://robustness.vision/" rel="external noopener nofollow">this http URL</a>.

Correlation Analysis between the Robustness of Sparse Neural Networks and their Random Hidden Structural Priors

Robustness of Deep Learning Models on Graphs: A Survey

An Empirical Study on Correlation between Coverage and Robustness for Deep Neural Networks

There is Limited Correlation Between Coverage and Robustness for Deep Neural Networks

The Search for Sparse, Robust Neural Networks

Correlation of Centralities: A Study Through Distinct Graph Robustness

Structural Analysis of Sparse Neural Networks

Sparse DNNs with Improved Adversarial Robustness.

How many perturbations break this model? Evaluating robustness beyond adversarial accuracy

An Adversarial Robustness Perspective on the Topology of Neural Networks

Deep Insights into Graph Adversarial Learning: An Empirical Study Perspective.

AdverSparse: An Adversarial Attack Framework for Deep Spatial-Temporal Graph Neural Networks

Are Defenses for Graph Neural Networks Robust?

Are Adversarial Robustness and Common Perturbation Robustness Independent Attributes ?

Understanding the effect of sparsity on neural networks robustness

Neural Architecture Design and Robustness: A Dataset

Achieving Adversarial Robustness Via Sparsity

A Comprehensive Analysis on Adversarial Robustness of Spiking Neural Networks

Exploring robust architectures for deep artificial neural networks

Post-Hoc Robustness Enhancement in Graph Neural Networks with Conditional Random Fields

How Robust Are Graph Neural Networks to Structural Noise?