Abstract:Including human analysis has the potential to positively affect the robustness of Deep Neural Networks and is relatively unexplored in the Adversarial Machine Learning literature. Neural network visual explanation maps have been shown to be prone to adversarial attacks. Further research is needed in order to select robust visualizations of explanations for the image analyst to evaluate a given model. These factors greatly impact Human-In-The-Loop (HITL) evaluation tools due to their reliance on adversarial images, including explanation maps and measurements of robustness. We believe models of human visual attention may improve interpretability and robustness of human-machine imagery analysis systems. Our challenge remains, how can HITL evaluation be robust in this adversarial landscape?

What problem does this paper attempt to address?

This paper aims to solve the problem of how to overcome adversarial attacks in Human - in - the - Loop (HITL) applications. Specifically, the paper focuses on the following key issues: 1. **The impact of adversarial attacks on deep neural networks**: - Adversarial images can degrade the performance of the model, and these perturbations are often designed to evade detection by image analysts. - Adversarial attacks not only affect the prediction accuracy of the model but also disrupt or circumvent additional tools used to evaluate the model, such as explanation graphs and robustness metrics. 2. **The role of human analysts in an adversarial environment**: - In the field of adversarial machine learning, incorporating human analysis may help improve the robustness of deep neural networks, but there is relatively little research in this area. - Visual explanation graphs of neural networks (such as those generated by Grad - CAM) are vulnerable to adversarial attacks, which makes it difficult for human analysts to accurately evaluate the model. 3. **How to select robust visual explanations**: - In order to enable image analysts to effectively evaluate a given model, further research is needed on how to select robust visual explanation methods. - These factors have a significant impact on HITL evaluation tools that rely on adversarial images, including explanation graphs and robustness measurements. 4. **Combining human visual attention models to improve interpretability and robustness**: - The author believes that human visual attention models may help improve the interpretability and robustness of human - machine image analysis systems. - However, the challenge lies in how to ensure that these models themselves are not affected by adversarial attacks and how to combine human and machine attention to identify adversarial images. ### Summary of the main problems in the paper The core problem of the paper is: **How to keep HITL evaluation tools robust in an adversarial environment?** Specifically, the paper explores the following aspects: - How to ensure that explanation graphs and other auxiliary tools remain reliable under adversarial attacks. - How to use human visual attention models to enhance the detection ability of adversarial attacks. - How to design effective HITL tools so that human analysts can better understand and evaluate models in an adversarial environment. Through the research of these problems, the author hopes to promote the further development of HITL applications in the field of adversarial machine learning.

Overcoming Adversarial Attacks for Human-in-the-Loop Applications

Fooling Neural Network Interpretations - Adversarial Noise to Attack Images.

Attack As Defense: Characterizing Adversarial Examples Using Robustness.

Adversarial alignment: Breaking the trade-off between the strength of an attack and its relevance to human perception

Adversarial Attacks Hidden in Plain Sight

Exploring Adversarial Attacks on Neural Networks: An Explainable Approach

Investigating Human-Identifiable Features Hidden in Adversarial Perturbations

An Extended Study of Human-like Behavior under Adversarial Training

Visual Analytics of Neuron Vulnerability to Adversarial Attacks on Convolutional Neural Networks

When and How to Fool Explainable Models (and Humans) with Adversarial Examples

Panda or not Panda? Understanding Adversarial Attacks with Interactive Visualization

Adversarial Attacks on Machine Learning-Aided Visualizations

Interpreting Adversarial Examples in Deep Learning: A Review

Explaining and Harnessing Adversarial Examples

Explaining Vulnerabilities to Adversarial Machine Learning through Visual Analytics

Understanding Deep Learning defenses Against Adversarial Examples Through Visualizations for Dynamic Risk Assessment

Attacking vision-based perception in end-to-end autonomous driving models

How Real Is Real? A Human Evaluation Framework for Unrestricted Adversarial Examples

Adversarial Examples that Fool both Computer Vision and Time-Limited Humans

Leveraging the Human Ventral Visual Stream to Improve Neural Network Robustness