Abstract:In overhead image segmentation tasks, including additional spectral bands beyond the traditional RGB channels can improve model performance. However, it is still unclear how incorporating this additional data impacts model robustness to adversarial attacks and natural perturbations. For adversarial robustness, the additional information could improve the model's ability to distinguish malicious inputs, or simply provide new attack avenues and vulnerabilities. For natural perturbations, the additional information could better inform model decisions and weaken perturbation effects or have no significant influence at all. In this work, we seek to characterize the performance and robustness of a multispectral (RGB and near infrared) image segmentation model subjected to adversarial attacks and natural perturbations. While existing adversarial and natural robustness research has focused primarily on digital perturbations, we prioritize on creating realistic perturbations designed with physical world conditions in mind. For adversarial robustness, we focus on data poisoning attacks whereas for natural robustness, we focus on extending ImageNet-C common corruptions for fog and snow that coherently and self-consistently perturbs the input data. Overall, we find both RGB and multispectral models are vulnerable to data poisoning attacks regardless of input or fusion architectures and that while physically realizable natural perturbations still degrade model performance, the impact differs based on fusion architecture and input data.

What problem does this paper attempt to address?

The paper primarily investigates the robustness issues of multispectral image segmentation models in deep learning when facing natural perturbations and data poisoning attacks. ### Research Background and Objectives - **Research Background**: With the increase of publicly available satellite imagery data and the development of large annotated satellite image datasets, deep learning models have achieved significant results in remote sensing applications (such as land cover classification, agricultural monitoring, and disaster assessment). Typically, satellite sensors collect multispectral image data, which contains information beyond the traditional red, green, and blue (RGB) bands. Deep learning models utilizing multispectral images have shown superior performance in certain applications compared to models using only RGB images. - **Research Objectives**: This paper aims to quantify the performance and robustness of multispectral image segmentation models (especially those combining RGB and near-infrared [NIR] bands) when subjected to adversarial attacks and natural perturbations. It focuses particularly on realistic perturbations designed under physical world conditions and emphasizes data poisoning attacks as well as extending the treatment methods for natural perturbations like fog and snow from ImageNet-C. ### Main Contributions - **Adversarial Robustness**: Investigated data poisoning attacks against multispectral image segmentation models. It was found that both RGB and multispectral models are susceptible to such attacks under different input architectures and fusion strategies. - **Natural Robustness**: Developed a method to apply natural perturbations (such as fog, snow) to multispectral images in a physically plausible manner. Experimental results show that early fusion models are more resistant to this type of natural perturbation compared to late fusion models and models using only RGB. - **Model Fusion Strategies**: Explored different combinations of input bands (NIR, RGB, RGB+NIR) and model architectures (early fusion vs. late fusion) to understand how these variables affect overall robustness and the potential trade-offs with model performance. ### Overview of Experimental Results - **Data Poisoning Attacks**: All models (including single-channel NIR models, RGB models, and models fusing RGB and NIR) showed vulnerability to data poisoning attacks. Although additional spectral band information can improve model performance, it also reduces adversarial robustness. - **Natural Perturbations**: For physically plausible natural perturbations (such as snow and fog), early fusion models demonstrated better robustness than late fusion models and models using only RGB. - **Model Performance**: In the presence of natural perturbations, models including NIR information generally performed better, especially in the recognition of vegetation categories, benefiting from the NIR band's ability to penetrate through adverse weather conditions. In summary, the paper validates through experiments the characteristics of multispectral image segmentation models in terms of adversarial and natural robustness and provides insights into the choice of model fusion strategies.

Quantifying the robustness of deep multispectral segmentation models against natural perturbations and data poisoning

Evaluating the Adversarial Robustness of Semantic Segmentation: Trying Harder Pays Off

Towards Robustness against Unsuspicious Adversarial Examples

Interpreting Global Perturbation Robustness of Image Models using Axiomatic Spectral Importance Decomposition

Robustness Analysis on Foundational Segmentation Models

On Adversarial Robustness of Point Cloud Semantic Segmentation

Sparse vs Contiguous Adversarial Pixel Perturbations in Multimodal Models: An Empirical Analysis

Towards Reliable Evaluation and Fast Training of Robust Semantic Segmentation Models

On the Robustness of Semantic Segmentation Models to Adversarial Attacks

Exploring Adversarial Robustness of Multi-Sensor Perception Systems in Self Driving

Robustifying Deep Networks for Medical Image Segmentation

Unrevealed Threats: A Comprehensive Study of the Adversarial Robustness of Underwater Image Enhancement Models

Color and Edge-Aware Adversarial Image Perturbations

Universal Adversarial Perturbations Against Semantic Image Segmentation

Transparency Distortion Robustness for SOTA Image Segmentation Tasks

Learning perturbation sets for robust machine learning

On the Robustness of Large Multimodal Models Against Image Adversarial Attacks

Analyzing Adversarial Robustness of Deep Neural Networks in Pixel Space: a Semantic Perspective

Stealthy Adversarial Examples for Semantic Segmentation in Remote Sensing

Towards Evaluating the Robustness of Visual State Space Models

Sensitivity-Informed Augmentation for Robust Segmentation