Abstract:We investigate the impact of entropy change in deep learning systems by noise injection at different levels, including the embedding space and the image. The series of models that employ our methodology are collectively known as Noisy Neural Networks (NoisyNN), with examples such as NoisyViT and NoisyCNN. Noise is conventionally viewed as a harmful perturbation in various deep learning architectures, such as convolutional neural networks (CNNs) and vision transformers (ViTs), as well as different learning tasks like image classification and transfer learning. However, this work shows noise can be an effective way to change the entropy of the learning system. We demonstrate that specific noise can boost the performance of various deep models under certain conditions. We theoretically prove the enhancement gained from positive noise by reducing the task complexity defined by information entropy and experimentally show the significant performance gain in large image datasets, such as the ImageNet. Herein, we use the information entropy to define the complexity of the task. We categorize the noise into two types, positive noise (PN) and harmful noise (HN), based on whether the noise can help reduce the task complexity. Extensive experiments of CNNs and ViTs have shown performance improvements by proactively injecting positive noise, where we achieved an unprecedented top 1 accuracy of 95$\%$ on ImageNet. Both theoretical analysis and empirical evidence have confirmed that the presence of positive noise, can benefit the learning process, while the traditionally perceived harmful noise indeed impairs deep learning models. The different roles of noise offer new explanations for deep models on specific tasks and provide a new paradigm for improving model performance. Moreover, it reminds us that we can influence the performance of learning systems via information entropy change.

Seeing is Not Always Believing: the Space of Harmless Perturbations

Contrasting Adversarial Perturbations: The Space of Harmless Perturbations

Fooling Neural Network Interpretations - Adversarial Noise to Attack Images.

Searching for the Essence of Adversarial Perturbations

Investigating Human-Identifiable Features Hidden in Adversarial Perturbations

Trustworthy adaptive adversarial perturbations in social networks

Adversarial Examples Detection Beyond Image Space.

Can Protective Perturbation Safeguard Personal Data from Being Exploited by Stable Diffusion?

Towards Robustness against Unsuspicious Adversarial Examples

Perturbative Neural Networks

Fidelity of Interpretability Methods and Perturbation Artifacts in Neural Networks

Sensitivity Analysis of Deep Neural Networks

Analyzing the Noise Robustness of Deep Neural Networks

NoisyNN: Exploring the Impact of Information Entropy Change in Learning Systems

Exploiting vulnerabilities of deep neural networks for privacy protection

Face image de‐identification by feature space adversarial perturbation

Analyzing Adversarial Robustness of Deep Neural Networks in Pixel Space: a Semantic Perspective

Detection of Adversarial Attacks via Disentangling Natural Images and Perturbations

Theoretical Understanding of Learning from Adversarial Perturbations

Intriguing properties of neural networks

On Residual Networks Learning a Perturbation from Identity