Abstract:We investigate the impact of entropy change in deep learning systems by noise injection at different levels, including the embedding space and the image. The series of models that employ our methodology are collectively known as Noisy Neural Networks (NoisyNN), with examples such as NoisyViT and NoisyCNN. Noise is conventionally viewed as a harmful perturbation in various deep learning architectures, such as convolutional neural networks (CNNs) and vision transformers (ViTs), as well as different learning tasks like image classification and transfer learning. However, this work shows noise can be an effective way to change the entropy of the learning system. We demonstrate that specific noise can boost the performance of various deep models under certain conditions. We theoretically prove the enhancement gained from positive noise by reducing the task complexity defined by information entropy and experimentally show the significant performance gain in large image datasets, such as the ImageNet. Herein, we use the information entropy to define the complexity of the task. We categorize the noise into two types, positive noise (PN) and harmful noise (HN), based on whether the noise can help reduce the task complexity. Extensive experiments of CNNs and ViTs have shown performance improvements by proactively injecting positive noise, where we achieved an unprecedented top 1 accuracy of 95$\%$ on ImageNet. Both theoretical analysis and empirical evidence have confirmed that the presence of positive noise, can benefit the learning process, while the traditionally perceived harmful noise indeed impairs deep learning models. The different roles of noise offer new explanations for deep models on specific tasks and provide a new paradigm for improving model performance. Moreover, it reminds us that we can influence the performance of learning systems via information entropy change.

Machine learning of network inference enhancement from noisy measurements

Robust Network Enhancement from Flawed Networks.

Noise is the Fatal Poison: A Noise-aware Network for Noisy Dataset Classification

Improving Data-Driven Inferential Sensor Modeling by Industrial Knowledge: A Bayesian Perspective

Solving the inverse problem of noise-driven dynamic networks.

Effective Model Integration Algorithm for Improving Link and Sign Prediction in Complex Networks

Reconstruction of Dynamic Networks with Time-Delayed Interactions in the Presence of Fast-Varying Noises

Exploiting Noise As a Resource for Computation and Learning in Spiking Neural Networks

Enhancing Robustness of Link Prediction for Noisy Complex Networks

Network enhancement as a general method to denoise weighted biological networks

Network Enhancement: a general method to denoise weighted biological networks

A Co-Evolutionary Model for Inferring Online Social Network User Behaviors.

A Model Framework for the Enhancement of Community Detection in Complex Networks

Deep-learning reconstruction of complex dynamical networks from incomplete data

Network Diffusions via Neural Mean-Field Dynamics

Learning of networked spreading models from noisy and incomplete data

Noise-tolerance community detection and evolution in dynamic social networks

NoisyNN: Exploring the Impact of Information Entropy Change in Learning Systems

Machine Learning Based Linear and Nonlinear Noise Estimation

Inferring networks from time series: a neural approach

Reconstruction of Noise-Driven Nonlinear Dynamic Networks with Some Hidden Nodes