Abstract:Abstract Computational neuroscience studies have shown that the structure of neural variability to an unchanged stimulus affects the amount of information encoded. Some artificial deep neural networks, such as those with Monte Carlo dropout layers, also have variable responses when the input is fixed. However, the structure of the trial-by-trial neural covariance in neural networks with dropout has not been studied, and its role in decoding accuracy is unknown. We studied the above questions in a convolutional neural network model with dropout in both the training and testing phases. We found that trial-by-trial correlation between neurons (i.e., noise correlation) is positive and low dimensional. Neurons that are close in a feature map have larger noise correlation. These properties are surprisingly similar to the findings in the visual cortex. We further analyzed the alignment of the main axes of the covariance matrix. We found that different images share a common trial-by-trial noise covariance subspace, and they are aligned with the global signal covariance. This evidence that the noise covariance is aligned with signal covariance suggests that noise covariance in dropout neural networks reduces network accuracy, which we further verified directly with a trial-shuffling procedure commonly used in neuroscience. These findings highlight a previously overlooked aspect of dropout layers that can affect network performance. Such dropout networks could also potentially be a computational model of neural variability.

Excitation Dropout: Encouraging Plasticity in Deep Neural Networks

Guided Dropout: Improving Deep Networks Without Increased Computation

Adaptive Dropout Method Based on Biological Principles

Probing the Structure and Functional Properties of the Dropout-Induced Correlated Variability in Convolutional Neural Networks

Adaptive Sparse Dropout: Learning the Certainty and Uncertainty in Deep Neural Networks

AutoDropout: Learning Dropout Patterns to Regularize Deep Networks

CamDrop

The Overfocusing Bias of Convolutional Neural Networks: A Saliency-Guided Regularization Approach

Multiple Explanations for Neural Network Based Dropout Prediction

Y-Drop: A Conductance based Dropout for fully connected layers

Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks

Energy-Based Dropout in Restricted Boltzmann Machines: Why Not Go Random

Understanding plasticity in neural networks

NFLB Dropout: Improve Generalization Ability by Dropping out the Best -A Biologically Inspired Adaptive Dropout Method for Unsupervised Learning.

Dropout as an Implicit Gating Mechanism For Continual Learning

R-Drop: Regularized Dropout for Neural Networks.

Continuous Dropout

Can We Understand Plasticity Through Neural Collapse?

Implicit Regularization of Dropout

Advanced Dropout: A Model-free Methodology for Bayesian Dropout Optimization.

Flipover outperforms dropout in deep learning