Abstract:Previous works show global covariance pooling (GCP) has great potential to improve deep architectures especially on visual recognition tasks, where post-normalization of GCP plays a very important role in final performance. Although several post-normalization strategies have been studied, these methods pay more close attention to effect of normalization on covariance representations rather than the whole GCP networks, and their effectiveness requires further understanding. Meanwhile, existing effective post-normalization strategies (e.g., matrix power normalization) usually suffer from high computational complexity (e.g., O(d(3)) for d-dimensional inputs). To handle above issues, this work first analyzes the effect of post-normalization from the perspective of training GCP networks. Particularly, we for the first time show that effective post-normalization can make a good trade-off between representation decorrelation and information preservation for GCP, which are crucial to alleviate over-fitting and increase representation ability of deep GCP networks, respectively. Based on this finding, we can improve existing post-normalization methods with some small modifications, providing further support to our observation. Furthermore, this finding encourages us to propose a novel pre-normalization method for GCP (namely DropCov), which develops an adaptive channel dropout on features right before GCP, aiming to reach trade-off between representation decorrelation and information preservation in a more efficient way. Our DropCov only has a linear complexity of O(d), while being free for inference. Extensive experiments on various benchmarks (i.e., ImageNet-1K, ImageNet-C, ImageNet-A, Stylized-ImageNet, and iNat2017) show our DropCov is superior to the counterparts in terms of efficiency and effectiveness, and provides a simple yet effective method to improve performance of deep architectures involving both deep convolutional neural networks (CNNs) and vision transformers (ViTs).

Predropout & Inhibition - A Brain-Inspired Method for Convolutional Neural Network.

A Pruning Method Based on the Dissimilarity of Angle among Channels and Filters

FocusedDropout for Convolutional Neural Network

Attentiondrop For Convolutional Neural Networks

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Sensitivity-Oriented Layer-Wise Acceleration and Compression for Convolutional Neural Network.

Effective and Efficient Dropout for Deep Convolutional Neural Networks

Cropout: A General Mechanism for Reducing Overfitting on Convolutional Neural Networks

Convolutional Neural Networks Exploiting Attributes of Biological Neurons

Accelerating Convolutional Neural Networks by Removing Interspatial and Interkernel Redundancies.

Flip-Rotate-Pooling Convolution and Split Dropout on Convolution Neural Networks for Image Classification

Learning Low Resource Consumption CNN through Pruning and Quantization

Brain inspired Robust Vision using Convolutional Neural Networks with Feedback

A Down-sampling Method Based on the Discrete Wavelet Transform for CNN Classification

Synaptic Strength For Convolutional Neural Network

DropCov: A Simple yet Effective Method for Improving Deep Architectures

Brain Cognition-Inspired Dual-Pathway CNN Architecture for Image Classification

Loss-Driven Channel Pruning of Convolutional Neural Networks.

Group-wise Inhibition based Feature Regularization for Robust Classification

Advanced Convolutional Neural Network with Feedforward Inhibition.

Enabling Efficient Bayesian Convolutional Neural Network with Feature Pre-extraction and Pre-pooling Strategies