Abstract:We propose an algorithm capable of identifying and eliminating irrelevant layers of a neural network during the early stages of training. In contrast to weight or filter-level pruning, layer pruning reduces the harder to parallelize sequential computation of a neural network. We employ a structure using residual connections around nonlinear network sections that allow the flow of information through the network once a nonlinear section is pruned. Our approach is based on variational inference principles using Gaussian scale mixture priors on the neural network weights and allows for substantial cost savings during both training and inference. More specifically, the variational posterior distribution of scalar Bernoulli random variables multiplying a layer weight matrix of its nonlinear sections is learned, similarly to adaptive layer-wise dropout. To overcome challenges of concurrent learning and pruning such as premature pruning and lack of robustness with respect to weight initialization or the size of the starting network, we adopt the "flattening" hyper-prior on the prior parameters. We prove that, as a result of its usage, the solutions of the resulting optimization problem describe deterministic networks with parameters of the posterior distribution at either 0 or 1. We formulate a projected SGD algorithm and prove its convergence to such a solution using stochastic approximation results. In particular, we prove conditions that lead to a layer's weights converging to zero and derive practical pruning conditions from the theoretical results. The proposed algorithm is evaluated on the MNIST, CIFAR-10 and ImageNet datasets and common LeNet, VGG16 and ResNet architectures. The simulations demonstrate that our method achieves state-of the-art performance for layer pruning at reduced computational cost in distinction to competing methods due to the concurrent training and pruning.

Training a V1 Like Layer Using Gabor Filters in Convolutional Neural Networks

Gabor Binary Layer in Convolutional Neural Networks

Convolutional neural network models of neuronal responses in macaque V1 reveal limited non-linear processing

Pruning by Training: A Novel Deep Neural Network Compression Framework for Image Processing.

Convolutional neural network models applied to neuronal responses in macaque V1 reveal limited nonlinear processing

GaborNet: Gabor filters with learnable parameters in deep convolutional neural networks

Gradient Local Binary Pattern Layer to Initialize the Convolutional Neural Networks

Concurrent Training and Layer Pruning of Deep Neural Networks

Non-linear Convolution Filters for CNN-based Learning

GaborNet Visual Encoding: A Lightweight Region-Based Visual Encoding Model With Good Expressiveness and Biological Interpretability

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Deep Gabor Convolution Network for Person Re-Identification.

Explaining V1 Properties with a Biologically Constrained Deep Learning Architecture

Comparison between layer-to-layer network training and conventional network training using Deep Convolutional Neural Networks

Towards Biologically Plausible Convolutional Networks

Layerwise complexity-matched learning yields an improved model of cortical area V2

Energy-Efficient Gabor Kernels in Neural Networks with Genetic Algorithm Training Method

How Do Training Methods Influence the Utilization of Vision Models?

Emergence of V1 connectivity pattern and Hebbian rule in a performance-optimized artificial neural network

Transformation-invariant Gabor Convolutional Networks

Hyperspectral Images Classification with Gabor Filtering and Convolutional Neural Network