Abstract:Deep convolutional neural networks (CNNs) have achieved impressive performance in many computer vision tasks. However, their large model sizes require heavy computational resources, making pruning redundant filters from existing pre-trained CNNs an essential task in developing efficient models for resource-constrained devices. Whole-network filter pruning algorithms prune varying fractions of filters from each layer, hence providing greater flexibility. Current whole-network pruning methods are either computationally expensive due to the need to calculate the loss for each pruned filter using a training dataset, or use various heuristic / learned criteria for determining the pruning fractions for each layer. This paper proposes a two-level hierarchical approach for whole-network filter pruning which is efficient and uses the classification loss as the final criterion. The lower-level algorithm (called filter-pruning) uses a sparse-approximation formulation based on linear approximation of filter weights. We explore two algorithms: orthogonal matching pursuit-based greedy selection and a greedy backward pruning approach. The backward pruning algorithm uses a novel closed-form error criterion for efficiently selecting the optimal filter at each stage, thus making the whole algorithm much faster. The higher-level algorithm (called layer-selection) greedily selects the best-pruned layer (pruning using the filter-selection algorithm) using a global pruning criterion. We propose algorithms for two different global-pruning criteria: (1) layer-wise relative error (HBGS), and (2) final classification error (HBGTS). Our suite of algorithms outperforms state-of-the-art pruning methods on ResNet18, ResNet32, ResNet56, VGG16, and ResNext101. Our method reduces the RAM requirement for ResNext101 from 7.6 GB to 1.5 GB and achieves a 94% reduction in FLOPS without losing accuracy on CIFAR-10.

A Local Training and Pruning Approach for Neural Networks.

Class-Aware Pruning for Efficient Neural Networks

Novel Training Algorithms for Long Short-Term Memory Neural Network

EKFNet: Edge-based Kalman filter network for real-time EEG signal denoising

Adaptive, global, extended Kalman filters for training feedforward neural networks

Improved training of cellular SRN using Unscented Kalman Filtering for ADP

A Cognition-Based Method to Ease the Computational Load for an Extended Kalman Filter

An on-line training algorithm for echo state networks with extended kalman filter

Students and teachers learning together: a robust training strategy for neural network pruning

Continuous-discrete extended Kalman filtering based on the neural ordinary differential equations method

PQK: Model Compression via Pruning, Quantization, and Knowledge Distillation

Neural network aided extended Kalman filtering for inverse imaging of cardiac transmembrane potential

A Novel Kalman Filter for Nonlinear Non-Gaussian Systems and Its Applications in Self-localization

Distillation Sparsity Training Algorithm for Accelerating Convolutional Neural Networks in Embedded Systems

Trainable Quaternion Extended Kalman Filter with Multi-Head Attention for Dead Reckoning in Autonomous Ground Vehicles

Neural Network Pruning with Residual-Connections and Limited-Data

EKFNet: Learning System Noise Covariance Parameters for Nonlinear Tracking

Skeleton Neural Networks via Low-rank Guided Filter Pruning

Optimal filtering algorithms for fast learning in feedforward neural networks

Sparse optimization guided pruning for neural networks

A Greedy Hierarchical Approach to Whole-Network Filter- Pruning in CNNs