Abstract:Deep neural networks (DNNs) have been regarded as fundamental tools for many disciplines. Meanwhile, they are known for their large-scale parameters, high redundancy in weights, and extensive computing resource consumptions, which pose a tremendous challenge to the deployment in real-time applications or on resource-constrained devices. To cope with this issue, compressing DNNs for accelerating its inference has drawn extensive interest recently. The basic idea is to prune parameters with little performance degradation. However, the overparameterized nature and the conflict between parameters reduction and performance maintenance make it prohibitive to manually search the pruning parameter space. In this paper, we formally establish filter pruning as a multiobjective optimization problem, and propose a knee-guided evolutionary algorithm (KGEA) that can automatically search for the solution with quality tradeoff between the scale of parameters and performance, in which both conflicting objectives can be optimized simultaneously. In particular, by incorporating a minimum Manhattan distance approach, the search effort in the proposed KGEA is explicitly guided toward the knee area, which greatly facilitates the manual search for a good tradeoff solution. Moreover, the parameter importance is directly estimated on the criterion of performance loss, which can robustly identify the redundancy. In addition to the knee solution, a performance-improved model can also be found in a fine-tuning-free fashion. The experiments on compressing fully convolutional LeNet and VGG-19 networks validate the superiority of the proposed algorithm over the state-of-the-art competing methods.

Evolutionary Shallowing Deep Neural Networks at Block Levels

A Gradient-Guided Evolutionary Approach to Training Deep Neural Networks

A Survey on Evolutionary Construction of Deep Neural Networks.

Pruning Deep Convolutional Neural Networks Architectures with Evolution Strategy

A Survey on Evolutionary Neural Architecture Search

Evolutionary Compression of Deep Neural Networks for Biomedical Image Segmentation

G-EvoNAS: Evolutionary Neural Architecture Search Based on Network Growth

A Fast Evolutionary Knowledge Transfer Search for Multiscale Deep Neural Architecture

Compression of deep neural networks: bridging the gap between conventional-based pruning and evolutionary approach

A Knee-Guided Evolutionary Algorithm for Compressing Deep Neural Networks

Is it conceivable that neurogenesis, neural Darwinism, and species evolution could all serve as inspiration for the creation of evolutionary deep neural networks?

Guest Editorial Evolutionary Neural Architecture Search

Evolutionary neural networks for deep learning: a review

A Survey of Advances in Evolutionary Neural Architecture Search

Accurate and Fast Deep Evolutionary Networks Structured Representation Through Activating and Freezing Dense Networks

A Self-adaptive neuroevolution approach to constructing Deep Neural Network architectures across different types

ESL-SNNs: An Evolutionary Structure Learning Strategy for Spiking Neural Networks

Brain-inspired Evolutionary Architectures for Spiking Neural Networks

Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon

Exploring the Imposition of Synaptic Precision Restrictions For Evolutionary Synthesis of Deep Neural Networks

Evolving Deep Neural Networks via Cooperative Coevolution With Backpropagation