Abstract:In general, deep neural networks (DNNs) are seriously overparameterized with enormous hardware resources demanded, which creates a heavy burden for inference applications especially for resource-constrained edge devices. To overcome this difficulty, there are two principal solutions: optimizing the overparameterized DNNs and designing high-efficiency DNN algorithms with lightweight architectures. In terms of the optimization methods, pruning is the most effective technique because the solution fundamentally optimizes the bloated DNNs by removing redundant structures from the network and can be seamlessly incorporated into all other optimization solutions as well as all kinds of DNN architectures. Nevertheless, the study reported in this paper reveals that the various excellent but also complicated pruning algorithms may not be as effective as proposals demonstrate and do not yield optimal solutions for all cases. In addition, the current lightweight DNN architectures are also overparameterized to a large extent. In this research, we propose a mechanism for determining lightweight DNN networks From Scratch (FS-DNN). First, we conduct a thorough study on the theoretical basis of evaluating the hardware resources demanded by DNNs, and establish the objective function for determining a lightweight DNN network. Based on the study, the theoretical FS-DNN for determining lightweight DNNs from scratch with high efficiency is proposed. Then, we perform a series of experiments with FS-DNN based lightweight DNNs on the public dataset CIFAR10/100 and a private dataset Kuzushiji, which prove the feasibility and efficiency of FS-DNN. According to the research, instead of adopting bloated DNN networks that demand complicated pruning algorithms to optimize the networks after the fact or the current so-called lightweight DNNs, the experimental results demonstrate that lightweight networks based on FS-DNN achieve superior performance in computing consumption with competitive or even better accuracy.

LightNets: the Concept of Weakening Layers

LWRN: Light-Weight Residual Network for Edge Detection

Lite It Fly: An All-Deformable-Butterfly Network

Lightweight Network Research Based on Deep Learning

Lightweight MIMO-WNet for Single Image Deblurring

On the importance of network architecture in training very deep neural networks

Make Deep Networks Shallow Again

LPRNet: Lightweight Deep Network by Low-rank Pointwise Residual Convolution

FalconNet: Factorization for the Light-weight ConvNets.

Super Interaction Neural Network.

Lightweight Deep Neural Network from Scratch

Deep Learning Architecture Improvement Based on Dynamic Pruning and Layer Fusion

WPNets and PWNets: from the Perspective of Channel Fusion.

Peeking Behind the Curtains of Residual Learning

Layer Pruning via Fusible Residual Convolutional Block for Deep Neural Networks

LRPRNet: Lightweight Deep Network by Low-Rank Pointwise Residual Convolution

More is Less: A More Complicated Network with Less Inference Complexity.

Log-DenseNet: How to Sparsify a DenseNet

Efficient Layer Compression Without Pruning

Research on Lightweight Convolutional Neural Network Technology

The Shallow End: Empowering Shallower Deep-Convolutional Networks through Auxiliary Outputs