Abstract:In general, deep neural networks (DNNs) are seriously overparameterized with enormous hardware resources demanded, which creates a heavy burden for inference applications especially for resource-constrained edge devices. To overcome this difficulty, there are two principal solutions: optimizing the overparameterized DNNs and designing high-efficiency DNN algorithms with lightweight architectures. In terms of the optimization methods, pruning is the most effective technique because the solution fundamentally optimizes the bloated DNNs by removing redundant structures from the network and can be seamlessly incorporated into all other optimization solutions as well as all kinds of DNN architectures. Nevertheless, the study reported in this paper reveals that the various excellent but also complicated pruning algorithms may not be as effective as proposals demonstrate and do not yield optimal solutions for all cases. In addition, the current lightweight DNN architectures are also overparameterized to a large extent. In this research, we propose a mechanism for determining lightweight DNN networks From Scratch (FS-DNN). First, we conduct a thorough study on the theoretical basis of evaluating the hardware resources demanded by DNNs, and establish the objective function for determining a lightweight DNN network. Based on the study, the theoretical FS-DNN for determining lightweight DNNs from scratch with high efficiency is proposed. Then, we perform a series of experiments with FS-DNN based lightweight DNNs on the public dataset CIFAR10/100 and a private dataset Kuzushiji, which prove the feasibility and efficiency of FS-DNN. According to the research, instead of adopting bloated DNN networks that demand complicated pruning algorithms to optimize the networks after the fact or the current so-called lightweight DNNs, the experimental results demonstrate that lightweight networks based on FS-DNN achieve superior performance in computing consumption with competitive or even better accuracy.

Efficient Deep Structure Learning for Resource-Limited IoT Devices

An efficient pruning scheme of deep neural networks for Internet of Things applications

Class-Aware Pruning for Efficient Neural Networks

Efficient Structure Slimming for Spiking Neural Networks

Identifying and Pruning Redundant Structures for Deep Neural Networks

Smart-DNN: Efficiently Reducing the Memory Requirements of Running Deep Neural Networks on Resource-constrained Platforms

Three-Stage Global Channel Pruning for Resources-Limited Platform

Complexity-Driven CNN Compression for Resource-constrained Edge AI

Only Train Once: A One-Shot Neural Network Training And Pruning Framework

A Dynamic Pruning Method on Multiple Sparse Structures in Deep Neural Networks

Enabling High Performance Deep Learning Networks on Embedded Systems

Improving Device-Edge Cooperative Inference of Deep Learning via 2-Step Pruning

NAIR: an Efficient Distributed Deep Learning Architecture for Resource Constrained IoT System

An efficient pruning and fine-tuning method for deep spiking neural network

Learning Low Resource Consumption CNN through Pruning and Quantization

Lightweight Deep Neural Network from Scratch

Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition

Cloud–Edge Collaborative Inference with Network Pruning

Efficient Network Construction Through Structural Plasticity

Structured Term Pruning for Computational Efficient Neural Networks Inference

DynSNN: A Dynamic Approach to Reduce Redundancy in Spiking Neural Networks