Abstract:Deep Neural Networks (DNNs) have achieved remarkable success in various real-world applications. However, running a Deep Neural Network (DNN) typically requires hundreds of megabytes of memory footprints, making it challenging to deploy on resource-constrained platforms such as mobile devices and IoT. Although mainstream DNNs compression techniques such as pruning, distillation, and quantization can reduce the memory overhead of model parameters during DNN inference, they suffer from three limitations: (i) low model compression ratio for the lightweight DNN structures with little redundancy, (ii) potential degradation in model inference accuracy, and (iii) inadequate memory compression ratio is attributable to ignoring the layering property of DNN inference. To address these issues, we propose a lightweight memory-efficient DNN inference framework called Smart-DNN+, which significantly reduces the memory costs of DNN inference without degrading the model quality. Specifically, ① Smart-DNN+ applies a layerwise binary-quantizer with a remapping mechanism to greatly reduce the model size by quantizing the typical floating-point DNN weights of 32-bit to the 1-bit signs layer by layer. To maintain model quality, ② Smart-DNN+ employs a bucket-encoder to keep the compressed quantization error by encoding the multiple similar floating-point residuals into the same integer bucket IDs. When running the compressed DNN in the user’s device, ③ Smart-DNN+ utilizes a partially decompressing strategy to greatly reduce the required memory overhead by first loading the compressed DNNs in memory and then dynamically decompressing the required materials for model inference layer by layer. Experimental results on popular DNNs and datasets demonstrate that Smart-DNN+ achieves lower 0.17%–0.92% memory costs at lower runtime overheads compared with the states of the art without degrading the inference accuracy. Moreover, Smart-DNN+ potentially reduces the inference runtime up to 2.04× that of conventional DNN inference workflow.

Residual encoding framework to compress DNN parameters for fast transfer

Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression

Efficient Neural Network Compression Inspired by Compressive Sensing.

Releasing the Parameter Latency of Neural Representation for High-Efficiency Video Compression

Design of a Quantization-Based DNN Delta Compression Framework for Model Snapshots and Federated Learning

A Highly Efficient Training-Aware Convolutional Neural Network Compression Paradigm

Fast data-free model compression via dictionary-pair reconstruction

Deep Learning Model Compression with Rank Reduction in Tensor Decomposition.

Parameters Sharing in Residual Neural Networks

Compressing Deep Networks by Neuron Agglomerative Clustering

Compressing and Accelerating Neural Network for Facial Point Localization.

Low-Rate Feature Compression for Collaborative Intelligence: Reducing Redundancy in Spatial and Statistical Levels

QD-Compressor: a Quantization-based Delta Compression Framework for Deep Neural Networks

Dictionary Pair-based Data-Free Fast Deep Neural Network Compression

Near-Lossless Gradient Compression for Data-Parallel Distributed DNN Training

Delta-DNN: Efficiently Compressing Deep Neural Networks Via Exploiting Floats Similarity.

Compressing Deep Neural Networks With Sparse Matrix Factorization

FedComp: A Federated Learning Compression Framework for Resource-Constrained Edge Computing Devices

Neural Network Coding of Difference Updates for Efficient Distributed Learning Communication

Smart-DNN+: A Memory-efficient Neural Networks Compression Framework for the Model Inference

Overfitting the Data: Compact Neural Video Delivery Via Content-aware Feature Modulation