Abstract:Due to a resource-constrained environment, network compression has become an important part of deep neural networks research. In this paper, we propose a new compression method, \textit{Inter-Layer Weight Prediction} (ILWP) and quantization method which quantize the predicted residuals between the weights in all convolution layers based on an inter-frame prediction method in conventional video coding schemes. Furthermore, we found a phenomenon \textit{Smoothly Varying Weight Hypothesis} (SVWH) which is that the weights in adjacent convolution layers share strong similarity in shapes and values, i.e., the weights tend to vary smoothly along with the layers. Based on SVWH, we propose a second ILWP and quantization method which quantize the predicted residuals between the weights in adjacent convolution layers. Since the predicted weight residuals tend to follow Laplace distributions with very low variance, the weight quantization can more effectively be applied, thus producing more zero weights and enhancing the weight compression ratio. In addition, we propose a new \textit{inter-layer loss} for eliminating non-texture bits, which enabled us to more effectively store only texture bits. That is, the proposed loss regularizes the weights such that the collocated weights between the adjacent two layers have the same values. Finally, we propose an ILWP with an inter-layer loss and quantization method. Our comprehensive experiments show that the proposed method achieves a much higher weight compression rate at the same accuracy level compared with the previous quantization-based compression methods in deep neural networks.

A High-Dimensional SOFM Vector Quantizer with Weightless Neural Predictor

An Image Compressing Algorithm Based on PCA/SOFM Hybrid Neural Network

A Predictive VQ Based Video Compression Scheme

Semantic-Oriented Feature Compression for Resource-Constrained Distributed Sensor Networks

A Compression Algorithm of Hyperspectral Remote Sensing Image Based on Vector Quantization

An Adaptive Vector Quantization Based on Neural Network

Distributed Vector Quantization over Sensor Network

Image coding using Address-Dependent Vector Quantization based on Kohonen neural network

Study on SOFM-based image vector quantization

Space Efficient Quantization for Deep Convolutional Neural Networks

Neighboring Pixels Based Low Complexity Predictive Vector Quantization Algorithms for Image Coding

Dynamically restricted codebook-based vector quantisation scheme for mesh geometry compression

VQ Image Coding Using Sub-Vector Techniques.

Quantization Aware Factorization for Deep Neural Network Compression

Selectively Tree-Structured Vector Quantizer Using Kohonen Neural Network

Deep Neural Network Compression With Single and Multiple Level Quantization

An Inter-Layer Weight Prediction and Quantization for Deep Neural Networks based on a Smoothly Varying Weight Hypothesis

SQuant: On-the-Fly Data-Free Quantization Via Diagonal Hessian Approximation

A VLSI neural processor for image data compression using self-organization networks

DeepCoder: A Deep Neural Network Based Video Compression