Abstract:As well known, the huge memory and compute costs of both artificial neural networks (ANNs) and spiking neural networks (SNNs) greatly hinder their deployment on edge devices with high efficiency. Model compression has been proposed as a promising technique to improve the running efficiency via parameter and operation reduction, whereas this technique is mainly practiced in ANNs rather than SNNs. It is interesting to answer how much an SNN model can be compressed without compromising its functionality, where two challenges should be addressed: 1) the accuracy of SNNs is usually sensitive to model compression, which requires an accurate compression methodology and 2) the computation of SNNs is event-driven rather than static, which produces an extra compression dimension on dynamic spikes. To this end, we realize a comprehensive SNN compression through three steps. First, we formulate the connection pruning and weight quantization as a constrained optimization problem. Second, we combine spatiotemporal backpropagation (STBP) and alternating direction method of multipliers (ADMMs) to solve the problem with minimum accuracy loss. Third, we further propose activity regularization to reduce the spike events for fewer active operations. These methods can be applied in either a single way for moderate compression or a joint way for aggressive compression. We define several quantitative metrics to evaluate the compression performance for SNNs. Our methodology is validated in pattern recognition tasks over MNIST, N-MNIST, CIFAR10, and CIFAR100 datasets, where extensive comparisons, analyses, and insights are provided. To the best of our knowledge, this is the first work that studies SNN compression in a comprehensive manner by exploiting all compressible components and achieves better results.

Adaptive Compression Algorithm Selection Using LSTM Network in Column-oriented Database

Reinforcement Learning Based Data Compression for Energy-Efficient Non-volatile Caches.

Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization

Time Series Compression Based on Adaptive Piecewise Recurrent Autoencoder

Adaptive Cache Management for Complex Storage Systems Using CNN-LSTM-Based Spatiotemporal Prediction

LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Time

ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model

On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression

Neural Network Compression Via Sparse Optimization

A Compressed Data Partition and Loop Scheduling Scheme for Neural Networks

Data Compression for Analytics over Large-scale In-memory Column Databases

Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework

LeCo: Lightweight Compression Via Learning Serial Correlations

Understanding The Effectiveness of Lossy Compression in Machine Learning Training Sets

Deep Lossless Compression Algorithm Based on Arithmetic Coding for Power Data

Enhancing Lossy Compression Through Cross-Field Information for Scientific Applications

Spatiotemporally adaptive compression for scientific dataset with feature preservation -- a case study on simulation data with extreme climate events analysis

Dynamic and Adaptive Threshold for DNN Compression from Scratch.

AlphaZip: Neural Network-Enhanced Lossless Text Compression

ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition

Time Series Compression based on Reinforcement Learning