Abstract:While post-training model compression can greatly reduce the inference cost of a deep neural network, uncompressed training still consumes a huge amount of hardware resources, run-time and energy. It is highly desirable to directly train a compact neural network from scratch with low memory and low computational cost. Low-rank tensor decomposition is one of the most effective approaches to reduce the memory and computing requirements of large-size neural networks. However, directly training a low-rank tensorized neural network is a very challenging task because it is hard to determine a proper tensor rank {\it a priori}, which controls the model complexity and compression ratio in the training process. This paper presents a novel end-to-end framework for low-rank tensorized training of neural networks. We first develop a flexible Bayesian model that can handle various low-rank tensor formats (e.g., CP, Tucker, tensor train and tensor-train matrix) that compress neural network parameters in training. This model can automatically determine the tensor ranks inside a nonlinear forward model, which is beyond the capability of existing Bayesian tensor methods. We further develop a scalable stochastic variational inference solver to estimate the posterior density of large-scale problems in training. Our work provides the first general-purpose rank-adaptive framework for end-to-end tensorized training. Our numerical results on various neural network architectures show orders-of-magnitude parameter reduction and little accuracy loss (or even better accuracy) in the training process. Specifically, on a very large deep learning recommendation system with over $4.2\times 10^9$ model parameters, our method can reduce the variables to only $1.6\times 10^5$ automatically in the training process (i.e., by $2.6\times 10^4$ times) while achieving almost the same accuracy.

Bayesian tensorized neural networks with automatic rank selection

Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination

Unified Framework for Neural Network Compression via Decomposition and Optimal Rank Selection

General-Purpose Bayesian Tensor Learning With Automatic Rank Determination and Uncertainty Quantification

Low-Rank+Sparse Tensor Compression for Neural Networks

DNN Compression Approach Based on Bayesian Optimization Tensor Ring Decomposition

Compression of Recurrent Neural Networks using Matrix Factorization

Convolutional Neural Network Compression Based on Low-Rank Decomposition

Bayesian CP Factorization of Incomplete Tensors with Automatic Rank Determination

Efficient Model Compression for Bayesian Neural Networks

Bayesian tensor-on-tensor regression with efficient computation

Convolutional neural networks compression with low rank and sparse tensor decompositions

Exploiting Elasticity in Tensor Ranks for Compressing Neural Networks

Bayesian tensor network structure search and its application to tensor completion

Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Network

Automatic Rank Selection for High-Speed Convolutional Neural Network

Bayesian Estimation and Tuning-Free Rank Detection for Probability Mass Function Tensors

Beyond Low Rank: A Data-Adaptive Tensor Completion Method

MARS: Masked Automatic Ranks Selection in Tensor Decompositions

T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor

Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization