Abstract:While post-training model compression can greatly reduce the inference cost of a deep neural network, uncompressed training still consumes a huge amount of hardware resources, run-time and energy. It is highly desirable to directly train a compact neural network from scratch with low memory and low computational cost. Low-rank tensor decomposition is one of the most effective approaches to reduce the memory and computing requirements of large-size neural networks. However, directly training a low-rank tensorized neural network is a very challenging task because it is hard to determine a proper tensor rank {\it a priori}, which controls the model complexity and compression ratio in the training process. This paper presents a novel end-to-end framework for low-rank tensorized training of neural networks. We first develop a flexible Bayesian model that can handle various low-rank tensor formats (e.g., CP, Tucker, tensor train and tensor-train matrix) that compress neural network parameters in training. This model can automatically determine the tensor ranks inside a nonlinear forward model, which is beyond the capability of existing Bayesian tensor methods. We further develop a scalable stochastic variational inference solver to estimate the posterior density of large-scale problems in training. Our work provides the first general-purpose rank-adaptive framework for end-to-end tensorized training. Our numerical results on various neural network architectures show orders-of-magnitude parameter reduction and little accuracy loss (or even better accuracy) in the training process. Specifically, on a very large deep learning recommendation system with over $4.2\times 10^9$ model parameters, our method can reduce the variables to only $1.6\times 10^5$ automatically in the training process (i.e., by $2.6\times 10^4$ times) while achieving almost the same accuracy.

Geometry-aware training of factorized layers in tensor Tucker format

Tensorized Neural Layer Decomposition for 2-D DOA Estimation

Incremental multi-domain learning with network latent tensor factorization

A tensor factorization model of multilayer network interdependence

Tucker tensor factor models: matricization and mode-wise PCA estimation

Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination

A-Tucker: Fast Input-Adaptive and Matricization-Free Tucker Decomposition of Higher-Order Tensors on GPUs

Adaptive Regularizing Tucker Decomposition for Knowledge Graph Completion

a-Tucker: Input-Adaptive and Matricization-Free Tucker Decomposition for Dense Tensors on CPUs and GPUs

FactorizeNet: Progressive Depth Factorization for Efficient Network Architecture Exploration Under Quantization Constraints

Deep neural network compression by Tucker decomposition with nonlinear response

ADA-Tucker: Compressing Deep Neural Networks via Adaptive Dimension Adjustment Tucker Decomposition

Scalable Tucker Factorization for Sparse Tensors - Algorithms and Discoveries

Dintucker: Scaling Up Gaussian Process Models On Large Multidimensional Arrays

DinTucker: Scaling up Gaussian process models on multidimensional arrays with billions of elements

Efficient Finite Initialization for Tensorized Neural Networks

An Iterative Reweighted Method for Tucker Decomposition of Incomplete Multiway Tensors

cuFasterTucker: A Stochastic Optimization Strategy for Parallel Sparse FastTucker Decomposition on GPU Platform

Ultimate tensorization: compressing convolutional and FC layers alike

Semi-tensor Product-based TensorDecomposition for Neural Network Compression

Schrödinger's FP: Dynamic Adaptation of Floating-Point Containers for Deep Learning Training