Compressing Recurrent Neural Network Models Through Principal Component Analysis
Haobo Qi,Jingxuan Cao,Shichong Chen,Jing Zhou
DOI: https://doi.org/10.4310/22-sii727
2023-01-01
Statistics and Its Interface
Abstract:Currently, deep learning-based neural network models, such as recurrent neural networks (RNNs) and long short-term memory (LSTM) architecture, are considered state-of-the-art solutions to most of the problems associated with the effective execution of tasks in the field of natural lan-guage processing (NLP). However, a large number of pa-rameters and significantly high memory complexity are re-quired to ensure the effective application of such models, thereby increasing the difficulty of deploying such models in embedded systems, such as those used in mobile devices and tablets. In this study, we propose a technique for com-pressing RNN-based models through principal component analysis. Our proposed compression approach begins with the embedding layer, after which it progresses to the final output layer. For each target layer, we propose a principal component analysis approach for reducing the dimensions in the two-dimensional (2D) estimated weight matrix. Through this approach, we develop a reduced model structure with fewer parameters than those of the benchmark model. Addi-tionally, our proposed approach ensures improved prediction accuracy compared to that of the benchmark model. More-over, we propose a novel parameter-initialization method based on the score matrix of the principal component. We evaluate the effectiveness of our proposed method by con-ducting experiments on various NLP-related tasks, such as text classification and language translation, and datasets. The results of our experiments are significantly encouraging, as they pertain to the compression of RNN models through principal component analysis.