USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models

Shaojin Ding,David Qiu,David Rim,Yanzhang He,Oleg Rybakov,Bo Li,Rohit Prabhavalkar,Weiran Wang,Tara N. Sainath,Zhonglin Han,Jian Li,Amir Yazdanbakhsh,Shivani Agrawal

2024-01-16

Abstract:End-to-end automatic speech recognition (ASR) models have seen revolutionary quality gains with the recent development of large-scale universal speech models (USM). However, deploying these massive USMs is extremely expensive due to the enormous memory usage and computational cost. Therefore, model compression is an important research topic to fit USM-based ASR under budget in real-world scenarios. In this study, we propose a USM fine-tuning approach for ASR, with a low-bit quantization and N:M structured sparsity aware paradigm on the model weights, reducing the model complexity from parameter precision and matrix topology perspectives. We conducted extensive experiments with a 2-billion parameter USM on a large-scale voice search dataset to evaluate our proposed method. A series of ablation studies validate the effectiveness of up to int4 quantization and 2:4 sparsity. However, a single compression technique fails to recover the performance well under extreme setups including int2 quantization and 1:4 sparsity. By contrast, our proposed method can compress the model to have 9.4% of the size, at the cost of only 7.3% relative word error rate (WER) regressions. We also provided in-depth analyses on the results and discussions on the limitations and potential solutions, which would be valuable for future studies.

Audio and Speech Processing,Sound

What problem does this paper attempt to address?

The problem this paper attempts to address is the high cost of deploying large-scale Universal Speech Models (USM) in practical applications. Specifically, these models have a large number of parameters, leading to high memory usage and computational costs, making them difficult to deploy efficiently in real-world applications. Therefore, the focus of the paper is to propose an effective model compression method to reduce the complexity and resource consumption of USM in Automatic Speech Recognition (ASR) tasks while maintaining high recognition performance.

To achieve this goal, the paper proposes a fine-tuning method that combines low-bit quantization and N:M structured sparsity, aiming to reduce the model's complexity from both parameter precision and matrix topology perspectives. Through this method, the researchers hope to significantly reduce the model size while keeping the performance loss within an acceptable range. Experimental results show that by combining 4-bit quantization and 2:4 sparsity, the model size can be compressed to 9.4% of the original model, with only a 7.3% increase in relative Word Error Rate (WER). This provides a new solution for the efficient deployment of large-scale USM in practical scenarios.