Connecting Speech Encoder and Large Language Model for ASR

Wenyi Yu,Changli Tang,Guangzhi Sun,Xianzhao Chen,Tian Tan,Wei Li,Lu Lu,Zejun Ma,Chao Zhang

2023-09-26

Abstract:The impressive capability and versatility of large language models (LLMs) have aroused increasing attention in automatic speech recognition (ASR), with several pioneering studies attempting to build integrated ASR models by connecting a speech encoder with an LLM. This paper presents a comparative study of three commonly used structures as connectors, including fully connected layers, multi-head cross-attention, and Q-Former. Speech encoders from the Whisper model series as well as LLMs from the Vicuna model series with different model sizes were studied. Experiments were performed on the commonly used LibriSpeech, Common Voice, and GigaSpeech datasets, where the LLMs with Q-Formers demonstrated consistent and considerable word error rate (WER) reductions over LLMs with other connector structures. Q-Former-based LLMs can generalise well to out-of-domain datasets, where 12% relative WER reductions over the Whisper baseline ASR model were achieved on the Eval2000 test set without using any in-domain training data from Switchboard. Moreover, a novel segment-level Q-Former is proposed to enable LLMs to recognise speech segments with a duration exceeding the limitation of the encoders, which results in 17% relative WER reductions over other connector structures on 90-second-long speech data.

Audio and Speech Processing,Computation and Language,Sound

What problem does this paper attempt to address?

The paper primarily explores how to achieve automatic speech recognition (ASR) by connecting a speech encoder with a large language model (LLM). Specifically, the research focuses on comparing three different connection structures: fully connected layers, multi-head cross-attention mechanisms, and Q-Former, and analyzes the performance of these structures when integrating the speech encoder with the LLM.

The key contributions of the paper include:
1. **Comparison of connection structures**: Through experiments, the effectiveness of fully connected layers, multi-head cross-attention, and Q-Former as bridges between the speech encoder and the LLM was compared. The results show that the model using Q-Former performs best in terms of Word Error Rate (WER).
2. **Advantages of Q-Former**: The study found that Q-Former not only effectively compresses information, reducing the amount of data input to the LLM, but also retains sufficient information to achieve high-quality speech recognition.
3. **Performance validation on large-scale datasets**: The model was trained on a large-scale dataset containing approximately 4000 hours of audio, and the consistently superior performance of the LLM connected with Q-Former was validated on multiple test sets, including good generalization to out-of-domain data.
4. **Handling long-form speech**: A novel paragraph-level Q-Former structure was proposed to handle long speech segments that exceed the input limits of the pre-trained speech encoder. This structure can process speech inputs up to 90 seconds or even longer without sacrificing recognition accuracy.

In summary, this research aims to demonstrate how to effectively utilize LLMs for speech recognition and proposes several effective techniques to improve recognition accuracy, especially in handling long speech segments.