Abstract:In recent years, the evolution of end-to-end (E2E) automatic speech recognition (ASR) models has been remarkable, largely due to advances in deep learning architectures like transformer. On top of E2E systems, researchers have achieved substantial accuracy improvement by rescoring E2E model's N-best hypotheses with a phoneme-based model. This raises an interesting question about where the improvements come from other than the system combination effect. We examine the underlying mechanisms driving these gains and propose an efficient joint training approach, where E2E models are trained jointly with diverse modeling units. This methodology does not only align the strengths of both phoneme and grapheme-based models but also reveals that using these diverse modeling units in a synergistic way can significantly enhance model accuracy. Our findings offer new insights into the optimal integration of heterogeneous modeling units in the development of more robust and accurate ASR systems.

What problem does this paper attempt to address?

The paper primarily explores how to enhance Connectionist Temporal Classification (CTC)-based Automatic Speech Recognition (ASR) systems by utilizing diverse modeling units. Specifically, the research team has made the following key contributions: 1. **Proposed a new joint training strategy**: This strategy leverages the intermediate representations naturally learned by the network, i.e., adding an additional CTC output layer on top of the baseline wordpiece model and training it for other types of modeling units (such as phonemes, characters, etc.). This method not only combines the advantages of wordpieces and phonemes but also improves the accuracy of the model. 2. **Explored optimal layer matching between different modeling units**: Experiments revealed that different types of modeling units need to be matched with specific layers in the model to optimize results. For example, phoneme-related modeling units are better suited to match with the middle layers of the model, while character modeling units may be more appropriate for deeper layers. 3. **Extended acoustic model fusion work**: The research team improved the Word Error Rate (WER) based on alternative modeling units (such as characters), demonstrating that performance improvements can be achieved by providing smaller unit representations even without using independent acoustic models. The main objective of the research is to verify whether different modeling units can significantly improve the accuracy of ASR systems without introducing additional models. Through a series of experiments (including English and Mandarin Chinese), the authors demonstrated the effectiveness of this joint training method and proved that it can reduce the need for multiple recognition passes, thereby improving the system's efficiency and response speed.

Enhancing CTC-based speech recognition with diverse modeling units

Multi-Level Modeling Units for End-to-End Mandarin Speech Recognition

End-to-end Code-switched TTS with Mix of Monolingual Recordings.

Exploring Model Units and Training Strategies for End-to-End Speech Recognition

Recent Advances in End-to-End Automatic Speech Recognition

Advancing Acoustic-to-Word CTC Model

An improved hybrid CTC-Attention model for speech recognition

Improving Non-Autoregressive End-to-End Speech Recognition with Pre-Trained Acoustic and Language Models

Cascaded CNN-resBiLSTM-CTC: an End-to-End Acoustic Model for Speech Recognition.

4D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoders

Syllable-Based Acoustic Modeling with CTC for Multi-Scenarios Mandarin Speech Recognition

Acoustic Modeling With Dfsmn-Ctc And Joint Ctc-Ce Learning

Acoustic Model Fusion for End-to-end Speech Recognition

Improving End-to-End Single-Channel Multi-Talker Speech Recognition.

Improved BLSTM RNN Based Accent Speech Recognition Using Multi-task Learning and Accent Embeddings

Multilingual Training and Cross-lingual Adaptation on CTC-based Acoustic Model

A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese

A hybrid CTC+Attention model based on end-to-end framework for multilingual speech recognition

Integrating Pre-Trained Speech and Language Models for End-to-End Speech Recognition

Tri-stage training with language-specific encoder and bilingual acoustic learner for code-switching speech recognition

E2E-based Multi-task Learning Approach to Joint Speech and Accent Recognition