Abstract:In automatic speech recognition (ASR) research, discriminative criteria have achieved superior performance in DNN-HMM systems. Given this success, the adoption of discriminative criteria is promising to boost the performance of end-to-end (E2E) ASR systems. With this motivation, previous works have introduced the minimum Bayesian risk (MBR, one of the discriminative criteria) into E2E ASR systems. However, the effectiveness and efficiency of the MBR-based methods are compromised: the MBR criterion is only used in system training, which creates a mismatch between training and decoding; the on-the-fly decoding process in MBR-based methods results in the need for pre-trained models and slow training speeds. To this end, novel algorithms are proposed in this work to integrate another widely used discriminative criterion, lattice-free maximum mutual information (LF-MMI), into E2E ASR systems not only in the training stage but also in the decoding process. The proposed LF-MMI training and decoding methods show their effectiveness on two widely used E2E frameworks: Attention-Based Encoder-Decoders (AEDs) and Neural Transducers (NTs). Compared with MBR-based methods, the proposed LF-MMI method: maintains the consistency between training and decoding; eschews the on-the-fly decoding process; trains from randomly initialized models with superior training efficiency. Experiments suggest that the LF-MMI method outperforms its MBR counterparts and consistently leads to statistically significant performance improvements on various frameworks and datasets from 30 hours to 14.3k hours. The proposed method achieves state-of-the-art (SOTA) results on Aishell-1 (CER 4.10%) and Aishell-2 (CER 5.02%) datasets. Code is released.

Improved lattice-based spoken document retrieval by directly learning from the evaluation measures

A New Syllable-Lattice Based Approach For Mandarin Spoken Document Retrieval

Chinese Spoken Document Retrieval Based On Syllable Neighbor Posterior Probability Matrix

Improved Speech Recognition Using Discriminative Integration of Multiple Local Classifiers in Lattice Rescoring

Improved spoken term detection using support vector machines based on lattice context consistency

Converted Lattice-Based Chinese Spontaneous Speech Retrieval Based on Mutual Information Confidence Measure

Improved Contextual Recognition In Automatic Speech Recognition Systems By Semantic Lattice Rescoring

Neural Lattice Search for Speech Recognition.

Improvement Comparison of Different Lattice-based Discriminative Training Methods in Chinese-monolingual and Chinese-English-bilingual Speech Recognition

Learning Spoken Language Representations with Neural Lattice Language Modeling

Contextualizing ASR Lattice Rescoring with Hybrid Pointer Network Language Model

A Study of Lattice-Based Spoken Term Detection for Chinese Spontaneous Speech

Lattice-based Indexing for Spontaneous Mandarin Speech

Integrating Lattice-Free MMI into End-to-End Speech Recognition

Effectiveness of Text, Acoustic, and Lattice-based representations in Spoken Language Understanding tasks

Improved Semantic Retrieval of Spoken Content by Document/Query Expansion with Random Walk Over Acoustic Similarity Graphs

Spoken Document Retrieval Based On Confusion Network With Syllable Fragments

Semantic Query Expansion and Context-Based Discriminative Term Modeling for Spoken Document Retrieval

Study on Chinese speech retrieval based on posterior probability

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

Different Evaluation Approaches of Confusion Network in Chinese Spoken Classification