Abstract:Despite the rapid progress of end-to-end (E2E) automatic speech recognition (ASR), it has been shown that incorporating external language models (LMs) into the decoding can further improve the recognition performance of E2E ASR systems. To align with the modeling units adopted in E2E ASR systems, subword-level (e.g., characters, BPE) LMs are usually used to cooperate with current E2E ASR systems. However, the use of subword-level LMs will ignore the word-level information, which may limit the strength of the external LMs in E2E ASR. Although several methods have been proposed to incorporate word-level external LMs in E2E ASR, these methods are mainly designed for languages with clear word boundaries such as English and cannot be directly applied to languages like Mandarin, in which each character sequence can have multiple corresponding word sequences. To this end, we propose a novel decoding algorithm where a word-level lattice is constructed on-the-fly to consider all possible word sequences for each partial hypothesis. Then, the LM score of the hypothesis is obtained by intersecting the generated lattice with an external word N-gram LM. The proposed method is examined on both Attention-based Encoder-Decoder (AED) and Neural Transducer (NT) frameworks. Experiments suggest that our method consistently outperforms subword-level LMs, including N-gram LM and neural network LM. We achieve state-of-the-art results on both Aishell-1 (CER 4.18%) and Aishell-2 (CER 5.06%) datasets and reduce CER by 14.8% relatively on a 21K-hour Mandarin dataset.

An effective and robust approach to Mandarin spoken language understanding in specific domain

A Joint and Domain-Adaptive Approach to Spoken Language Understanding

Incorporating Complete Syntactical Knowledge for Spoken Language Understanding

Grammar-Supervised End-to-End Speech Recognition with Part-of-Speech Tagging and Dependency Parsing

Universal Dependencies Parsing For Colloquial Singaporean English

From Genesis to Creole Language

Integrating Multi-Level Linguistic Knowledge with a Unified Framework for Mandarin Speech Recognition

Mandarin Language Understanding in Dialogue Context

Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model

End-to-End Cross-Lingual Spoken Language Understanding Model with Multilingual Pretraining.

SLFNet: Generating Semantic Logic Forms from Natural Language Using Semantic Probability Graphs

Towards ASR Robust Spoken Language Understanding Through In-Context Learning With Word Confusion Networks

Syntax Matters: Towards Spoken Language Understanding Via Syntax-Aware Attention

ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding

A Syntax-aware Multi-task Learning Framework for Chinese Semantic Role Labeling

Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning

Recurrent Neural Network Language Model with Part-of-speech for Mandarin Speech Recognition.

Toward a Unified Approach to Statistical Language Modeling for Chinese

Prosodic Modeling with Rich Syntactic Context in HMM-based Mandarin Speech Synthesis

Knowledge-based Linguistic Encoding for End-to-End Mandarin Text-to-Speech Synthesis

FC-MTLF: A Fine- and Coarse-grained Multi-Task Learning Framework for Cross-Lingual Spoken Language Understanding.