Abstract:Despite the rapid progress of end-to-end (E2E) automatic speech recognition (ASR), it has been shown that incorporating external language models (LMs) into the decoding can further improve the recognition performance of E2E ASR systems. To align with the modeling units adopted in E2E ASR systems, subword-level (e.g., characters, BPE) LMs are usually used to cooperate with current E2E ASR systems. However, the use of subword-level LMs will ignore the word-level information, which may limit the strength of the external LMs in E2E ASR. Although several methods have been proposed to incorporate word-level external LMs in E2E ASR, these methods are mainly designed for languages with clear word boundaries such as English and cannot be directly applied to languages like Mandarin, in which each character sequence can have multiple corresponding word sequences. To this end, we propose a novel decoding algorithm where a word-level lattice is constructed on-the-fly to consider all possible word sequences for each partial hypothesis. Then, the LM score of the hypothesis is obtained by intersecting the generated lattice with an external word N-gram LM. The proposed method is examined on both Attention-based Encoder-Decoder (AED) and Neural Transducer (NT) frameworks. Experiments suggest that our method consistently outperforms subword-level LMs, including N-gram LM and neural network LM. We achieve state-of-the-art results on both Aishell-1 (CER 4.18%) and Aishell-2 (CER 5.06%) datasets and reduce CER by 14.8% relatively on a 21K-hour Mandarin dataset.

Word Alignment Modeling with Context Dependent Deep Neural Network.

mammothwill be a jiang shi yixiang juda gongcheng job ( a )

Neural Baselines for Word Alignment

Standalone Training of Context-Dependent Deep Neural Network Acoustic Models

Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment

Word Alignment by Fine-tuning Embeddings on Parallel Corpora

Discriminative Word Alignment by Linear Modeling

Conversational Speech Transcription Using Context-Dependent Deep Neural Networks

Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment

Inter-Weighted Alignment Network for Sentence Pair Modeling.

Multi-Domain Neural Machine Translation with Word-Level Domain Context Discrimination

Research of English-Chinese Alignment at Word Granularity on Parallel Corpora

Word Alignment in the Era of Deep Learning: A Tutorial

Alignment-consistent Recursive Neural Networks for Bilingual Phrase Embeddings.

Multilingual Approach to Joint Speech and Accent Recognition with DNN-HMM Framework

Mask-Align: Self-Supervised Neural Word Alignment

Context-dependent Deep Neural Networks for audio indexing of real-life data

Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model

Maam: A Morphology-Aware Alignment Model For Unsupervised Bilingual Lexicon Induction

Exploring Discriminative Word-Level Domain Contexts for Multi-Domain Neural Machine Translation

A comparative study on selecting acoustic modeling units in deep neural networks based large vocabulary Chinese speech recognition