Abstract: Language modeling is a crucial component in a wide range of applications including speech recognition. Language models LMs are usually constructed by splitting a sentence into words and computing the probability of a word based on its word history. This sentence probability calculation, making use of conditional probability distributions, assumes that there is little impact from approximations used in the LMs, including the word history representations and finite training data. This motivates examining models that make use of additional information from the sentence. In this paper, future word information, in addition to the history, is used to predict the probability of the current word. For recurrent neural network LMs RNNLMs, this information can be encapsulated in a bi-directional model. However, if used directly, this form of model is computationally expensive when trained on large quantities of data, and can be problematic when used with word lattices. This paper proposes a novel neural network language model structure, the succeeding-word RNNLM, su-RNNLM, to address these issues. Instead of using a recurrent unit to capture the complete future word contexts, a feedforward unit is used to model a fixed finite number of succeeding words. This is more efficient in training than bi-directional models and can be applied to lattice rescoring. The generated lattices can be used for downstream applications, such as confusion network decoding and keyword search. Experimental results on speech recognition and keyword spotting tasks illustrate the empirical usefulness of future word information, and the flexibility of the proposed model to represent this information.

A Word Language Model Based Contextual Language Processing On Chinese Character Recognition

Influence of Language Models and Candidate Set Size on Contextual Post-processing for Chinese Script Recognition.

Integrating Language Model in Handwritten Chinese Text Recognition.

Post-Processing Approach for Printed Chinese Character Recognition

A Hybrid Post-Processing System For Offline Handwritten Chinese Character Recognition Based On A Statistical Language Model

A Post-processing Approach for Handwritten Chinese Address Recognition

Distant BI-Gram Model, Collocation, and Their Applications in Post-Processing for Chinese Character Recognition

A New Word Language Model Evaluation Metric For Character Based Languages

A Chinese OCR Spelling Check Approach Based on Statistical Language Models.

Joint n-gram Chinese language modeling with an application to Chinese word segmentation

Joint-Character-Poc N-Gram Language Modeling for Chinese Speech Recognition

Exploiting Future Word Contexts in Neural Network Language Models for Speech Recognition.

Retrieval-based Language Model Adaptation for Handwritten Chinese Text Recognition

Context driven chinese string segmentation and recognition

Topic Language Model Adaption for Recognition of Homologous Offline Handwritten Chinese Text Image

An Efficient Post-Processing Approach for Off-Line Handwritten Chinese Address Recognition

Spatio-temporal unified model for on-line handwritten Chinese character recognition

Improving Mandarin End-to-End Speech Recognition with Word N-gram Language Model

Domain-Aware Word Segmentation for Chinese Language: A Document-Level Context-Aware Model

A similarity-based smoothing algorithm for Chinese language modeling and its application in pinyin-to-character conversion

A new linguistic decoding method for online handwritten Chinese character recognition