Abstract: Language modeling is a crucial component in a wide range of applications including speech recognition. Language models LMs are usually constructed by splitting a sentence into words and computing the probability of a word based on its word history. This sentence probability calculation, making use of conditional probability distributions, assumes that there is little impact from approximations used in the LMs, including the word history representations and finite training data. This motivates examining models that make use of additional information from the sentence. In this paper, future word information, in addition to the history, is used to predict the probability of the current word. For recurrent neural network LMs RNNLMs, this information can be encapsulated in a bi-directional model. However, if used directly, this form of model is computationally expensive when trained on large quantities of data, and can be problematic when used with word lattices. This paper proposes a novel neural network language model structure, the succeeding-word RNNLM, su-RNNLM, to address these issues. Instead of using a recurrent unit to capture the complete future word contexts, a feedforward unit is used to model a fixed finite number of succeeding words. This is more efficient in training than bi-directional models and can be applied to lattice rescoring. The generated lattices can be used for downstream applications, such as confusion network decoding and keyword search. Experimental results on speech recognition and keyword spotting tasks illustrate the empirical usefulness of future word information, and the flexibility of the proposed model to represent this information.

Learning Recurrent Neural Network Language Models with Context-Sensitive Label Smoothing for Automatic Speech Recognition

Data Noising as Smoothing in Neural Network Language Models

Learning better structured representations using low-rank adaptive label smoothing

Context-Aware Selective Label Smoothing for Calibrating Sequence Recognition Model

Learning label smoothing for text classification

Variance regularization of RNNLM for speech recognition

Discriminative method for recurrent neural network language models

An Empirical Study of Non-Stationary Ngram Model and its Smoothing Techniques

Revisiting the Role of Label Smoothing in Enhanced Text Sentiment Classification

MRSLN: A Multimodal Residual Speaker-LSTM Network to alleviate the over-smoothing issue for Emotion Recognition in Conversation

APAM: Adaptive Pre-training and Adaptive Meta Learning in Language Model for Noisy Labels and Long-tailed Learning

Global context-dependent recurrent neural network language model with sparse feature learning

Multiple-target Deep Learning for LSTM-RNN Based Speech Enhancement

Learning with Noisy Labels Via Sparse Regularization

Improving Time Series Classification with Representation Soft Label Smoothing

Enhancing Reinforcement Learning with Label-Sensitive Reward for Natural Language Understanding

Focus on the Target's Vocabulary: Masked Label Smoothing for Machine Translation.

Exploiting Future Word Contexts in Neural Network Language Models for Speech Recognition.

Revisiting Over-Smoothness in Text to Speech

The Role of $n$-gram Smoothing in the Age of Neural Networks

Learning Utterance-level Representations with Label Smoothing for Speech Emotion Recognition