Abstract:End-to-end attention-based modeling is increasingly popular for tackling sequence-to-sequence mapping tasks. Traditional attention mechanisms utilize prior input information to derive attention, which then conditions the output. However, we believe that knowledge of posterior output information may convey some advantage when modeling attention. A recent technique proposed for machine translation called the posterior attention model (PAM) demonstrates that posterior output information can be used in that way for machine translation. This paper explores the use of posterior information for attention modeling in an automatic speech recognition (ASR) task. We demonstrate that direct application of PAM to ASR is unsatisfactory, due to two deficiencies; Firstly, PAM adopts attention based weighted single-frame output prediction by assuming a single focused attention variable, whereas wider contextual information from acoustic frames is important for output prediction in ASR. Secondly, in addition to the well-known exposure bias problem, PAM introduces additional mismatches in attention training and inference calculations. We present extensive experiments combining a number of alternative approaches to solving these problems, leading to a high performance technique which we call extended PAM (EPAM). To counter the first deficiency, EPAM modifies the encoder to introduce additional context information for output prediction. The second deficiency is overcome in EPAM through a two part solution of a mismatch penalty term and an alternate learning strategy. The former applies a divergence-based loss to correct the mismatch bias distribution, while the latter employs a novel update strategy which relies on introducing iterative inference steps alongside each training step. In experiments with both WSJ-80hrs and Switchboard-300hrs datasets we found significant performance gains. For example, the full EPAM system model achieved a word error rate (WER) of 10.6% on the WSJ eval92 test set, compared to 11.6% for traditional prior-attention modeling. Meanwhile, on the Switchboard eval2000 test set, we achieved 16.3% WER compared to the traditional method WER of 17.3%.

Future Context Attention for Unidirectional LSTM Based Acoustic Model

Gated Recurrent Unit Based Acoustic Modeling with Future Context

Multi-Modal Attention Mechanisms In Lstm And Its Application To Acoustic Scene Classification

An Improved Residual LSTM Architecture for Acoustic Modeling

Compact Feedforward Sequential Memory Networks for Large Vocabulary Continuous Speech Recognition

Bidirectional LSTM with Extended Input Context

Towards Efficient Recurrent Architectures: A Deep LSTM Neural Network Applied to Speech Enhancement and Recognition

Acoustic Modeling With Dfsmn-Ctc And Joint Ctc-Ce Learning

High-Accuracy and Low-Latency Speech Recognition with Two-Head Contextual Layer Trajectory LSTM Model

Improving Gated Recurrent Unit Based Acoustic Modeling with Batch Normalization and Enlarged Context.

Future word contexts in neural network language models

CTC Regularized Model Adaptation for Improving LSTM RNN Based Multi-Accent Mandarin Speech Recognition

Speaker-aware Training of LSTM-RNNS for Acoustic Modelling

Deep causal speech enhancement and recognition using efficient long-short term memory Recurrent Neural Network

Exploiting Future Word Contexts in Neural Network Language Models for Speech Recognition.

Effective Exploitation of Posterior Information for Attention-Based Speech Recognition.

Maxout Neurons Based Deep Bidirectional Lstm For Acoustic Modeling

Advancing Acoustic-to-Word CTC Model

Learning Contextual Representation with Convolution Bank and Multi-head Self-attention for Speech Emphasis Detection.

Bi-LSTM-attention Based on ACNN Model for Disfluency Detection

A Context-focused Attention Evolution Model for Aspect-based Sentiment Classification