Abstract:Attention mechanism has been a key component in Recurrent Neural Networks (RNNs) based sequence to sequence learning framework, which has been adopted in many text understanding tasks, such as neural machine translation and abstractive summarization. In these tasks, the attention mechanism models how important each part of the source sentence is to generate a target side word. To compute such importance scores, the attention mechanism summarizes the source side information in the encoder RNN hidden states (i.e., h_t), and then builds a context vector for a target side word upon a subsequence representation of the source sentence, since h_t actually summarizes the information of the subsequence containing the first t-th words in the source sentence. We in this paper, show that an additional attention mechanism called word attention, that builds itself upon word level representations, significantly enhances the performance of sequence to sequence learning. Our word attention can enrich the source side contextual representation by directly promoting the clean word level information in each step. Furthermore, we propose to use contextual gates to dynamically combine the subsequence level and word level contextual information. Experimental results on abstractive summarization and neural machine translation show that word attention significantly improve over strong baselines.

RNN-Based Sequence-Preserved Attention for Dependency Parsing.

Stack-based Multi-layer Attention for Transition-based Dependency Parsing.

Improving Feature-Rich Transition-Based Constituent Parsing Using Recurrent Neural Networks.

Self-attentive Biaffine Dependency Parsing.

Transition-based Dependency Parsing Using Two Heterogeneous Gated Recursive Neural Networks.

A Neural Transition-Based Approach for Semantic Dependency Graph Parsing

SLDP: Sequence Learning Dependency Parsing Model Using Long Short-Term Memory

Scene Parsing via Dense Recurrent Neural Networks with Attentional Selection

A Sequence-to-Action Architecture for Character-Based Chinese Dependency Parsing with Status History.

Improved Character-Based Chinese Dependency Parsing by Using Stack-Tree LSTM.

High-order Graph-based Neural Dependency Parsing.

Word Attention for Sequence to Sequence Text Understanding.

A Neural Probabilistic Structured-Prediction Model for Transition-Based Dependency Parsing.

Attention-Based Recurrent Neural Network for Sequence Labeling.

Deterministic Attention For Sequence-To-Sequence Constituent Parsing

Seq2seq Model with RNN Attention for Abstractive Summarization

Word attention for sequence to sequence text understanding

Improving AMR Parsing by Exploiting the Dependency Parsing As an Auxiliary Task.

Lexicalized Neural Unsupervised Dependency Parsing

Incremental Graph-based Neural Dependency Parsing.

Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling