Abstract:Recurrent neural nets (RNN) and convolutional neural nets (CNN) are widely used on NLP tasks to capture the long-term and local dependencies, respectively. Attention mechanisms have recently attracted enormous interest due to their highly parallelizable computation, significantly less training time, and flexibility in modeling dependencies. We propose a novel attention mechanism in which the attention between elements from input sequence(s) is directional and multi-dimensional (i.e., feature-wise). A light-weight neural net, "Directional Self-Attention Network (DiSAN)," is then proposed to learn sentence embedding, based solely on the proposed attention without any RNN/CNN structure. DiSAN is only composed of a directional self-attention with temporal order encoded, followed by a multi-dimensional attention that compresses the sequence into a vector representation. Despite its simple form, DiSAN outperforms complicated RNN models on both prediction quality and time efficiency. It achieves the best test accuracy among all sentence encoding methods and improves the most recent best result by 1.02% on the Stanford Natural Language Inference (SNLI) dataset, and shows state-of-the-art test accuracy on the Stanford Sentiment Treebank (SST), Multi-Genre natural language inference (MultiNLI), Sentences Involving Compositional Knowledge (SICK), Customer Review, MPQA, TREC question-type classification and Subjectivity (SUBJ) datasets.

Phrase-level Self-Attention Networks for Universal Sentence Encoding

Deep Attentive Sentence Ordering Network.

SAC: Accelerating and Structuring Self-Attention Via Sparse Adaptive Connection.

A Window-Based Self-Attention Approach for Sentence Encoding

Multiple Structural Priors Guided Self Attention Network for Language Understanding

Deps-SAN: Neural Machine Translation with Dependency-Scaled Self-Attention Network

Boosting Neural Machine Translation with Dependency-Scaled Self-Attention Network.

Reinforced Self-Attention Network: a Hybrid of Hard and Soft Attention for Sequence Modeling

SG-Net: Syntax-Guided Machine Reading Comprehension.

Multi-Granularity Self-Attention for Neural Machine Translation

Phrase-Level Global-Local Hybrid Model for Sentence Embedding

Context-aware Positional Representation for Self-Attention Networks.

Self-Attention with Cross-Lingual Position Representation

S2SAN: A sentence-to-sentence attention network for sentiment analysis of online reviews

SG-Net: Syntax Guided Transformer for Language Representation

Prior Knowledge Representation for Self-Attention Networks

Multiple Positional Self-Attention Network for Text Classification

DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding

Structured Self-Attention Weights Encode Semantics in Sentiment Analysis

Encoding Syntactic Knowledge in Neural Networks for Sentiment Classification.

Learning Universal Sentence Representations with Mean-Max Attention Autoencoder