Abstract:Dependency syntax has been demonstrated highly useful for a number of natural language processing (NLP) tasks. Typical approaches of utilizing dependency syntax include TreeRNN and Tree-Linearization, both of which exploit explicit 1-best tree outputs from a welltrained parser as inputs. However, these approaches may suffer from error propagation due to the inevitable errors contained in the 1-best tree outputs. In this work, we propose a novel approach to integrate dependency syntax without using the discrete tree outputs. The key idea is to use the intermediate hidden representations of a welltrained encoder-decoder dependency parser, which are referred to as Dependency-based Syntax-Aware Word Representations (Dep-SAWRs). Then, we simply concatenate such DepSAWRs with the conventional context-insensitive word embeddings to compose input word representations, without requiring to modify the model architecture of the downstream tasks. We evaluate the proposed method on four kinds of typical NLP tasks, including sentence classification, sentence matching, sequence labeling and machine translation. Experimental results show that the proposed approach is highly promising. On the one hand, it can utilize dependency syntax effectively, bringing consistently better performance on the four tasks compared with baselines without using syntax. On the other hand, the proposed method can outperform the Tree-RNN and Tree-Linearization approaches in most settings, and meanwhile are highly efficient in syntax integration. In addition, the proposed method would be easily extendable to encoding other structural attributes of language. (C) 2020 Elsevier B.V. All rights reserved.

Syntax-aware Neural Machine Translation Directed by Syntactic Dependency Degree

Boosting Neural Machine Translation with Dependency-Scaled Self-Attention Network.

Deps-SAN: Neural Machine Translation with Dependency-Scaled Self-Attention Network

Syntax-aware Transformer Encoder for Neural Machine Translation

Syntax-Directed Attention for Neural Machine Translation

Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder

Neural Machine Translation with Attention Based on a New Syntactic Branch Distance

Syntax-Enhanced Neural Machine Translation with Syntax-Aware Word Representations

Dependency-Based Local Attention Approach to Neural Machine Translation

Dependency-based syntax-aware word representations

Syntax-Aware Complex-Valued Neural Machine Translation

A Dependency Treelet String Correspondence Model for Statistical Machine Translation.

Syntax-based Transformer for Neural Machine Translation

Improving Neural Machine Translation with Neural Syntactic Distance

Enhancing low-resource neural machine translation with syntax-graph guided self-attention

A neural machine translation method based on split graph convolutional self-attention encoding

Source Dependency-Aware Transformer with Supervised Self-Attention

Dependency Distance: A New Perspective on Syntactic Patterns in Natural Languages.

SG-Net: Syntax Guided Transformer for Language Representation

Modeling Source Syntax for Neural Machine Translation

Neural Machine Translation with Source Dependency Representation.