Learning Sentence Representations over Tree Structures for Target-Dependent Classification.

Junwen Duan,Xiao Ding,Ting Liu

DOI: https://doi.org/10.18653/v1/n18-1051

2018-01-01

Abstract:Junwen Duan, Xiao Ding, Ting Liu. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

What problem does this paper attempt to address?

Dependency-based Siamese Long Short-Term Memory Network for Learning Sentence Representations.

Wenhao Zhu,Tengjun Yao,Jianyue Ni,Baogang Wei,Zhiguo Lu

DOI: https://doi.org/10.1371/journal.pone.0193919

IF: 3.9

2019-01-01

IEEE Access

Abstract:Textual representations play an important role in the field of natural language processing (NLP). The efficiency of NLP tasks, such as text comprehension and information extraction, can be significantly improved with proper textual representations. As neural networks are gradually applied to learn the representation of words and phrases, fairly efficient models of learning short text representations have been developed, such as the continuous bag of words (CBOW) and skip-gram models, and they have been extensively employed in a variety of NLP tasks. Because of the complex structure generated by the longer text lengths, such as sentences, algorithms appropriate for learning short textual representations are not applicable for learning long textual representations. One method of learning long textual representations is the Long Short-Term Memory (LSTM) network, which is suitable for processing sequences. However, the standard LSTM does not adequately address the primary sentence structure (subject, predicate and object), which is an important factor for producing appropriate sentence representations. To resolve this issue, this paper proposes the dependency-based LSTM model (D-LSTM). The D-LSTM divides a sentence representation into two parts: a basic component and a supporting component. The D-LSTM uses a pre-trained dependency parser to obtain the primary sentence information and generate supporting components, and it also uses a standard LSTM model to generate the basic sentence components. A weight factor that can adjust the ratio of the basic and supporting components in a sentence is introduced to generate the sentence representation. Compared with the representation learned by the standard LSTM, the sentence representation learned by the D-LSTM contains a greater amount of useful information. The experimental results show that the D-LSTM is superior to the standard LSTM for sentences involving compositional knowledge (SICK) data.
Learning Target-Dependent Sentence Representations For Chinese Event Detection

Wenbo Zhang,Xiao Ding,Ting Liu

DOI: https://doi.org/10.1007/978-3-030-01012-6_20

2018-01-01

Abstract:Chinese event detection is a particularly challenging task in information extraction. Previous work mainly consider the sequential representation of sentences. However, long-range dependencies between words in the sentences may hurt the performance of these approaches. We believe that syntactic representations can provide an effective mechanism to directly link words to their informative context in the sentences. In this paper, we propose a novel event detection model based on dependency trees. In particular, we propose transforming dependency trees to target-dependent trees where leaf nodes are words and internal nodes are dependency relations, to distinguish the target words. Experimental results on the ACE 2005 corpus show that our approach significantly outperforms state-of-the-art baseline methods.
TreeNet: Learning Sentence Representations with Unconstrained Tree Structure.

Zhou Cheng,Chun Yuan,Jiancheng Li,Haiqin Yang

DOI: https://doi.org/10.24963/ijcai.2018/557

2018-01-01

Abstract:Recursive neural network (RvNN) has been proved to be an effective and promising tool to learn sentence representations by explicitly exploiting the sentence structure. However, most existing work can only exploit simple tree structure, e.g., binary trees, or ignore the order of nodes, which yields suboptimal performance. In this paper, we proposed a novel neural network, namely TreeNet, to capture sentences structurally over the raw unconstrained constituency trees, where the number of child nodes can be arbitrary. In TreeNet, each node is learning from its left sibling and right child in a bottom-up left-to-right order, thus enabling the net to learn over any tree. Furthermore, multiple soft gates and a memory cell are employed in implementing the TreeNet to determine to what extent it should learn, remember and output, which proves to be a simple and efficient mechanism for semantic synthesis. Moreover, TreeNet significantly suppresses convolutional neural networks (CNN) and Long Short-Term Memory (LSTM) with fewer parameters. It improves the classification accuracy by 2%-5% with 42% of the best CNN’s parameters or 94% of standard LSTM’s. Extensive experiments demonstrate TreeNet achieves the state-of-the-art performance on all four typical text classification tasks.
When Are Tree Structures Necessary for Deep Learning of Representations?

Jiwei Li,Minh-Thang Luong,Dan Jurafsky,Eudard Hovy

DOI: https://doi.org/10.18653/v1/d15-1278

2015-01-01

Abstract:Recursive neural models, which use syntactic parse trees to recursively generate representations bottom-up, are a popular architecture. However there have not been rigorous evaluations showing for exactly which tasks this syntax-based method is appropriate. In this paper, we benchmark recursive neural models against sequential recurrent neural models, enforcing applesto-apples comparison as much as possible. We investigate 4 tasks: (1) sentiment classification at the sentence level and phrase level; (2) matching questions to answerphrases; (3) discourse parsing; (4) semantic relation extraction. Our goal is to understand better when, and why, recursive models can outperform simpler models. We find that recursive models help mainly on tasks (like semantic relation extraction) that require longdistance connection modeling, particularly on very long sequences. We then introduce a method for allowing recurrent models to achieve similar performance: breaking long sentences into clause-like units at punctuation and processing them separately before combining. Our results thus help understand the limitations of both classes of models, and suggest directions for improving recurrent models.
Learning Structured Representation for Text Classification via Reinforcement Learning.

Tianyang Zhang,Minlie Huang,Li Zhao

DOI: https://doi.org/10.1609/aaai.v32i1.12047

2018-01-01

Abstract:Representation learning is a fundamental problem in natural language processing. This paper studies how to learn a structured representation for text classification. Unlike most existing representation models that either use no structure or rely on pre-specified structures, we propose a reinforcement learning (RL) method to learn sentence representation by discovering optimized structures automatically. We demonstrate two attempts to build structured representation: Information Distilled LSTM (ID-LSTM) and Hierarchically Structured LSTM (HS-LSTM). ID-LSTM selects only important, task-relevant words, and HS-LSTM discovers phrase structures in a sentence. Structure discovery in the two representation models is formulated as a sequential decision problem: current decision of structure discovery affects following decisions, which can be addressed by policy gradient RL. Results show that our method can learn task-friendly representations by identifying important words or task-relevant structures without explicit structure annotations, and thus yields competitive performance.
Learning Domain Representation for Multi-Domain Sentiment Classification.

Qi Liu,Yue Zhang,Jiangming Liu

DOI: https://doi.org/10.18653/v1/n18-1050

2018-01-01

Abstract:Qi Liu, Yue Zhang, Jiangming Liu. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.
Learning to compose over tree structures via POS tags for sentence representation.

Gehui Shen,Zhi-Hong Deng,Ting Huang,Xi Chen

DOI: https://doi.org/10.1016/j.eswa.2019.112917

IF: 8.5

2020-01-01

Expert Systems with Applications

Abstract:•We propose a new approach for sentence representation.•Our approach can capture the complex semantic compositionality.•Extensive experiments show that our proposed approach is effective.
Discriminative Neural Sentence Modeling by Tree-Based Convolution

Lili Mou,Hao Peng,Ge Li,Yan Xu,Lu Zhang,Zhi Jin

DOI: https://doi.org/10.48550/arXiv.1504.01106

2015-06-02

Abstract:This paper proposes a tree-based convolutional neural network (TBCNN) for discriminative sentence modeling. Our models leverage either constituency trees or dependency trees of sentences. The tree-based convolution process extracts sentences' structural features, and these features are aggregated by max pooling. Such architecture allows short propagation paths between the output layer and underlying feature detectors, which enables effective structural feature learning and extraction. We evaluate our models on two tasks: sentiment analysis and question classification. In both experiments, TBCNN outperforms previous state-of-the-art results, including existing neural networks and dedicated feature/rule engineering. We also make efforts to visualize the tree-based convolution process, shedding light on how our models work.

Computation and Language,Machine Learning,Neural and Evolutionary Computing
Learning Program Representations with a Tree-Structured Transformer

Wenhan Wang,Kechi Zhang,Ge Li,Shangqing Liu,Anran Li,Zhi Jin,Yang Liu

DOI: https://doi.org/10.48550/arXiv.2208.08643

2022-08-18

Software Engineering

Abstract:Learning vector representations for programs is a critical step in applying deep learning techniques for program understanding tasks. Various neural network models are proposed to learn from tree-structured program representations, e.g., abstract syntax tree (AST) and concrete syntax tree (CST). However, most neural architectures either fail to capture long-range dependencies which are ubiquitous in programs, or cannot learn effective representations for syntax tree nodes, making them incapable of performing the node-level prediction tasks, e.g., bug localization. In this paper, we propose Tree-Transformer, a novel recursive tree-structured neural network to learn the vector representations for source codes. We propose a multi-head attention mechanism to model the dependency between siblings and parent-children node pairs. Moreover, we propose a bi-directional propagation strategy to allow node information passing in two directions, bottom-up and top-down along trees. In this way, Tree-Transformer can learn the information of the node features as well as the global contextual information. The extensive experimental results show that our Tree-Transformer significantly outperforms the existing tree-based and graph-based program representation learning approaches in both the tree-level and node-level prediction tasks.
Tree-based Convolution for Sentence Modeling.

Mingbo Ma,Liang Huang,Bowen Zhou,Bing Xiang

2015-01-01

Abstract:In sentence modeling and classification, convolutional neural network approaches have recently achieved state-of-the-art results, but all such efforts process word vectors sequentially and neglect long-distance dependencies. To exploit both deep learning and linguistic structures, we propose a tree-based convolutional neural network model which exploit various long-distance relationships between words. Our model improves the sequential baselines on all three sentiment and question classification tasks, and achieves the highest published accuracy on TREC.
Learning to Embed Sentences Using Attentive Recursive Trees.

Jiaxin Shi,Lei Hou,Juanzi Li,Zhiyuan Liu,Hanwang Zhang

DOI: https://doi.org/10.1609/aaai.v33i01.33016991

2019-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:Sentence embedding is an effective feature representation for most deep learning-based NLP tasks. One prevailing line of methods is using recursive latent tree-structured networks to embed sentences with task-specific structures. However, existing models have no explicit mechanism to emphasize taskinformative words in the tree structure. To this end, we propose an Attentive Recursive Tree model (AR-Tree), where the words are dynamically located according to their importance in the task. Specifically, we construct the latent tree for a sentence in a proposed important-first strategy, and place more attentive words nearer to the root; thus, AR-Tree can inherently emphasize important words during the bottomup composition of the sentence embedding. We propose an end-to-end reinforced training strategy for AR-Tree, which is demonstrated to consistently outperform, or be at least comparable to, the state-of-the-art sentence embedding methods on three sentence understanding tasks.
Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

Kai Sheng Tai,Richard Socher,Christopher D. Manning

DOI: https://doi.org/10.48550/arXiv.1503.00075

2015-02-28

Computation and Language

Abstract:Because of their superior ability to preserve sequence information over time, Long Short-Term Memory (LSTM) networks, a type of recurrent neural network with a more complex computational unit, have obtained strong results on a variety of sequence modeling tasks. The only underlying LSTM structure that has been explored so far is a linear chain. However, natural language exhibits syntactic properties that would naturally combine words to phrases. We introduce the Tree-LSTM, a generalization of LSTMs to tree-structured network topologies. Tree-LSTMs outperform all existing systems and strong LSTM baselines on two tasks: predicting the semantic relatedness of two sentences (SemEval 2014, Task 1) and sentiment classification (Stanford Sentiment Treebank).
Effective LSTMs for Target-Dependent Sentiment Classification

Duyu Tang,Bing Qin,Xiaocheng Feng,Ting Liu

DOI: https://doi.org/10.48550/arXiv.1512.01100

2015-12-03

Computation and Language

Abstract:Target-dependent sentiment classification remains a challenge: modeling the semantic relatedness of a target with its context words in a sentence. Different context words have different influences on determining the sentiment polarity of a sentence towards the target. Therefore, it is desirable to integrate the connections between target word and context words when building a learning system. In this paper, we develop two target dependent long short-term memory (LSTM) models, where target information is automatically taken into account. We evaluate our methods on a benchmark dataset from Twitter. Empirical results show that modeling sentence representation with standard LSTM does not perform well. Incorporating target information into LSTM can significantly boost the classification accuracy. The target-dependent LSTM models achieve state-of-the-art performances without using syntactic parser or external sentiment lexicons.
Tree-based convolution: A new architecture for sentence modeling

Lili Mou,Hao Peng,Ge Li,Yan Xu,Lu Zhang,Zhi Jin

2015-01-01

Abstract:This paper proposes a new convolutional neural architecture based on treestructures, called the tree-based convolutional neural network (TBCNN). Two variants take advantage of constituency trees and dependency trees, respectively, to model sentences. Compared with traditional “flat” convolutional neural networks (CNNs), TBCNNs explore explicitly the structural information of sentences; compared with recursive neural networks, TBCNNs have much shorter propagation paths, enabling more effective feature learning and extraction. In the experiment of sentiment analysis, our two models consistently outperform CNNs and RNNs in a controlled setting. Our models are also competitive to most stateof-the-art results, including RNNs based on long short term memory and deep CNNs/RNNs.
On the Importance of Word and Sentence Representation Learning in Implicit Discourse Relation Classification

Xin Liu,Jiefu Ou,Yangqiu Song,Xin Jiang

DOI: https://doi.org/10.24963/ijcai.2020/530

2020-07-01

Abstract:Implicit discourse relation classification is one of the most difficult parts in shallow discourse parsing as the relation prediction without explicit connectives requires the language understanding at both the text span level and the sentence level. Previous studies mainly focus on the interactions between two arguments. We argue that a powerful contextualized representation module, a bilateral multi-perspective matching module, and a global information fusion module are all important to implicit discourse analysis. We propose a novel model to combine these modules together. Extensive experiments show that our proposed model outperforms BERT and other state-of-the-art systems on the PDTB dataset by around 8% and CoNLL 2016 datasets around 16%. We also analyze the effectiveness of different modules in the implicit discourse relation classification task and demonstrate how different levels of representation learning can affect the results.
Learning Dynamic Representations for Discourse Dependency Parsing.

Tianyi Liu,Yansong Feng,Dongyan Zhao

2023-01-01

Abstract:Transition systems have been widely used for the discourse dependency parsing task. Existing works often characterize transition states by examining a certain number of elementary discourse units (EDUs), while neglecting the arcs obtained from the transition history. In this paper, we propose to employ GAT-based encoder to learn dynamic representations for sub-trees constructed in previous transition steps. By incorporating these representations, our model is able to retain accessibility to all parsed EDUs through the obtained arcs, thus better utilizing the structural information of the document, particularly when handling lengthy text spans with complex structures. For the discourse relation recognition task, we employ edge-featured GATs to derive better representations for EDU pairs. Experimental results show that our model can achieve state-of-the-art performance on widely adopted datasets including RST-DT, SciDTB and CDTB. Our code is available at https://github.com/lty-lty/Discourse-Dependency-Parsing .
Representations Using a Two-Tier Convolutional Neural Network

Tao Chen,Ruifeng Xu,Yulan He,Xuan Wang

2016-01-01

Abstract:We introduce a type of two-tier convolutional neural network model for learning distributed paragraph representations for a special task (e.g. topic classification or sentiment classification). We decompose the paragraph semantics into three cascaded constitutes: word representation, sentence composition and document composition. Specifically, we learn distributed word representations by a continuous bag-of-words model from a large unstructured text corpus. Then, using these word representations as pre-trained vectors, the first tier of our model learns the distributed task-specific sentence representations from a corpus where each sentence is annotated with a task-specific label. Subsequently, the second tier of our model learns the distributed paragraph representations of a different document corpus from the learned sentence representations. Our proposed model has been evaluated on topic classification based on the DBpedia ontology and sentiment classification of Amazon reviews. Empirical results show the effectiveness of our proposed learning model for generating distributed paragraph representations.
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

Sandeep Subramanian,Adam Trischler,Yoshua Bengio,Christopher J Pal

DOI: https://doi.org/10.48550/arXiv.1804.00079

2018-03-30

Computation and Language

Abstract:A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpose representations.
On Tree-Based Neural Sentence Modeling

Haoyue Shi,Hao Zhou,Jiaze Chen,Lei Li

DOI: https://doi.org/10.18653/v1/d18-1492

2018-01-01

Abstract:Neural networks with tree-based sentence encoders have shown better results on many downstream tasks. Most of existing tree-based encoders adopt syntactic parsing trees as the explicit structure prior. To study the effectiveness of different tree structures, we replace the parsing trees with trivial trees (i.e., binary balanced tree, left-branching tree and right-branching tree) in the encoders. Though trivial trees contain no syntactic information, those encoders get competitive or even better results on all of the ten downstream tasks we investigated. This surprising result indicates that explicit syntax guidance may not be the main contributor to the superior performances of tree-based neural sentence modeling. Further analysis show that tree modeling gives better results when crucial words are closer to the final representation. Additional experiments give more clues on how to design an effective tree-based encoder. Our code is open-source and available at https://github.com/ExplorerFreda/TreeEnc.
Dependency-based Convolutional Neural Networks for Sentence Embedding

Mingbo Ma,Liang Huang,Bing Xiang,Bowen Zhou

DOI: https://doi.org/10.48550/arXiv.1507.01839

2015-08-03

Abstract:In sentence modeling and classification, convolutional neural network approaches have recently achieved state-of-the-art results, but all such efforts process word vectors sequentially and neglect long-distance dependencies. To exploit both deep learning and linguistic structures, we propose a tree-based convolutional neural network model which exploit various long-distance relationships between words. Our model improves the sequential baselines on all three sentiment and question classification tasks, and achieves the highest published accuracy on TREC.

Computation and Language,Artificial Intelligence,Machine Learning

Learning Sentence Representations over Tree Structures for Target-Dependent Classification.

Dependency-based Siamese Long Short-Term Memory Network for Learning Sentence Representations.

Learning Target-Dependent Sentence Representations For Chinese Event Detection

TreeNet: Learning Sentence Representations with Unconstrained Tree Structure.

When Are Tree Structures Necessary for Deep Learning of Representations?

Learning Structured Representation for Text Classification via Reinforcement Learning.

Learning Domain Representation for Multi-Domain Sentiment Classification.

Learning to compose over tree structures via POS tags for sentence representation.

Discriminative Neural Sentence Modeling by Tree-Based Convolution

Learning Program Representations with a Tree-Structured Transformer

Tree-based Convolution for Sentence Modeling.

Learning to Embed Sentences Using Attentive Recursive Trees.

Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks

Effective LSTMs for Target-Dependent Sentiment Classification

Tree-based convolution: A new architecture for sentence modeling

On the Importance of Word and Sentence Representation Learning in Implicit Discourse Relation Classification

Learning Dynamic Representations for Discourse Dependency Parsing.

Representations Using a Two-Tier Convolutional Neural Network

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning

On Tree-Based Neural Sentence Modeling

Dependency-based Convolutional Neural Networks for Sentence Embedding