Abstract:Neural network-based approaches have become the driven forces for Natural Language Processing (NLP) tasks. Conventionally, there are two mainstream neural architectures for NLP tasks: the recurrent neural network (RNN) and the convolution neural network (ConvNet). RNNs are good at modeling long-term dependencies over input texts, but preclude parallel computation. ConvNets do not have memory capability and it has to model sequential data as un-ordered features. Therefore, ConvNets fail to learn sequential dependencies over the input texts, but it is able to carry out high-efficient parallel computation. As each neural architecture, such as RNN and ConvNets, has its own pro and con, integration of different architectures is assumed to be able to enrich the semantic representation of texts, thus enhance the performance of NLP tasks. However, few investigation explores the reconciliation of these seemingly incompatible architectures. To address this issue, we propose a hybrid architecture based on a novel hierarchical multi-granularity attention mechanism, named Multi-granularity Attention-based Hybrid Neural Network (MahNN). The attention mechanism is to assign different weights to different parts of the input sequence to increase the computation efficiency and performance of neural models. In MahNN, two types of attentions are introduced: the syntactical attention and the semantical attention. The syntactical attention computes the importance of the syntactic elements (such as words or sentence) at the lower symbolic level and the semantical attention is used to compute the importance of the embedded space dimension corresponding to the upper latent semantics. We adopt the text classification as an exemplifying way to illustrate the ability of MahNN to understand texts. The experimental results on a variety of datasets demonstrate that MahNN outperforms most of the state-of-the-arts for text classification.

Hierarchical Neural Networks for Sequential Sentence Classification in Medical Scientific Abstracts

Neural Networks for Joint Sentence Classification in Medical Paper Abstracts

A Hierarchical Model with Recurrent Convolutional Neural Networks for Sequential Sentence Classification

LSTM-based Deep Neural Network With A Focus on Sentence Representation for Sequential Sentence Classification in Medical Scientific Abstracts

Online Neural Sequence Detection with Hierarchical Dirichlet Point Process

A Deep Neural Network Sentence Level Classification Method with Context Information

Hierarchical Multi-Granularity Attention- Based Hybrid Neural Network for Text Classification.

Sectioning of Biomedical Abstracts: A Sequence of Sequence Classification Task

A deep learning classifier for sentence classification in biomedical and computer science abstracts

Sliding Hierarchical Recurrent Neural Networks for Sequence Classification

Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks

Jointly Trained Sequential Labeling and Classification by Sparse Attention Neural Networks.

Deep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchies

Biomedical Literature Classification With A Cnns-Based Hybrid Learning Network

A Sequential Model for Classifying Temporal Relations between Intra-Sentence Events

A Novel Neural Network-Based Method for Medical Text Classification

Biomedical word sense disambiguation with bidirectional long short-term memory and attention-based neural networks

Matching Natural Language Sentences with Hierarchical Sentence Factorization

Hierarchical Neural Network with Serial Attention Mechanism for Review Sentiment Classifification

Hierarchical learning for DNN-based acoustic scene classification