Abstract:Neural network-based approaches have become the driven forces for Natural Language Processing (NLP) tasks. Conventionally, there are two mainstream neural architectures for NLP tasks: the recurrent neural network (RNN) and the convolution neural network (ConvNet). RNNs are good at modeling long-term dependencies over input texts, but preclude parallel computation. ConvNets do not have memory capability and it has to model sequential data as un-ordered features. Therefore, ConvNets fail to learn sequential dependencies over the input texts, but it is able to carry out high-efficient parallel computation. As each neural architecture, such as RNN and ConvNets, has its own pro and con, integration of different architectures is assumed to be able to enrich the semantic representation of texts, thus enhance the performance of NLP tasks. However, few investigation explores the reconciliation of these seemingly incompatible architectures. To address this issue, we propose a hybrid architecture based on a novel hierarchical multi-granularity attention mechanism, named Multi-granularity Attention-based Hybrid Neural Network (MahNN). The attention mechanism is to assign different weights to different parts of the input sequence to increase the computation efficiency and performance of neural models. In MahNN, two types of attentions are introduced: the syntactical attention and the semantical attention. The syntactical attention computes the importance of the syntactic elements (such as words or sentence) at the lower symbolic level and the semantical attention is used to compute the importance of the embedded space dimension corresponding to the upper latent semantics. We adopt the text classification as an exemplifying way to illustrate the ability of MahNN to understand texts. The experimental results on a variety of datasets demonstrate that MahNN outperforms most of the state-of-the-arts for text classification.

A Multi-feature Fusion Method with Attention Mechanism for Long Text Classification

Database Systems for Advanced Applications

Chinese text classification based on attention mechanism and feature-enhanced fusion neural network

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

An effective multi-modal adaptive contextual feature information fusion method for Chinese long text classification

A multi granularity information fusion text classification model based on attention mechanism

Feature Fusion Text Classification Model Combining CNN and BiGRU with Multi-Attention Mechanism

Attention-based BiLSTM fused CNN with gating mechanism model for Chinese long text classification

A multi-semantic passing framework for semi-supervised long text classification

AMFF: A new attention-based multi-feature fusion method for intention recognition

A text classification network model combining machine learning and deep learning

An Attention-Based Multi-Representational Fusion Method for Social-Media-Based Text Classification

Hierarchical Multi-Granularity Attention- Based Hybrid Neural Network for Text Classification.

Multichannel CNN with Attention for Text Classification

Feature-enhanced text-inception model for Chinese long text classification

Densely Connected CNN with Multi-scale Feature Attention for Text Classification

Convolutional Long Short-term Memory for Long Length Document Classification

A Long-Text Classification Method of Chinese News Based on BERT and CNN

A Multi-Branch Feature Fusion Strategy Based on an Attention Mechanism for Remote Sensing Image Scene Classification