Abstract:Neural network-based approaches have become the driven forces for Natural Language Processing (NLP) tasks. Conventionally, there are two mainstream neural architectures for NLP tasks: the recurrent neural network (RNN) and the convolution neural network (ConvNet). RNNs are good at modeling long-term dependencies over input texts, but preclude parallel computation. ConvNets do not have memory capability and it has to model sequential data as un-ordered features. Therefore, ConvNets fail to learn sequential dependencies over the input texts, but it is able to carry out high-efficient parallel computation. As each neural architecture, such as RNN and ConvNets, has its own pro and con, integration of different architectures is assumed to be able to enrich the semantic representation of texts, thus enhance the performance of NLP tasks. However, few investigation explores the reconciliation of these seemingly incompatible architectures. To address this issue, we propose a hybrid architecture based on a novel hierarchical multi-granularity attention mechanism, named Multi-granularity Attention-based Hybrid Neural Network (MahNN). The attention mechanism is to assign different weights to different parts of the input sequence to increase the computation efficiency and performance of neural models. In MahNN, two types of attentions are introduced: the syntactical attention and the semantical attention. The syntactical attention computes the importance of the syntactic elements (such as words or sentence) at the lower symbolic level and the semantical attention is used to compute the importance of the embedded space dimension corresponding to the upper latent semantics. We adopt the text classification as an exemplifying way to illustrate the ability of MahNN to understand texts. The experimental results on a variety of datasets demonstrate that MahNN outperforms most of the state-of-the-arts for text classification.

Hierarchical Task-aware Multi-Head Attention Network

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

HirMTL: Hierarchical Multi-Task Learning for dense scene understanding

Hierarchical Inter-Attention Network for Document Classification with Multi-Task Learning.

Hierarchical Multi-Granularity Attention- Based Hybrid Neural Network for Text Classification.

HFedMTL: Hierarchical Federated Multi-Task Learning

Deep Multi-Task Augmented Feature Learning via Hierarchical Graph Neural Network

Adaptive and Dynamic Knowledge Transfer in Multi-task Learning with Attention Networks.

Multi-task Hierarchical Cross-Attention Network for Multi-label Text Classification

Multiple Relational Attention Network for Multi-task Learning

Hierarchical Deep Multi-task Learning with Attention Mechanism for Similarity Learning

Deep Multi-Task Learning with Shared Memory

Multi-Task Networks With Universe, Group, and Task Feature Learning

An Adaptive Node Network Based on Multitask Deep Learning

Deep collaborative multi-task network: A human decision process inspired model for hierarchical image classification

Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictions

Adaptive Task-Wise Message Passing for Multi-Task Learning: A Spatial Interaction Perspective

A Hybrid Multi-Task Learning Network for Hyperspectral Image Classification with Few Labels

Adaptively Sharing Multi-Levels of Distributed Representations in Multi-Task Learning

HD-MTL: Hierarchical Deep Multi-Task Learning for Large-Scale Visual Recognition

A Hierarchical LSTM Model for Joint Tasks.