Gated Multi-Task Network for Text Classification

Liqiang Xiao,Honglun Zhang,Wenqing Chen

DOI: https://doi.org/10.18653/v1/n18-2114

2018-01-01

Abstract:Liqiang Xiao, Honglun Zhang, Wenqing Chen. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 2018.

What problem does this paper attempt to address?

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

Xiaoyu Ji,Wanyang Hu,Yanyan Liang

DOI: https://doi.org/10.3390/math11244895

IF: 2.4

2023-01-01

Mathematics

Abstract:The MASSIVE dataset is a spoken-language comprehension resource package for slot filling, intent classification, and virtual assistant evaluation tasks. It contains multi-language utterances from human beings communicating with a virtual assistant. In this paper, we exploited the relationship between intent classification and slot filling to improve the exact match accuracy by proposing five models with hierarchical and bidirectional architectures. There are two variants for hierarchical architectures and three variants for bidirectional architectures. These are the hierarchical concatenation model, the hierarchical attention-based model, the bidirectional max-pooling model, the bidirectional LSTM model, and the bidirectional attention-based model. The results of our models showed a significant improvement in the averaged exact match accuracy. The hierarchical attention-based model improved the accuracy by 1.01 points for the full training dataset. As for the zero-shot setup, we observed that the exact match accuracy increased from 53.43 to 53.91. In this study, we observed that, for multi-task problems, utilizing the relevance between different tasks can help in improving the model's overall performance.
Database Systems for Advanced Applications

Zhongwei Yue,Jingwei Zhang,Huibing Zhang,Qing Yang

DOI: https://doi.org/10.1007/978-3-319-05813-9

2014-01-01

Abstract:Text classification is a very important problem in Nature Language Processing (NLP). The text classification based on shallow machine-learning models takes too much time and energy to extract features of data, but only obtains poor performance. Recently, deep learning methods are widely used in text classification and result in good performance. In this paper, we propose a Convolutional Neural Network (CNN) with multi-size convolution and multi-type pooling for text classification. In our method, we adopt CNNs to extract features of the texts and then select the important information of these features through multi-type pooling. Experiments show that the CNN with multi-convolution and multi-type pooling (CNN-MCMP) obtains better performance on text classification compared with both the shallow machine-learning models and other CNN architectures.
Learning What to Share: Leaky Multi-Task Network for Text Classification.

Liqiang Xiao,Honglun Zhang,Wenqing Chen,Yongkun Wang,Yaohui Jin

2018-01-01

Abstract:Neural network based multi-task learning has achieved great success on many NLP problems, which focuses on sharing knowledge among tasks by linking some layers to enhance the performance. However, most existing approaches suffer from the interference between tasks because they lack of selection mechanism for feature sharing. In this way, the feature spaces of tasks may be easily contaminated by helpless features borrowed from others, which will confuse the models for making correct prediction. In this paper, we propose a multi-task convolutional neural network with the Leaky Unit, which has memory and forgetting mechanism to filter the feature flows between tasks. Experiments on five different datasets for text classification validate the benefits of our approach.
A Generalized Recurrent Neural Architecture for Text Classification with Multi-Task Learning.

Honglun Zhang,Liqiang Xiao,Yongkun Wang,Yaohui Jin

DOI: https://doi.org/10.24963/ijcai.2017/473

2017-01-01

Abstract:Multi-task learning leverages potential correlations among related tasks to extract common features and yield performance gains. However, most previous works only consider simple or weak interactions, thereby failing to model complex correlations among three or more tasks. In this paper, we propose a multi-task learning architecture with four types of recurrent neural layers to fuse information across multiple related tasks. The architecture is structurally flexible and considers various interactions among tasks, which can be regarded as a generalized case of many previous works. Extensive experiments on five benchmark datasets for text classification show that our model can significantly improve performances of related tasks with additional information from others.
Hierarchical Multi-Granularity Attention- Based Hybrid Neural Network for Text Classification.

Zhenyu Liu,Chaohong Lu,Haiwei Huang,Shengfei Lyu,Zhenchao Tao

DOI: https://doi.org/10.1109/access.2020.3016727

IF: 3.9

2020-01-01

IEEE Access

Abstract:Neural network-based approaches have become the driven forces for Natural Language Processing (NLP) tasks. Conventionally, there are two mainstream neural architectures for NLP tasks: the recurrent neural network (RNN) and the convolution neural network (ConvNet). RNNs are good at modeling long-term dependencies over input texts, but preclude parallel computation. ConvNets do not have memory capability and it has to model sequential data as un-ordered features. Therefore, ConvNets fail to learn sequential dependencies over the input texts, but it is able to carry out high-efficient parallel computation. As each neural architecture, such as RNN and ConvNets, has its own pro and con, integration of different architectures is assumed to be able to enrich the semantic representation of texts, thus enhance the performance of NLP tasks. However, few investigation explores the reconciliation of these seemingly incompatible architectures. To address this issue, we propose a hybrid architecture based on a novel hierarchical multi-granularity attention mechanism, named Multi-granularity Attention-based Hybrid Neural Network (MahNN). The attention mechanism is to assign different weights to different parts of the input sequence to increase the computation efficiency and performance of neural models. In MahNN, two types of attentions are introduced: the syntactical attention and the semantical attention. The syntactical attention computes the importance of the syntactic elements (such as words or sentence) at the lower symbolic level and the semantical attention is used to compute the importance of the embedded space dimension corresponding to the upper latent semantics. We adopt the text classification as an exemplifying way to illustrate the ability of MahNN to understand texts. The experimental results on a variety of datasets demonstrate that MahNN outperforms most of the state-of-the-arts for text classification.
MCapsNet: Capsule Network for Text with Multi-Task Learning

Liqiang Xiao,Honglun Zhang,Wenqing Chen,Yongkun Wang,Yaohui Jin

DOI: https://doi.org/10.18653/v1/d18-1486

2018-01-01

Abstract:Multi-task learning has an ability to share the knowledge among related tasks and implicitly increase the training data. However, it has long been frustrated by the interference among tasks. This paper investigates the performance of capsule network for text, and proposes a capsule-based multi-task learning architecture, which is unified, simple and effective. With the advantages of capsules for feature clustering, proposed task routing algorithm can cluster the features for each task in the network, which helps reduce the interference among tasks. Experiments on six text classification datasets demonstrate the effectiveness of our models and their characteristics for feature clustering.
Multilabel Text Classification Using Multilayer DGAT

Hui Chen,Jian Huang,Nana Tao,Jijie Huang,Jing Wang

DOI: https://doi.org/10.1109/access.2022.3225445

IF: 3.9

2022-01-01

IEEE Access

Abstract:Text classification is one of the most fundamental tasks in text data analysis. Generally, textual data are extensive in scale and complex in the inherent relationship, leading to low accuracy in traditional classification models. This paper proposes a multi-label classification model based on multi-layer neural network architecture. We first construct a dual-attention mechanism graph neural network (named DGAT for short) to fuse the typological and informational features of the target node and the connected nodes. Secondly, we also build a multi-layer network architecture with multiple DGATs to expand the range of neighborhood nodes participating in the feature fusion to meet the needs of classifying different datasets. To ensure the learning ability of the model, we also use the residual network to solve the problem of error rise and gradient descent caused by multi-layer network architecture. Finally, we conducted a large number of experiments on five benchmark datasets. The results show that the accuracy of the proposed model is significantly better than that of the traditional models, and that there is a noticeable improvement when compared with other deep learning methods.

computer science, information systems,telecommunications,engineering, electrical & electronic
Recurrent Neural Network for Text Classification with Multi-Task Learning

Pengfei Liu,Xipeng Qiu,Xuanjing Huang

DOI: https://doi.org/10.48550/arxiv.1605.05101

2016-01-01

Abstract:Neural network based methods have obtained great progress on a variety of natural language processing tasks. However, in most previous works, the models are learned based on single-task supervised objectives, which often suffer from insufficient training data. In this paper, we use the multi-task learning framework to jointly learn across multiple related tasks. Based on recurrent neural network, we propose three different mechanisms of sharing information to model text with task-specific and shared layers. The entire network is trained jointly on all these tasks. Experiments on four benchmark text classification tasks show that our proposed models can improve the performance of a task with the help of other related tasks.
Adversarial Multi-task Learning for Text Classification.

Pengfei Liu,Xipeng Qiu,Xuanjing Huang

DOI: https://doi.org/10.18653/v1/p17-1001

2017-01-01

Abstract:Neural network models have shown their promising opportunities for multi-task learning, which focus on learning the shared layers to extract the common and task-invariant features. However, in most existing approaches, the extracted shared features are prone to be contaminated by task-specific features or the noise brought by other tasks. In this paper, we propose an adversarial multi-task learning framework, alleviating the shared and private latent feature spaces from interfering with each other. We conduct extensive experiments on 16 different text classification tasks, which demonstrates the benefits of our approach. Besides, we show that the shared knowledge learned by our proposed model can be regarded as off-the-shelf knowledge and easily transferred to new tasks. The datasets of all 16 tasks are publicly available at http://nlp.fudan.edu.cn/data/.
Multi-applicable Text Classification Based on Deep Neural Network

Jingjing Yang,Feng Deng,Suhuan Lv,Rui Wang,Qi Guo,Zongchun Kou,Shiqiang Chen

DOI: https://doi.org/10.1504/ijsnet.2022.127841

2022-01-01

International Journal of Sensor Networks

Abstract:Most long text classification methods based on deep learning have problems such as semantics sparsity and long-distance dependence. To tackle these problems, a novel multi-applicable text classification based on deep neural network (MTDNN) is proposed, which contains a bidirectional encoder representation from transformer (BERT), a dimension reduction layer, and the bidirectional long short-term memory (Bi-LSTM) combining attention mechanism. BERT is used to pre-train the words into the word embedding vectors. The dimension reduction layer extracts the feature phrase representations with higher weight from the word embedding vectors. The Bi-LSTM captures both the forward and backward context representations. An attention mechanism is employed to focus on the information outputted from the Bi-LSTM. The experimental results illustrate that the accuracy of the MTDNN for long text, short text classification, and sentiment analysis reaches 94.95%, 93.53% and 92.32%, respectively. The results show that our method outperforms the other state-of-the-art text classification methods.
MLGN:A Multi-Label Guided Network for Improving Text Classification

Qiang Liu,Jingzhe Chen,Fan Chen,Kejie Fang,Peng An,Yiming Zhang,Shiyu Du

DOI: https://doi.org/10.1109/access.2023.3299566

IF: 3.9

2023-08-09

IEEE Access

Abstract:Within natural language processing, multi-label classification is an important but challenging task. It is more complex than single-label classification since the document representations need to cover fine-grained label information, while the labels predicted by the model are often related. Recently, large pre-trained language models have achieved great performance on multi-label classification tasks, typically using embedding of [CLS] vector as the semantic representation of entire document and matching it with candidate labels. However, existing methods tend to ignore label semantics, and the relationships between labels and documents are not effectively mined. In addition, the linear layers used for fine-tuning do not take the correlations between labels into account. In this work, we propose a Multi-Label Guided Network (MLGN) capable to guide document representation with multi-label semantic information. Furthermore, we utilize correlation knowledge to enhance the original label prediction in downstream tasks. The extensive experimental trials show that MLGN transcends previous works on several publicly available datasets. Our source code is available at https://github.com/L199Q/MLGN.

computer science, information systems,telecommunications,engineering, electrical & electronic
BanditMTL: Bandit-based Multi-task Learning for Text Classification.

Yuren Mao,Zekai Wang,Weiwei Liu,Xuemin Lin,Wenbin Hu

DOI: https://doi.org/10.18653/v1/2021.acl-long.428

2021-01-01

Abstract:Yuren Mao, Zekai Wang, Weiwei Liu, Xuemin Lin, Wenbin Hu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Multi-label text classification based on semantic-sensitive graph convolutional network

Delong Zeng,Enze Zha,Jiayi Kuang,Ying Shen

DOI: https://doi.org/10.1016/j.knosys.2023.111303

IF: 8.139

2024-01-01

Knowledge-Based Systems

Abstract:Multi-Label Text Classification (MLTC) is an important but challenging task in the field of natural language processing. In this paper, we propose a novel method, Semantic-sensitive Graph Convolutional Network (S-GCN), by simultaneously considering semantic and word-global associations. More specifically, we first leverage texts, words, and labels to construct a global graph, which helps mine the relevance between similar documents. Then we design and pre-train an encoder to initialize text nodes in the graph, from which the semantic features of documents are extracted. Next, we employ a graph convolutional network to classify text nodes, which can well fuse node information. Finally, we normalize the adjacency matrix and store hidden layer representations of word nodes, tackling the issue that conventional graph-based methods cannot predict texts that did not appear during training. We conduct experiments on three public datasets, AAPD, RMSC-V2, and Reuters-21578, and demonstrate the superiority of our model over the baselines on the MLTC task. Source code is available at https://github.com/sysu18364004/SGCN.

computer science, artificial intelligence
Hierarchical Inter-Attention Network for Document Classification with Multi-Task Learning.

Bing Tian,Yong Zhang,Jin Wang,Chunxiao Xing

DOI: https://doi.org/10.24963/ijcai.2019/495

2019-01-01

Abstract:Document classification is an essential task in many real world applications. Existing approaches adopt both text semantics and document structure to obtain the document representation. However, these models usually require a large collection of annotated training instances, which are not always feasible, especially in low-resource settings. In this paper, we propose a multi-task learning framework to jointly train multiple related document classification tasks. We devise a hierarchical architecture to make use of the shared knowledge from all tasks to enhance the document representation of each task. We further propose an inter-attention approach to improve the task-specific modeling of documents with global information. Experimental results on 15 public datasets demonstrate the benefits of our proposed model.
Gated Neural Network with Regularized Loss for Multi-label Text Classification.

Yunlai Xu,Xiangying Ran,Wei Sun,Xiangyang Luo,Chongjun Wang

DOI: https://doi.org/10.1109/ijcnn.2019.8851686

2019-01-01

Abstract:Multi-label text classification is generally more difficult for its exponential output label space and more flexible input document considering its corresponding number of labels. We observed that some long documents have only one or two labels while some short documents are related to much more labels. In this paper, we propose a dynamic Gated Neural Network architecture to simultaneously process the document through two parts, one to extract the most informative semantics and filter redundant information, the other to capture context semantics with most of the information in the document kept. And semantics from these two parts are dynamically controlled by a gate to perform subsequent classification. And to better the training we incorporate label dependencies into traditional binary cross-entropy loss by exploiting label co-occurrences. Experimental results on AAPD and RCV1-V2 datasets show that our proposed methods achieve state-of-art performance. Further analysis of experimental results demonstrate that the proposed methods not only capture enough feature information from both long and short documents assigned with various labels, but also exploit label dependency to regularize the proposed model to further improve its performance.
Multi-Task Label Embedding for Text Classification

Honglun Zhang,Liqiang Xiao,Wenqing Chen,Yongkun Wang,Yaohui Jin

DOI: https://doi.org/10.18653/v1/d18-1484

2017-01-01

Abstract:Multi-task learning in text classification leverages implicit correlations among related tasks to extract common features and yield performance gains. However, a large body of previous work treats labels of each task as independent and meaningless one-hot vectors, which cause a loss of potential label information. In this paper, we propose Multi-Task Label Embedding to convert labels in text classification into semantic vectors, thereby turning the original tasks into vector matching tasks. Our model utilizes semantic correlations among tasks and makes it convenient to scale or transfer when new tasks are involved. Extensive experiments on five benchmark datasets for text classification show that our model can effectively improve the performances of related tasks with semantic representations of labels and additional information from each other.
Multi-task Learning with Bidirectional Language Models for Text Classification

Qi Yang,Lin Shang

DOI: https://doi.org/10.1109/ijcnn.2019.8852388

2019-01-01

Abstract:Multi-task learning is an effective approach to extract task-invariant features by leveraging potential information among related tasks, which improves the performance of a single task. Most existing work simply divides the whole model into shared and private spaces. Unfortunately, there is no explicit mechanism to prevent the two spaces from merging information from each other. As a result, the shared space may be mixed with task-specific features, while the private space may extract some task-invariant features. To alleviate the problem mentioned, in this paper, we propose a bidirectional language models based multi-task learning method for text classification. More specifically, we add language modelling as an auxiliary task to the private part, aiming to enhance its ability to extract task-specific features. In addition, to promote the shared part to learn common features, a loss constraint via uniform label distribution is introduced to the shared part. Finally, put task-specific features and taskinvariant features together in a weighted addition way to form the final representation, and it is then fed to the corresponding softmax layer. We do experiments on the FDU-MTL dataset which consists of 16 different text classification tasks. The experimental results show that our approach outperforms other typical methods.
Deep Multi-Task Learning with Shared Memory for Text Classification

Pengfei Liu,Xipeng Qiu,Xuanjing Huang

DOI: https://doi.org/10.18653/v1/d16-1012

2016-01-01

Abstract:Neural network based models have achieved impressive results on various specific tasks. However, in previous works, most models are learned separately based on single-task supervised objectives, which often suffer from insufficient training data. In this paper, we propose two deep architectures which can be trained jointly on multiple related tasks. More specifically, we augment neural model with an external memory, which is shared by several tasks. Experiments on two groups of text classification tasks show that our proposed architectures can improve the performance of a task with the help of other related tasks.
A Gated Graph Neural Network With Attention for Text Classification Based on Coupled P Systems

Jiaqi Zhang,Xiyu Liu

DOI: https://doi.org/10.1109/access.2023.3295572

IF: 3.9

2023-07-26

IEEE Access

Abstract:Text classification is a classical task in natural language processing. Prior traditional text classification methods rely on manually extracted features to a great extent, which are easily influenced by human subjectivity. Some existing text classification methods based on the artificial neural networks sometimes neglect the contextual semantic relationships of discontinuous word sequences, resulting in poor learning results. To alleviate these problems, we propose an attention-based gated graph neural network in the framework of coupled P systems (CPGANN) to automatically extract feature representations of nodes. The gating unit with attention is introduced to aggregate neighbor information to capture context semantic relations, and effectively alleviate the long-term dependence on discontinuous words. In order to obtain more discriminative nodes for classification, the attention mechanism is employed in CPGANN to extract keyword nodes before readout to aggregate subgraph representations. Extensive experiments on four real-world datasets demonstrate that CPGANN outperforms all other state-of-the-art baseline algorithms.

computer science, information systems,telecommunications,engineering, electrical & electronic
TW-TGNN: Two Windows Graph-Based Model for Text Classification

Xinyu Wu,Zheng Luo,Zhanwei Du,Jiaxin Wang,Chao Gao,Xianghua Li

DOI: https://doi.org/10.1109/IJCNN52387.2021.9534150

2021-01-01

Abstract:Text classification is the most fundamental and classical task in the natural language processing (NLP). Recently, graph neural network (GNN) methods, especially the graph-based model, have been applied for solving this issue because of their superior capacity of capturing the global co-occurrence information. However, some existing GNN-based methods adopt a corpus-level graph structure which causes a high memory consumption. In addition, these methods have not taken account of the global co-occurrence information and local semantic information at the same time. To address these problems, we propose a new GNN-based model, namely two windows text gnn model (TW-TGNN), for text classification. More specifically, we build text-level graph for each text with a local sliding window and a dynamic global window. For one thing, the local window sliding inside the text will acquire enough local semantic features. For another, the dynamic global window sliding betweent texts can generate dynamic shared weight matrix, which overcomes the limitation of the fixed corpus level co-occurrence and provides richer dynamic global information. Our experimental results on four benchmark datasets illustrate the improvement of the proposed method over state-of-the-art text classification methods. Moreover, we find that our method captures adequate global information for the short text which is beneficial for overcoming the insufficient contextual information in the process of the short text classification.

Gated Multi-Task Network for Text Classification

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

Database Systems for Advanced Applications

Learning What to Share: Leaky Multi-Task Network for Text Classification.

A Generalized Recurrent Neural Architecture for Text Classification with Multi-Task Learning.

Hierarchical Multi-Granularity Attention- Based Hybrid Neural Network for Text Classification.

MCapsNet: Capsule Network for Text with Multi-Task Learning

Multilabel Text Classification Using Multilayer DGAT

Recurrent Neural Network for Text Classification with Multi-Task Learning

Adversarial Multi-task Learning for Text Classification.

Multi-applicable Text Classification Based on Deep Neural Network

MLGN:A Multi-Label Guided Network for Improving Text Classification

BanditMTL: Bandit-based Multi-task Learning for Text Classification.

Multi-label text classification based on semantic-sensitive graph convolutional network

Hierarchical Inter-Attention Network for Document Classification with Multi-Task Learning.

Gated Neural Network with Regularized Loss for Multi-label Text Classification.

Multi-Task Label Embedding for Text Classification

Multi-task Learning with Bidirectional Language Models for Text Classification

Deep Multi-Task Learning with Shared Memory for Text Classification

A Gated Graph Neural Network With Attention for Text Classification Based on Coupled P Systems

TW-TGNN: Two Windows Graph-Based Model for Text Classification