BanditMTL: Bandit-based Multi-task Learning for Text Classification.

Yuren Mao,Zekai Wang,Weiwei Liu,Xuemin Lin,Wenbin Hu

DOI: https://doi.org/10.18653/v1/2021.acl-long.428

2021-01-01

Abstract:Yuren Mao, Zekai Wang, Weiwei Liu, Xuemin Lin, Wenbin Hu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

What problem does this paper attempt to address?

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

Xiaoyu Ji,Wanyang Hu,Yanyan Liang

DOI: https://doi.org/10.3390/math11244895

IF: 2.4

2023-01-01

Mathematics

Abstract:The MASSIVE dataset is a spoken-language comprehension resource package for slot filling, intent classification, and virtual assistant evaluation tasks. It contains multi-language utterances from human beings communicating with a virtual assistant. In this paper, we exploited the relationship between intent classification and slot filling to improve the exact match accuracy by proposing five models with hierarchical and bidirectional architectures. There are two variants for hierarchical architectures and three variants for bidirectional architectures. These are the hierarchical concatenation model, the hierarchical attention-based model, the bidirectional max-pooling model, the bidirectional LSTM model, and the bidirectional attention-based model. The results of our models showed a significant improvement in the averaged exact match accuracy. The hierarchical attention-based model improved the accuracy by 1.01 points for the full training dataset. As for the zero-shot setup, we observed that the exact match accuracy increased from 53.43 to 53.91. In this study, we observed that, for multi-task problems, utilizing the relevance between different tasks can help in improving the model's overall performance.
Cross-Lingual Text Image Recognition Via Multi-Task Sequence to Sequence Learning.

Zhuo Chen,Fei Yin,Xu-Yao Zhang,Qing Yang,Chena-Lin Liu

DOI: https://doi.org/10.1109/icpr48806.2021.9412281

2021-01-01

Abstract:This paper considers recognizing texts shown in a source language and translating into a target language, without generating the intermediate source language text image recognition results. We call this problem Cross-Lingual Text Image Recognition (CLTIR). To solve this problem, we propose a multi-task system containing a main task of CLTIR and an auxiliary task of Mono-Lingual Text Image Recognition (MLTIR) simultaneously. Two different sequence to sequence learning methods, a convolution based attention model and a Bidirectional Long Short-Term Memory (BLSTM) model with Connectionist Temporal Classification (CTC), are adopted for these tasks respectively. We evaluate the system on a newly collected Chinese-English bilingual movie subtitle image dataset. Experimental results demonstrate the multi-task learning framework performs superiorly in both languages.
Gated Multi-Task Network for Text Classification

Liqiang Xiao,Honglun Zhang,Wenqing Chen

DOI: https://doi.org/10.18653/v1/n18-2114

2018-01-01

Abstract:Liqiang Xiao, Honglun Zhang, Wenqing Chen. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 2018.
Multi-task Learning with Bidirectional Language Models for Text Classification

Qi Yang,Lin Shang

DOI: https://doi.org/10.1109/ijcnn.2019.8852388

2019-01-01

Abstract:Multi-task learning is an effective approach to extract task-invariant features by leveraging potential information among related tasks, which improves the performance of a single task. Most existing work simply divides the whole model into shared and private spaces. Unfortunately, there is no explicit mechanism to prevent the two spaces from merging information from each other. As a result, the shared space may be mixed with task-specific features, while the private space may extract some task-invariant features. To alleviate the problem mentioned, in this paper, we propose a bidirectional language models based multi-task learning method for text classification. More specifically, we add language modelling as an auxiliary task to the private part, aiming to enhance its ability to extract task-specific features. In addition, to promote the shared part to learn common features, a loss constraint via uniform label distribution is introduced to the shared part. Finally, put task-specific features and taskinvariant features together in a weighted addition way to form the final representation, and it is then fed to the corresponding softmax layer. We do experiments on the FDU-MTL dataset which consists of 16 different text classification tasks. The experimental results show that our approach outperforms other typical methods.
Task Variance Regularized Multi-Task Learning.

Yuren Mao,Zekai Wang,Weiwei Liu,Xuemin Lin,Wenbin Hu

DOI: https://doi.org/10.1109/tkde.2022.3207049

IF: 9.235

2022-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Multi-task Learning (MTL), which involves the simultaneous learning of multiple tasks, can achieve better performance than learning each task independently. It has achieved great success in various applications, ranging from Computer Vision (CV) to Natural Language Processing (NLP). In MTL, the losses of the including tasks are jointly optimized. However, it is common for these tasks to be competing. When the tasks are competing, minimizing the losses of some tasks increases the losses of others, which accordingly increases the task variance (variance between the task-specific loss); furthermore, it induces under-fitting in some tasks and over-fitting in others, which degenerates the generalization performance of an MTL model. To address this issue, it is necessary to control the task variance; thus, task variance regularization is a natural choice. While intuitive, task variance regularization remains unexplored in MTL. Accordingly, to fill this gap, we study the generalization error bound of MTL through the lens of task variance and propose the task variance matters the generalization performance of MTL. Furthermore, this paper investigates how the task variance might be effectively regularized, and consequently proposes a multi-task learning method based on adversarial multi-armed bandit. The proposed method, dubbed BanditMTL, regularizes the task variance by means of a mirror gradient ascent-descent algorithm. Adopting BanditMTL both in CV and NLP applications is found to achieve state-of-the-art performance. The results of extensive experiments back up our theoretical analysis and validate the superiority of our proposals.
All Birds with One Stone: Multi-task Text Classification for Efficient Inference with One Forward Pass

Jiaxin Huang,Tianqi Liu,Jialu Liu,Adam D. Lelkes,Cong Yu,Jiawei Han

DOI: https://doi.org/10.48550/arXiv.2205.10744

2022-05-22

Abstract:Multi-Task Learning (MTL) models have shown their robustness, effectiveness, and efficiency for transferring learned knowledge across tasks. In real industrial applications such as web content classification, multiple classification tasks are predicted from the same input text such as a web article. However, at the serving time, the existing multitask transformer models such as prompt or adaptor based approaches need to conduct N forward passes for N tasks with O(N) computation cost. To tackle this problem, we propose a scalable method that can achieve stronger performance with close to O(1) computation cost via only one forward pass. To illustrate real application usage, we release a multitask dataset on news topic and style classification. Our experiments show that our proposed method outperforms strong baselines on both the GLUE benchmark and our news dataset. Our code and dataset are publicly available at <a class="link-external link-https" href="https://bit.ly/mtop-code" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Machine Learning
Active Multi-task Learning Via Bandits.

Meng Fang,Dacheng Tao

DOI: https://doi.org/10.1137/1.9781611974010.57

2015-01-01

Abstract:In multi-task learning, the multiple related tasks allow each one to benefit from the learning of the others, and labeling instances for one task can also affect the other tasks especially when the task has a small number of labeled data. Thus labeling effective instances across different learning tasks is important for improving the generalization error of all tasks. In this paper, we propose a new active multi-task learning paradigm, which selectively samples effective instances for multi-task learning. Inspired by the multi-armed bandits, which can balance the trade-off between the exploitation and exploration, we introduce a new active learning strategy and cast the selection procedure as a bandit framework. We consider both the risk of multi-task learner and the corresponding confidence bounds and our selection tries to balance this trade-off. Our proposed method is a sequential algorithm, which at each round maintains a sampling distribution on the pool of data, queries the label for an instance according to this distribution and updates the distribution based on the newly trained multi-task learner. We provide an implementation of our algorithm based on a popular multi-task learning algorithm that is trace-norm regularization method. Theoretical guarantees are developed by exploiting the Rademacher complexities. Comprehensive experiments show the effectiveness and efficiency of the proposed approach.
Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits

Julia Kreutzer,David Vilar,Artem Sokolov

DOI: https://doi.org/10.48550/arXiv.2110.06997

2021-10-14

Abstract:Training data for machine translation (MT) is often sourced from a multitude of large corpora that are multi-faceted in nature, e.g. containing contents from multiple domains or different levels of quality or complexity. Naturally, these facets do not occur with equal frequency, nor are they equally important for the test scenario at hand. In this work, we propose to optimize this balance jointly with MT model parameters to relieve system developers from manual schedule design. A multi-armed bandit is trained to dynamically choose between facets in a way that is most beneficial for the MT system. We evaluate it on three different multi-facet applications: balancing translationese and natural training data, or data from multiple domains or multiple language pairs. We find that bandit learning leads to competitive MT systems across tasks, and our analysis provides insights into its learned strategies and the underlying data sets.

Computation and Language,Artificial Intelligence
Multi-Task Label Embedding for Text Classification

Honglun Zhang,Liqiang Xiao,Wenqing Chen,Yongkun Wang,Yaohui Jin

DOI: https://doi.org/10.18653/v1/d18-1484

2017-01-01

Abstract:Multi-task learning in text classification leverages implicit correlations among related tasks to extract common features and yield performance gains. However, a large body of previous work treats labels of each task as independent and meaningless one-hot vectors, which cause a loss of potential label information. In this paper, we propose Multi-Task Label Embedding to convert labels in text classification into semantic vectors, thereby turning the original tasks into vector matching tasks. Our model utilizes semantic correlations among tasks and makes it convenient to scale or transfer when new tasks are involved. Extensive experiments on five benchmark datasets for text classification show that our model can effectively improve the performances of related tasks with semantic representations of labels and additional information from each other.
The UMD Neural Machine Translation Systems at WMT17 Bandit Learning Task

Amr Sharaf,Shi Feng,Khanh Nguyen,Kianté Brantley,Hal Daumé III

DOI: https://doi.org/10.48550/arXiv.1708.01318

2017-08-08

Abstract:We describe the University of Maryland machine translation systems submitted to the WMT17 German-English Bandit Learning Task. The task is to adapt a translation system to a new domain, using only bandit feedback: the system receives a German sentence to translate, produces an English sentence, and only gets a scalar score as feedback. Targeting these two challenges (adaptation and bandit learning), we built a standard neural machine translation system and extended it in two ways: (1) robust reinforcement learning techniques to learn effectively from the bandit feedback, and (2) domain adaptation using data selection from a large corpus of parallel data.

Computation and Language,Artificial Intelligence,Human-Computer Interaction
Adversarial Multi-task Learning for Text Classification.

Pengfei Liu,Xipeng Qiu,Xuanjing Huang

DOI: https://doi.org/10.18653/v1/p17-1001

2017-01-01

Abstract:Neural network models have shown their promising opportunities for multi-task learning, which focus on learning the shared layers to extract the common and task-invariant features. However, in most existing approaches, the extracted shared features are prone to be contaminated by task-specific features or the noise brought by other tasks. In this paper, we propose an adversarial multi-task learning framework, alleviating the shared and private latent feature spaces from interfering with each other. We conduct extensive experiments on 16 different text classification tasks, which demonstrates the benefits of our approach. Besides, we show that the shared knowledge learned by our proposed model can be regarded as off-the-shelf knowledge and easily transferred to new tasks. The datasets of all 16 tasks are publicly available at http://nlp.fudan.edu.cn/data/.
An End-to-End Scalable Iterative Sequence Tagging with Multi-Task Learning.

Lin Gui,Jiachen Du,Zhishan Zhao,Yulan He,Ruifeng Xu,Chuang Fan

DOI: https://doi.org/10.1007/978-3-319-99501-4_25

2018-01-01

Abstract:Multi-task learning (MTL) models, which pool examples arisen out of several tasks, have achieved remarkable results in language processing. However, multi-task learning is not always effective when compared with the single-task methods in sequence tagging. One possible reason is that existing methods to multi-task sequence tagging often reply on lower layer parameter sharing to connect different tasks. The lack of interactions between different tasks results in limited performance improvement. In this paper, we propose a novel multi-task learning architecture which could iteratively utilize the prediction results of each task explicitly. We train our model for part-of-speech (POS) tagging, chunking and named entity recognition (NER) tasks simultaneously. Experimental results show that without any task-specific features, our model obtains the state-of-the-art performance on both chunking and NER.
A Generalized Recurrent Neural Architecture for Text Classification with Multi-Task Learning.

Honglun Zhang,Liqiang Xiao,Yongkun Wang,Yaohui Jin

DOI: https://doi.org/10.24963/ijcai.2017/473

2017-01-01

Abstract:Multi-task learning leverages potential correlations among related tasks to extract common features and yield performance gains. However, most previous works only consider simple or weak interactions, thereby failing to model complex correlations among three or more tasks. In this paper, we propose a multi-task learning architecture with four types of recurrent neural layers to fuse information across multiple related tasks. The architecture is structurally flexible and considers various interactions among tasks, which can be regarded as a generalized case of many previous works. Extensive experiments on five benchmark datasets for text classification show that our model can significantly improve performances of related tasks with additional information from others.
Multi-Task Learning in Natural Language Processing: An Overview

Shijie Chen,Yu Zhang,Qiang Yang

2024-04-28

Abstract:Deep learning approaches have achieved great success in the field of Natural Language Processing (NLP). However, directly training deep neural models often suffer from overfitting and data scarcity problems that are pervasive in NLP tasks. In recent years, Multi-Task Learning (MTL), which can leverage useful information of related tasks to achieve simultaneous performance improvement on these tasks, has been used to handle these problems. In this paper, we give an overview of the use of MTL in NLP tasks. We first review MTL architectures used in NLP tasks and categorize them into four classes, including parallel architecture, hierarchical architecture, modular architecture, and generative adversarial architecture. Then we present optimization techniques on loss construction, gradient regularization, data sampling, and task scheduling to properly train a multi-task model. After presenting applications of MTL in a variety of NLP tasks, we introduce some benchmark datasets. Finally, we make a conclusion and discuss several possible research directions in this field.

Artificial Intelligence
Many-Class Text Classification with Matching

Yi Song,Yuxian Gu,Minlie Huang

DOI: https://doi.org/10.48550/arXiv.2205.11409

2022-05-23

Abstract:In this work, we formulate \textbf{T}ext \textbf{C}lassification as a \textbf{M}atching problem between the text and the labels, and propose a simple yet effective framework named TCM. Compared with previous text classification approaches, TCM takes advantage of the fine-grained semantic information of the classification labels, which helps distinguish each class better when the class number is large, especially in low-resource scenarios. TCM is also easy to implement and is compatible with various large pretrained language models. We evaluate TCM on 4 text classification datasets (each with 20+ labels) in both few-shot and full-data settings, and this model demonstrates significant improvements over other text classification paradigms. We also conduct extensive experiments with different variants of TCM and discuss the underlying factors of its success. Our method and analyses offer a new perspective on text classification.

Computation and Language
Multi-Task Learning Architectures and Applications

Andy Yan,Xin Wang,Yanlai Yang,Roy Fox,Xiaolong Wang,Joseph Gonzalez

2020-01-01

Abstract:Multi-Task Learning Architectures and Applications by Andy Yan Master of Science in Electrical Engineering and Computer Science University of California, Berkeley Professor Joseph Gonzalez, Doctor Xiaolong Wang, Multi-task learning fundamentally involves utilizing multiple tasks to assist with generalization. In this report, we first investigate the motivation for research in multi-task learning, showing that when training multiple tasks together on the same neural network, performance may benefit if the tasks are related and suffer if the tasks are not. We then study multi-task architectures and evaluate in depth a mixture-of-experts model. We show that in experiments on the CIFAR-100 MTL domain, multi-clustering outperforms prior architectures in accuracy and computation time. Next, we apply multi-task learning to the object detection task using the BDD100K dataset. We explain our method of training the object detection task with the self-supervised tasks of angle and distance prediction and colorization, demonstrating performance benefits. Lastly, we demonstrate our work in few-shot learning, where we proposed a method to train a task to which we have little to no data by exploiting the task’s compositionality.
Multi-applicable Text Classification Based on Deep Neural Network

Jingjing Yang,Feng Deng,Suhuan Lv,Rui Wang,Qi Guo,Zongchun Kou,Shiqiang Chen

DOI: https://doi.org/10.1504/ijsnet.2022.127841

2022-01-01

International Journal of Sensor Networks

Abstract:Most long text classification methods based on deep learning have problems such as semantics sparsity and long-distance dependence. To tackle these problems, a novel multi-applicable text classification based on deep neural network (MTDNN) is proposed, which contains a bidirectional encoder representation from transformer (BERT), a dimension reduction layer, and the bidirectional long short-term memory (Bi-LSTM) combining attention mechanism. BERT is used to pre-train the words into the word embedding vectors. The dimension reduction layer extracts the feature phrase representations with higher weight from the word embedding vectors. The Bi-LSTM captures both the forward and backward context representations. An attention mechanism is employed to focus on the information outputted from the Bi-LSTM. The experimental results illustrate that the accuracy of the MTDNN for long text, short text classification, and sentiment analysis reaches 94.95%, 93.53% and 92.32%, respectively. The results show that our method outperforms the other state-of-the-art text classification methods.
Hierarchical Multilabel Text Classification Via Multitask Learning.

Yipeng Yu,Zixun Sun,Chi Sun,Wenqiang Liu

DOI: https://doi.org/10.1109/ictai52525.2021.00180

2021-01-01

Abstract:Hierarchical multilabel classification is a variant of classification where instances might belong to multiple labels and these labels come from a hierarchy. In this paper, we solve the hierarchical multilabel text classification problem of professionally-generated content via multitask learning. More specifically, we focus on (1) how to build models that can share features well in multitask learning, (2) how to incorporate the label dependence into the training procedure of the models, and (3) how to combine the predicted labels of different levels in the hierarchy. To make the experiments simple and comparable, we bring in the state-of-art BERT model as the base model in our work. Experiment results show that the multitask models we build are competitive, the penalty loss we propose is able to improve the performance, and the union operation is the best choice to handle prediction contradiction. In other words, the time cost is reduced but performance is improved via our multitask learning approach.
Exploring multi-tasking learning in document attribute classification

Tanmoy Mondal,Abhijit Das,Zuheng Ming

DOI: https://doi.org/10.1016/j.patrec.2022.02.015

IF: 4.757

2022-05-01

Pattern Recognition Letters

Abstract:In this work, we adhere to explore a Multi-Tasking learning (MTL) based network to perform document attribute classification such as the font type, font size, font emphasis and scanning resolution classification of a document image. To accomplish these tasks, we operate on either segmented word level or on uniformed size patches randomly cropped out of the document. Furthermore, a hybrid convolution neural network (CNN) architecture "MTL+MI", which is based on the combination of MTL and Multi-Instance (MI) of patch and word is used to accomplish joint learning for the classification of the same document attributes. The contribution of this paper are three fold: firstly, based on segmented word images and patches, we present a MTL based network for the classification of a full document image. Secondly, we propose a MTL and MI (using segmented words and patches) based combined CNN architecture ("MTL+MI") for the classification of same document attributes. Thirdly, based on the multi-tasking classifications of the words and/or patches, we propose an intelligent voting system which is based on the posterior probabilities of each words and/or patches to perform the classification of document's attributes of complete document image.

computer science, artificial intelligence

BanditMTL: Bandit-based Multi-task Learning for Text Classification.

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

Cross-Lingual Text Image Recognition Via Multi-Task Sequence to Sequence Learning.

Gated Multi-Task Network for Text Classification

Multi-task Learning with Bidirectional Language Models for Text Classification

Task Variance Regularized Multi-Task Learning.

All Birds with One Stone: Multi-task Text Classification for Efficient Inference with One Forward Pass

Active Multi-task Learning Via Bandits.

Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Bandits

Multi-Task Label Embedding for Text Classification

The UMD Neural Machine Translation Systems at WMT17 Bandit Learning Task

Adversarial Multi-task Learning for Text Classification.

An End-to-End Scalable Iterative Sequence Tagging with Multi-Task Learning.

A Generalized Recurrent Neural Architecture for Text Classification with Multi-Task Learning.

Multi-Task Learning in Natural Language Processing: An Overview

Many-Class Text Classification with Matching

Multi-Task Learning Architectures and Applications

Multi-applicable Text Classification Based on Deep Neural Network

Hierarchical Multilabel Text Classification Via Multitask Learning.

Exploring multi-tasking learning in document attribute classification