Rethinking Data Augmentation in Text-to-text Paradigm.

Yanan Chen,Yang Liu

2022-01-01

Abstract:As manually labelling data can be costly, some recent studies tend to augment the training data for improving the generalization power of machine learning models, known as

What problem does this paper attempt to address?

Text Data Augmentation for Deep Learning

Connor Shorten,Taghi M Khoshgoftaar,Borko Furht

DOI: https://doi.org/10.1186/s40537-021-00492-0

Abstract:Natural Language Processing (NLP) is one of the most captivating applications of Deep Learning. In this survey, we consider how the Data Augmentation training strategy can aid in its development. We begin with the major motifs of Data Augmentation summarized into strengthening local decision boundaries, brute force training, causality and counterfactual examples, and the distinction between meaning and form. We follow these motifs with a concrete list of augmentation frameworks that have been developed for text data. Deep Learning generally struggles with the measurement of generalization and characterization of overfitting. We highlight studies that cover how augmentations can construct test sets for generalization. NLP is at an early stage in applying Data Augmentation compared to Computer Vision. We highlight the key differences and promising ideas that have yet to be tested in NLP. For the sake of practical implementation, we describe tools that facilitate Data Augmentation such as the use of consistency regularization, controllers, and offline and online augmentation pipelines, to preview a few. Finally, we discuss interesting topics around Data Augmentation in NLP such as task-specific augmentations, the use of prior knowledge in self-supervised learning versus Data Augmentation, intersections with transfer and multi-task learning, and ideas for AI-GAs (AI-Generating Algorithms). We hope this paper inspires further research interest in Text Data Augmentation.
Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification

Shuhuai Ren,Jinchao Zhang,Lei Li,Xu Sun,Jie Zhou

2021-01-01

Abstract:Data augmentation aims to enrich training samples for alleviating the overfitting issue in low-resource or class-imbalanced situations. Traditional methods first devise task-specific operations such as Synonym Substitute, then preset the corresponding parameters such as the substitution rate artificially, which require a lot of prior knowledge and are prone to fall into the sub-optimum. Besides, the number of editing operations is limited in the previous methods, which decreases the diversity of the augmented data and thus restricts the performance gain. To overcome the above limitations, we propose a framework named Text AutoAugment (TAA) to establish a compositional and learnable paradigm for data augmentation. We regard a combination of various operations as an augmentation policy and utilize an efficient Bayesian Optimization algorithm to automatically search for the best policy, which substantially improves the generalization capability of models. Experiments on six benchmark datasets show that TAA boosts classification accuracy in low-resource and class-imbalanced regimes by an average of 8.8% and 9.7%, respectively, outperforming strong baselines.(1)
SDA: Improving Text Generation with Self Data Augmentation

Ping Yu,Ruiyi Zhang,Yang Zhao,Yizhe Zhang,Chunyuan Li,Changyou Chen

DOI: https://doi.org/10.48550/arXiv.2101.03236

2021-01-02

Abstract:Data augmentation has been widely used to improve deep neural networks in many research fields, such as computer vision. However, less work has been done in the context of text, partially due to its discrete nature and the complexity of natural languages. In this paper, we propose to improve the standard maximum likelihood estimation (MLE) paradigm by incorporating a self-imitation-learning phase for automatic data augmentation. Unlike most existing sentence-level augmentation strategies, which are only applied to specific models, our method is more general and could be easily adapted to any MLE-based training procedure. In addition, our framework allows task-specific evaluation metrics to be designed to flexibly control the generated sentences, for example, in terms of controlling vocabulary usage and avoiding nontrivial repetitions. Extensive experimental results demonstrate the superiority of our method on two synthetic and several standard real datasets, significantly improving related baselines.

Computation and Language,Machine Learning
Data augmentation in natural language processing: a novel text generation approach for long and short text classifiers

Markus Bayer,Marc-André Kaufhold,Björn Buchhold,Marcel Keller,Jörg Dallmeyer,Christian Reuter

DOI: https://doi.org/10.1007/s13042-022-01553-3

2022-04-12

International Journal of Machine Learning and Cybernetics

Abstract:Abstract In many cases of machine learning, research suggests that the development of training data might have a higher relevance than the choice and modelling of classifiers themselves. Thus, data augmentation methods have been developed to improve classifiers by artificially created training data. In NLP, there is the challenge of establishing universal rules for text transformations which provide new linguistic patterns. In this paper, we present and evaluate a text generation method suitable to increase the performance of classifiers for long and short texts. We achieved promising improvements when evaluating short as well as long text tasks with the enhancement by our text generation method. Especially with regard to small data analytics, additive accuracy gains of up to 15.53% and 3.56% are achieved within a constructed low data regime, compared to the no augmentation baseline and another data augmentation technique. As the current track of these constructed regimes is not universally applicable, we also show major improvements in several real world low data tasks (up to +4.84 F1-score). Since we are evaluating the method from many perspectives (in total 11 datasets), we also observe situations where the method might not be suitable. We discuss implications and patterns for the successful application of our approach on different types of datasets.

computer science, artificial intelligence
Exploring Data Augmentation Methods on Social Media Corpora

Isabel Garcia Pietri,Kineret Stanley

2023-03-04

Abstract:Data augmentation has proven widely effective in computer vision. In Natural Language Processing (NLP) data augmentation remains an area of active research. There is no widely accepted augmentation technique that works well across tasks and model architectures. In this paper we explore data augmentation techniques in the context of text classification using two social media datasets. We explore popular varieties of data augmentation, starting with oversampling, Easy Data Augmentation (Wei and Zou, 2019) and Back-Translation (Sennrich et al., 2015). We also consider Greyscaling, a relatively unexplored data augmentation technique that seeks to mitigate the intensity of adjectives in examples. Finally, we consider a few-shot learning approach: Pattern-Exploiting Training (PET) (Schick et al., 2020). For the experiments we use a BERT transformer architecture. Results show that augmentation techniques provide only minimal and inconsistent improvements. Synonym replacement provided evidence of some performance improvement and adjective scales with Grayscaling is an area where further exploration would be valuable. Few-shot learning experiments show consistent improvement over supervised training, and seem very promising when classes are easily separable but further exploration would be valuable.

Computation and Language
Data Augmentation Using Virtual Word Insertion Techniques in Text Classification Tasks

Zhigao Long,Hong Li,Jiawen Shi,Xin Ma

DOI: https://doi.org/10.1111/exsy.13519

IF: 3.3

2024-01-01

Expert Systems

Abstract:Labelling multiple training examples for text classification models is usually time-consuming and complex. Data augmentation can be used to automatically expand the dataset by transforming the original data. However, it may cause semantic changes without modifying the labels, which reduces the effectiveness of the classifiers. In this paper, we propose a data-augmentation method called the virtual word insertion technique, which generates new sentences by randomly inserting virtual words into existing sentences. Two methods are used to achieve virtual word embedding: unweighted average and weighted average. Furthermore, a new concept of weight is proposed: the class deviation factor, which demonstrates the correlation between words and classes. Based on this new concept, different weights are assigned to words of different classes. Experiments are conducted on five different classification tasks. Ablation experiments are also performed to explore the effects of random operation and number of augmented sentences for classification results. The results of these experiments show that our method improves the classification performance and outperforms two other contrasting data-augmentation methods in automatically augmenting the dataset. Compared to raw datasets, the average accuracy improvement of our method is 3.5% for a small-scale dataset and 1% for a large-scale dataset.
AugGPT: Leveraging ChatGPT for Text Data Augmentation

Haixing Dai,Zhengliang Liu,Wenxiong Liao,Xiaoke Huang,Yihan Cao,Zihao Wu,Lin Zhao,Shaochen Xu,Wei Liu,Ninghao Liu,Sheng Li,Dajiang Zhu,Hongmin Cai,Lichao Sun,Quanzheng Li,Dinggang Shen,Tianming Liu,Xiang Li

DOI: https://doi.org/10.48550/arXiv.2302.13007

2023-03-20

Abstract:Text data augmentation is an effective strategy for overcoming the challenge of limited sample sizes in many natural language processing (NLP) tasks. This challenge is especially prominent in the few-shot learning scenario, where the data in the target domain is generally much scarcer and of lowered quality. A natural and widely-used strategy to mitigate such challenges is to perform data augmentation to better capture the data invariance and increase the sample size. However, current text data augmentation methods either can't ensure the correct labeling of the generated data (lacking faithfulness) or can't ensure sufficient diversity in the generated data (lacking compactness), or both. Inspired by the recent success of large language models, especially the development of ChatGPT, which demonstrated improved language comprehension abilities, in this work, we propose a text data augmentation approach based on ChatGPT (named AugGPT). AugGPT rephrases each sentence in the training samples into multiple conceptually similar but semantically different samples. The augmented samples can then be used in downstream model training. Experiment results on few-shot learning text classification tasks show the superior performance of the proposed AugGPT approach over state-of-the-art text data augmentation methods in terms of testing accuracy and distribution of the augmented samples.

Computation and Language,Artificial Intelligence,Machine Learning
GenAug: Data Augmentation for Finetuning Text Generators

Steven Y. Feng,Varun Gangal,Dongyeop Kang,Teruko Mitamura,Eduard Hovy

DOI: https://doi.org/10.48550/arXiv.2010.01794

2020-10-10

Abstract:In this paper, we investigate data augmentation for text generation, which we call GenAug. Text generation and language modeling are important tasks within natural language processing, and are especially challenging for low-data regimes. We propose and evaluate various augmentation methods, including some that incorporate external knowledge, for finetuning GPT-2 on a subset of Yelp Reviews. We also examine the relationship between the amount of augmentation and the quality of the generated text. We utilize several metrics that evaluate important aspects of the generated text including its diversity and fluency. Our experiments demonstrate that insertion of character-level synthetic noise and keyword replacement with hypernyms are effective augmentation methods, and that the quality of generations improves to a peak at approximately three times the amount of original data.

Computation and Language,Artificial Intelligence,Machine Learning
On Evaluation Protocols for Data Augmentation in a Limited Data Scenario

Frédéric Piedboeuf,Philippe Langlais

2024-09-17

Abstract:Textual data augmentation (DA) is a prolific field of study where novel techniques to create artificial data are regularly proposed, and that has demonstrated great efficiency on small data settings, at least for text classification tasks. In this paper, we challenge those results, showing that classical data augmentation (which modify sentences) is simply a way of performing better fine-tuning, and that spending more time doing so before applying data augmentation negates its effect. This is a significant contribution as it answers several questions that were left open in recent years, namely~: which DA technique performs best (all of them as long as they generate data close enough to the training set, as to not impair training) and why did DA show positive results (facilitates training of network). We further show that zero- and few-shot DA via conversational agents such as ChatGPT or LLama2 can increase performances, confirming that this form of data augmentation is preferable to classical methods.

Computation and Language,Artificial Intelligence,Machine Learning
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP

Jiaao Chen,Derek Tam,Colin Raffel,Mohit Bansal,Diyi Yang

DOI: https://doi.org/10.1162/tacl_a_00542

IF: 10.9

2023-01-01

Transactions of the Association for Computational Linguistics

Abstract:Abstract NLP has achieved great progress in the past decade through the use of neural models and large labeled datasets. The dependence on abundant data prevents NLP models from being applied to low-resource settings or novel tasks where significant time, money, or expertise is required to label massive amounts of textual data. Recently, data augmentation methods have been explored as a means of improving data efficiency in NLP. To date, there has been no systematic empirical overview of data augmentation for NLP in the limited labeled data setting, making it difficult to understand which methods work in which settings. In this paper, we provide an empirical survey of recent progress on data augmentation for NLP in the limited labeled data setting, summarizing the landscape of methods (including token-level augmentations, sentence-level augmentations, adversarial augmentations, and hidden-space augmentations) and carrying out experiments on 11 datasets covering topics/news classification, inference tasks, paraphrasing tasks, and single-sentence tasks. Based on the results, we draw several conclusions to help practitioners choose appropriate augmentations in different settings and discuss the current challenges and future directions for limited data learning in NLP.

English Else
Data Augmentation for Sentiment Classification with Semantic Preservation and Diversity

Guoqing Chao,Jingyao Liu,Mingyu Wang,Dianhui Chu

DOI: https://doi.org/10.1016/j.knosys.2023.111038

IF: 8.139

2023-01-01

Knowledge-Based Systems

Abstract:Data augmentation is a commonly-used technique to avoid over-fitting in deep learning. However, the mechanism behind effective data augmentation methods is unclear. To address this issue, we explore and identify two critical factors: semantic preservation and diversity to assess the quality of data augmentation in natural language processing. Our study focus on text sentiment classification and examines these two factors on two commonly-used data augmentation methods: synonym replacement and random deletion. Based on the discovery, we propose two new augmentation methods: TF-IDF word dropout and adaptive synonym replacement. Experimental results demonstrate that these two new data augmentation methods are effective. Moreover, with further experiments, we summarize three strategies for improving data augmentation methods in sentiment classification task. These strategies are employing online augmentation, introducing word importance into word sampling process, and filtering augmented data based on the current model state. We hope that our study will inspire some new perspectives on the underlying principles of data augmentation’s effectiveness and contribute to a systematic study of data augmentation methods in future.
Improving Text Classification with Large Language Model-Based Data Augmentation

Huanhuan Zhao,Haihua Chen,Thomas A. Ruggles,Yunhe Feng,Debjani Singh,Hong-Jun Yoon

DOI: https://doi.org/10.3390/electronics13132535

IF: 2.9

2024-06-28

Electronics

Abstract:Large Language Models (LLMs) such as ChatGPT possess advanced capabilities in understanding and generating text. These capabilities enable ChatGPT to create text based on specific instructions, which can serve as augmented data for text classification tasks. Previous studies have approached data augmentation (DA) by either rewriting the existing dataset with ChatGPT or generating entirely new data from scratch. However, it is unclear which method is better without comparing their effectiveness. This study investigates the application of both methods to two datasets: a general-topic dataset (Reuters news data) and a domain-specific dataset (Mitigation dataset). Our findings indicate that: 1. ChatGPT generated new data consistently enhanced model's classification results for both datasets. 2. Generating new data generally outperforms rewriting existing data, though crafting the prompts carefully is crucial to extract the most valuable information from ChatGPT, particularly for domain-specific data. 3. The augmentation data size affects the effectiveness of DA; however, we observed a plateau after incorporating 10 samples. 4. Combining the rewritten sample with new generated sample can potentially further improve the model's performance.

engineering, electrical & electronic,computer science, information systems,physics, applied
Data Augmentation Methods for Enhancing Robustness in Text Classification Tasks

Huidong Tang,Sayaka Kamei,Yasuhiko Morimoto

DOI: https://doi.org/10.3390/a16010059

2023-01-17

Algorithms

Abstract:Text classification is widely studied in natural language processing (NLP). Deep learning models, including large pre-trained models like BERT and DistilBERT, have achieved impressive results in text classification tasks. However, these models' robustness against adversarial attacks remains an area of concern. To address this concern, we propose three data augmentation methods to improve the robustness of such pre-trained models. We evaluated our methods on four text classification datasets by fine-tuning DistilBERT on the augmented datasets and exposing the resulting models to adversarial attacks to evaluate their robustness. In addition to enhancing the robustness, our proposed methods can improve the accuracy and F1-score on three datasets. We also conducted comparison experiments with two existing data augmentation methods. We found that one of our proposed methods demonstrates a similar improvement in terms of performance, but all demonstrate a superior robustness improvement.
DAGAM: Data Augmentation with Generation And Modification

Byeong-Cheol Jo,Tak-Sung Heo,Yeongjoon Park,Yongmin Yoo,Won Ik Cho,Kyungsun Kim

DOI: https://doi.org/10.48550/arXiv.2204.02633

2022-04-06

Computation and Language

Abstract:Text classification is a representative downstream task of natural language processing, and has exhibited excellent performance since the advent of pre-trained language models based on Transformer architecture. However, in pre-trained language models, under-fitting often occurs due to the size of the model being very large compared to the amount of available training data. Along with significant importance of data collection in modern machine learning paradigm, studies have been actively conducted for natural language data augmentation. In light of this, we introduce three data augmentation schemes that help reduce underfitting problems of large-scale language models. Primarily we use a generation model for data augmentation, which is defined as Data Augmentation with Generation (DAG). Next, we augment data using text modification techniques such as corruption and word order change (Data Augmentation with Modification, DAM). Finally, we propose Data Augmentation with Generation And Modification (DAGAM), which combines DAG and DAM techniques for a boosted performance. We conduct data augmentation for six benchmark datasets of text classification task, and verify the usefulness of DAG, DAM, and DAGAM through BERT-based fine-tuning and evaluation, deriving better results compared to the performance with original datasets.
On Data Augmentation for Extreme Multi-label Classification

Danqing Zhang,Tao Li,Haiyang Zhang,Bing Yin

DOI: https://doi.org/10.48550/arXiv.2009.10778

2020-09-23

Abstract:In this paper, we focus on data augmentation for the extreme multi-label classification (XMC) problem. One of the most challenging issues of XMC is the long tail label distribution where even strong models suffer from insufficient supervision. To mitigate such label bias, we propose a simple and effective augmentation framework and a new state-of-the-art classifier. Our augmentation framework takes advantage of the pre-trained GPT-2 model to generate label-invariant perturbations of the input texts to augment the existing training data. As a result, it present substantial improvements over baseline models. Our contributions are two-factored: (1) we introduce a new state-of-the-art classifier that uses label attention with RoBERTa and combine it with our augmentation framework for further improvement; (2) we present a broad study on how effective are different augmentation methods in the XMC task.

Computation and Language,Artificial Intelligence,Information Retrieval,Machine Learning
Effective Data Augmentation Approaches to End-to-End Task-Oriented Dialogue

Jun Quan,Deyi Xiong

DOI: https://doi.org/10.1109/ialp48816.2019.9037690

2019-01-01

Abstract:The training of task-oriented dialogue systems is often confronted with the lack of annotated data. In contrast to previous work which augments training data through expensive crowd-sourcing efforts, we propose four different automatic approaches to data augmentation at both the word and sentence level for end-to-end task-oriented dialogue and conduct an empirical study on their impact. Experimental results on the CamRest676 and KVRET datasets demonstrate that each of the four data augmentation approaches is able to obtain a significant improvement over a strong baseline in terms of Success F1 score and that the ensemble of the four approaches achieves the state-of-the-art results in the two datasets. In-depth analyses further confirm that our methods adequately increase the diversity of user utterances, which enables the end-to-end model to learn features robustly.
Text Augmentation in a Multi-Task View

Jason Wei,Chengyu Huang,Shiqi Xu,Soroush Vosoughi

DOI: https://doi.org/10.48550/arXiv.2101.05469

2021-01-14

Abstract:Traditional data augmentation aims to increase the coverage of the input distribution by generating augmented examples that strongly resemble original samples in an online fashion where augmented examples dominate training. In this paper, we propose an alternative perspective -- a multi-task view (MTV) of data augmentation -- in which the primary task trains on original examples and the auxiliary task trains on augmented examples. In MTV data augmentation, both original and augmented samples are weighted substantively during training, relaxing the constraint that augmented examples must resemble original data and thereby allowing us to apply stronger levels of augmentation. In empirical experiments using four common data augmentation techniques on three benchmark text classification datasets, we find that the MTV leads to higher and more robust performance improvements than traditional augmentation.

Computation and Language
Adversarial Word Dilution as Text Data Augmentation in Low-Resource Regime

Junfan Chen,Richong Zhang,Zheyan Luo,Chunming Hu,Yongyi Mao

DOI: https://doi.org/10.48550/arXiv.2305.09287

2023-08-09

Abstract:Data augmentation is widely used in text classification, especially in the low-resource regime where a few examples for each class are available during training. Despite the success, generating data augmentations as hard positive examples that may increase their effectiveness is under-explored. This paper proposes an Adversarial Word Dilution (AWD) method that can generate hard positive examples as text data augmentations to train the low-resource text classification model efficiently. Our idea of augmenting the text data is to dilute the embedding of strong positive words by weighted mixing with unknown-word embedding, making the augmented inputs hard to be recognized as positive by the classification model. We adversarially learn the dilution weights through a constrained min-max optimization process with the guidance of the labels. Empirical studies on three benchmark datasets show that AWD can generate more effective data augmentations and outperform the state-of-the-art text data augmentation methods. The additional analysis demonstrates that the data augmentations generated by AWD are interpretable and can flexibly extend to new examples without further training.

Computation and Language
Data Augmentation on Problem and Method Sentence Classification Task in Scientific Paper: A Mechanism Analysis Study.

Yingyi Zhang,Chengzhi Zhang

DOI: https://doi.org/10.1007/978-3-031-57867-0_2

2024-01-01

Abstract:Billions of scientific papers lead to the need to identify essential parts of the massive text. Scientific research is an activity from putting forward problems to using methods. To learn the main idea from scientific papers, we focus on extracting problem and method sentences. Annotating sentences in scientific papers is labor-intensive, resulting in the creation of small-scale datasets that limit model learning. To tackle this challenge, data augmentation has been adopted due to its ability to generate synthetic data with minor variations, thereby expanding the scale of the original training dataset. Nowadays, there are various data augmentation methods, such as those based on random word replacement or back translation. Nevertheless, their suitability for sentence classification tasks in scientific papers remains unexplored. Thus, this paper constructs two manually annotation datasets and evaluates their performance. Furthermore, this paper delves into the mechanisms underlying their effects. Previous studies have suggested that data augmentation can diminish reliance on high-frequency patterns in models. Therefore, this paper employs attention values to represent the model's dependence on words and analyzes how data augmentation methods alter the attention values of individual words within sentences. The experimental results indicate that data augmentation methods can improve the macro F 1 score in sentence classification tasks. Furthermore, data augmentation methods effectively reduce the attention values assigned to stop words, commonly used words in scientific papers, and commonly used words in method and problem sentences.
Data Augmentation For Chinese Text Classification Using Back-Translation

Jun Ma,Langlang Li

DOI: https://doi.org/10.1088/1742-6596/1651/1/012039

2020-11-01

Journal of Physics: Conference Series

Abstract:Abstract Text classification is a basic task in natural language processing. When the amount of data is insufficient, the classification accuracy will be greatly affected. We propose to use the back-translation method to expand three Chinese data sets used for text classification, and then train and predict the data sets through deep learning classification model. The results prove that using back-translation to expand the data is particularly helpful on a smaller dataset, it also can reduce the unbalanced distribution of samples and improve the classification performance.

Rethinking Data Augmentation in Text-to-text Paradigm.

Text Data Augmentation for Deep Learning

Text AutoAugment: Learning Compositional Augmentation Policy for Text Classification

SDA: Improving Text Generation with Self Data Augmentation

Data augmentation in natural language processing: a novel text generation approach for long and short text classifiers

Exploring Data Augmentation Methods on Social Media Corpora

Data Augmentation Using Virtual Word Insertion Techniques in Text Classification Tasks

AugGPT: Leveraging ChatGPT for Text Data Augmentation

GenAug: Data Augmentation for Finetuning Text Generators

On Evaluation Protocols for Data Augmentation in a Limited Data Scenario

An Empirical Survey of Data Augmentation for Limited Data Learning in NLP

Data Augmentation for Sentiment Classification with Semantic Preservation and Diversity

Improving Text Classification with Large Language Model-Based Data Augmentation

Data Augmentation Methods for Enhancing Robustness in Text Classification Tasks

DAGAM: Data Augmentation with Generation And Modification

On Data Augmentation for Extreme Multi-label Classification

Effective Data Augmentation Approaches to End-to-End Task-Oriented Dialogue

Text Augmentation in a Multi-Task View

Adversarial Word Dilution as Text Data Augmentation in Low-Resource Regime

Data Augmentation on Problem and Method Sentence Classification Task in Scientific Paper: A Mechanism Analysis Study.

Data Augmentation For Chinese Text Classification Using Back-Translation