Abstract:Data augmentation is essential to achieve state-of-the-art performance in many deep learning applications. However, the most effective augmentation techniques become computationally prohibitive for even medium-sized datasets. To address this, we propose a rigorous technique to select subsets of data points that when augmented, closely capture the training dynamics of full data augmentation. We first show that data augmentation, modeled as additive perturbations, improves learning and generalization by relatively enlarging and perturbing the smaller singular values of the network Jacobian, while preserving its prominent directions. This prevents overfitting and enhances learning the harder to learn information. Then, we propose a framework to iteratively extract small subsets of training data that when augmented, closely capture the alignment of the fully augmented Jacobian with labels/residuals. We prove that stochastic gradient descent applied to the augmented subsets found by our approach has similar training dynamics to that of fully augmented data. Our experiments demonstrate that our method achieves 6.3x speedup on CIFAR10 and 2.2x speedup on SVHN, and outperforms the baselines by up to 10% across various subset sizes. Similarly, on TinyImageNet and ImageNet, our method beats the baselines by up to 8%, while achieving up to 3.3x speedup across various subset sizes. Finally, training on and augmenting 50% subsets using our method on a version of CIFAR10 corrupted with label noise even outperforms using the full dataset. Our code is available at: <a class="link-external link-https" href="https://github.com/tianyu139/data-efficient-augmentation" rel="external noopener nofollow">this https URL</a>

Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules

AutoAugment: Learning Augmentation Policies from Data

Automatic Data Augmentation by Learning the Deterministic Policy

DADA: Differentiable Automatic Data Augmentation

Deep AutoAugment

Data-Efficient Augmentation for Training Neural Networks

Randaugment: Practical automated data augmentation with a reduced search space

A Unified Search Framework for Data Augmentation and Neural Architecture on Small-scale Image Datasets

Learning Optimal Data Augmentation Policies via Bayesian Optimization for Image Classification Tasks

Data Augmentation Policy Search for Long-Term Forecasting

LA3: Efficient Label-Aware AutoAugment

Improving 3D Object Detection through Progressive Population Based Augmentation

Automatic Data Augmentation for Generalization in Deep Reinforcement Learning

Local Patch AutoAugment with Multi-Agent Collaboration

AdaAugment: A Tuning-Free and Adaptive Approach to Enhance Data Augmentation

LatentAugment: Dynamically Optimized Latent Probabilities of Data Augmentation

MetaAugment: Sample-Aware Data Augmentation Policy Learning

Towards Controlled Data Augmentations for Active Learning.

Active Learning with Controllable Augmentation Induced Acquisition

DAAS: Differentiable Architecture and Augmentation Policy Search

Automatic Data Augmentation for Generalization in Reinforcement Learning