Abstract:Deep neural networks (DNNs) often rely on massive labelled data for training, which is inaccessible in many applications. Data augmentation (DA) tackles data scarcity by creating new labelled data from available ones. Different DA methods have different mechanisms and therefore using their generated labelled data for DNN training may help improving DNN's generalisation to different degrees. Combining multiple DA methods, namely multi-DA, for DNN training, provides a way to boost generalisation. Among existing multi-DA based DNN training methods, those relying on knowledge distillation (KD) have received great attention. They leverage knowledge transfer to utilise the labelled data sets created by multiple DA methods instead of directly combining them for training DNNs. However, existing KD-based methods can only utilise certain types of DA methods, incapable of utilising the advantages of arbitrary DA methods. We propose a general multi-DA based DNN training framework capable to use arbitrary DA methods. To train a DNN, our framework replicates a certain portion in the latter part of the DNN into multiple copies, leading to multiple DNNs with shared blocks in their former parts and independent blocks in their latter parts. Each of these DNNs is associated with a unique DA and a newly devised loss that allows comprehensively learning from the data generated by all DA methods and the outputs from all DNNs in an online and adaptive way. The overall loss, i.e., the sum of each DNN's loss, is used for training the DNN. Eventually, one of the DNNs with the best validation performance is chosen for inference. We implement the proposed framework by using three distinct DA methods and apply it for training representative DNNs. Experiments on the popular benchmarks of image classification demonstrate the superiority of our method to several existing single-DA and multi-DA based training methods.

Training by Pairing Correlated Samples Improves Deep Network Generalization

DCCD: Reducing Neural Network Redundancy Via Distillation

Gradient Agreement Hinders the Memorization of Noisy Labels

Deep Companion Learning: Enhancing Generalization Through Historical Consistency

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Can we learn better with hard samples?

Learning Generalizable Models via Disentangling Spurious and Enhancing Potential Correlations

Growing Deep Neural Network Considering with Similarity between Neurons

Leveraging Class Similarity to Improve Deep Neural Network Robustness

DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networks

Compare More Nuanced:Pairwise Alignment Bilinear Network For Few-shot Fine-grained Learning

On the Versatile Uses of Partial Distance Correlation in Deep Learning

Structure-Based Training: A Training Method Aimed at Pixel Errors for a Correlation-Coefficient-Based Neural Network

A Novel DNN Training Framework via Data Sampling and Multi-Task Optimization

Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?

Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate

Enhancing Neural Training via a Correlated Dynamics Model

A General Multiple Data Augmentation Based Framework for Training Deep Neural Networks

Nonlinear Collaborative Scheme for Deep Neural Networks.

Deeply-supervised Knowledge Synergy