Abstract:Deep neural networks (DNNs) thrive in recent years, wherein batch normalization (BN) plays an indispensable role. However, it has been observed that BN is costly due to the huge reduction and elementwise operations that are hard to be executed in parallel, which heavily reduces the training speed. To address this issue, in this article, we propose a methodology to alleviate the BN's cost by using only a few sampled or generated data for mean and variance estimation at each iteration. The key challenge to reach this goal is how to achieve a satisfactory balance between normalization effectiveness and execution efficiency. We identify that the effectiveness expects less data correlation in sampling while the efficiency expects more regular execution patterns. To this end, we design two categories of approach: sampling or creating a few uncorrelated data for statistics' estimation with certain strategy constraints. The former includes "batch sampling (BS)" that randomly selects a few samples from each batch and "feature sampling (FS)" that randomly selects a small patch from each feature map of all samples, and the latter is "virtual data set normalization (VDN)" that generates a few synthetic random samples to directly create uncorrelated data for statistics' estimation. Accordingly, multiway strategies are designed to reduce the data correlation for accurate estimation and optimize the execution pattern for running acceleration in the meantime. The proposed methods are comprehensively evaluated on various DNN models, where the loss of model accuracy and the convergence rate are negligible. Without the support of any specialized libraries, 1.98× BN layer acceleration and 23.2% overall training speedup can be practically achieved on modern GPUs. Furthermore, our methods demonstrate powerful performance when solving the well-known "micro-BN" problem in the case of a tiny batch size. This article provides a promising solution for the efficient training of high-performance DNNs.

An Improved Mini-Batching Technique: Sample-and-Learn

Can we learn better with hard samples?

Minibatch training of neural network ensembles via trajectory sampling

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Dynamic Batch Scheduler for Efficient Neural Networks Training

One Backward from Ten Forward, Subsampling for Large-Scale Deep Learning

Towards Better Generalization of Deep Neural Networks via Non-Typicality Sampling Scheme

Batch Normalization Sampling.

Carpe Diem, Seize the Samples Uncertain "at the Moment" for Adaptive Batch Selection

Influence-Based Mini-Batching for Graph Neural Networks

Batching for Green AI -- An Exploratory Study on Inference

Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches

Subbatch learning method for BP neural networks

Accelerating Minibatch Stochastic Gradient Descent Using Typicality Sampling

AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks

Adaptive Sampling for Deep Learning via Efficient Nonparametric Proxies

Effective and Efficient Batch Normalization Using a Few Uncorrelated Data for Statistics Estimation

BatchSampler: Sampling Mini-Batches for Contrastive Learning in Vision, Language, and Graphs

On Sampling Strategies for Neural Network-based Collaborative Filtering

Learn2Mix: Training Neural Networks Using Adaptive Data Integration

Analysis, Design and Evaluation of a High-Performance Stochastic Multilayer Perceptron: from Mini-Batch Training to Inference