Abstract:Large batch distributed synchronous stochastic gradient descent (SGD) has been widely used to train deep neural networks on a distributed memory system with multi-nodes, which can leverage parallel resources to reduce the number of iterative steps and speed up the convergence of training process. However, the large-batch SGD leads to a poor test accuracy, which would counteract the benefits of large scale parallel SGD. Existing solutions for large-batch training either significantly degrade accuracy or require massive additional hyper-parameter tuning. To overcome the difficulty above, we propose a novel variable batchsize strategy. With an in-depth analysis of the different stages in the recent multi-step schedule, we find that the training process at the first stage is sensitive to the batchsize, while different batchsize do not significantly impact the latter stages. Based on this discovery, we first claim that different stages of training should use different batchsize. Hence, the variable batchsize strategy is proposed for the large scale distributed training. Furthermore, in order to turn existing hyper-parameters automatically, an auto-tuning engine is designed for the variable batchsize strategy to achieve higher testing accuracy in the extremely large batchsize cases. By using our strategy, we successfully scale the batchsize to 120K in latter stages on ImageNet-1K with ResNet50 without accuracy loss and 128K with slight accuracy loss. The experimental evaluation on 2048 GPUs shows that the variable batchsize strategy with our auto-tuning engine could complete the training of ResNet-50 in 25 minutes. Furthermore, the new strategy successfully decreases the number of parameter updates by about 1.7 times compared with Facebook's multi-step schedule.

Determination of Optimal Batch Size of Deep Learning Models with Time Series Data

On Mini-Batch Training with Varying Length Time Series

Better Batch for Deep Probabilistic Time Series Forecasting

Time Series Forecasting (TSF) Using Various Deep Learning Models

A Time Series Forecasting Model Selection Framework using CNN and Data Augmentation for Small Sample Data

AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks

Scaling Law for Time Series Forecasting

Long Input Sequence Network for Long Time Series Forecasting

Do We Really Need Deep Learning Models for Time Series Forecasting?

Enabling Large Batch Size Training for DNN Models Beyond the Memory Limit While Maintaining Performance

A Variable Batch Size Strategy for Large Scale Distributed DNN Training

Unlocking optimal batch size schedules using continuous-time control and perturbation theory

A new perspective for understanding generalization gap of deep neural networks trained with large batch sizes

Accelerating Deep Learning with Fixed Time Budget

Time series forecasting and anomaly detection using deep learning

A Survey of Deep Learning and Foundation Models for Time Series Forecasting

Impact of Recurrent Neural Networks and Deep Learning Frameworks on Real-time Lightweight Time Series Anomaly Detection

How Does Critical Batch Size Scale in Pre-training?

Sparse Deep Learning for Time Series Data: Theory and Applications

Data Scaling Effect of Deep Learning in Financial Time Series Forecasting

A Review of Deep Learning Models for Time Series Prediction