Abstract:Conventional DNN training paradigms typically rely on one training set and one validation set, obtained by partitioning an annotated dataset used for training, namely gross training set, in a certain way. The training set is used for training the model while the validation set is used to estimate the generalization performance of the trained model as the training proceeds to avoid over-fitting. There exist two major issues in this paradigm. Firstly, the validation set may hardly guarantee an unbiased estimate of generalization performance due to potential mismatching with test data. Secondly, training a DNN corresponds to solve a complex optimization problem, which is prone to getting trapped into inferior local optima and thus leads to undesired training results. To address these issues, we propose a novel DNN training framework. It generates multiple pairs of training and validation sets from the gross training set via random splitting, trains a DNN model of a pre-specified structure on each pair while making the useful knowledge (e.g., promising network parameters) obtained from one model training process to be transferred to other model training processes via multi-task optimization, and outputs the best, among all trained models, which has the overall best performance across the validation sets from all pairs. The knowledge transfer mechanism featured in this new framework can not only enhance training effectiveness by helping the model training process to escape from local optima but also improve on generalization performance via implicit regularization imposed on one model training process from other model training processes. We implement the proposed framework, parallelize the implementation on a GPU cluster, and apply it to train several widely used DNN models. Experimental results demonstrate the superiority of the proposed framework over the conventional training paradigm.

TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism

Optimal distributed parallel algorithms for deep learning framework Tensorflow

Optimizing DNN Compilation for Distributed Training with Joint OP and Tensor Fusion

TopoOpt: Co-optimizing Network Topology and Parallelization Strategy for Distributed Training Jobs

TAP: Accelerating Large-Scale DNN Training Through Tensor Automatic Parallelisation

NeutronTP: Load-Balanced Distributed Full-Graph GNN Training with Tensor Parallelism

MP-DPS: Adaptive Distributed Training for Deep Learning Based on Node Merging and Path Prediction

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Distributed Training Optimization for DCU

Joint Dynamic Data and Model Parallelism for Distributed Training of DNNs over Heterogeneous Infrastructure

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

Training Acceleration for Deep Neural Networks: A Hybrid Parallelization Strategy

A Novel DNN Training Framework via Data Sampling and Multi-Task Optimization

Optimizing Task Placement and Online Scheduling for Distributed GNN Training Acceleration

A Multidimensional Communication Scheduling Method for Hybrid Parallel DNN Training

Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities

Adaptive Distributed Parallel Training Method for a Deep Learning Model Based on Dynamic Critical Paths of DAG

Proteus: Simulating the Performance of Distributed DNN Training

dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training

Hybrid-Parallel: Achieving High Performance and Energy Efficient Distributed Inference on Robots

RTP: Rethinking Tensor Parallelism with Memory Deduplication