Abstract:Deep convolutional neural network (DCNN) has led to significant breakthroughs in deep learning.However, larger models and larger datasets result in longer training times slowing down the development progress of deep learning.In this paper, following the idea of domain decomposition methods, we propose and study a new method to parallelize the training of DCNNs by decomposing and composing DCNNs.First, a global network is decomposed into several sub-networks by partitioning the width of the network (i.e., along the channel dimension) while keeping the depth constant.All the sub-networks are individually trained, in parallel without any interprocessor communication, with the corresponding decomposed samples from the input data.Then, following the idea of nonlinear preconditioning, we propose a sub-network transfer learning strategy in which the weights of the trained sub-networks are recomposed to initialize the global network, which is then trained to further adapt the parameters.Some theoretical analyses are provided to show the effectiveness of the sub-network transfer learning strategy.More precisely speaking, we prove that (1) the initialized global network can extract the feature maps learned by the sub-networks; (2) the initialization of the global network can provide an upper bound and a lower bound for the cost function and the classification accuracy with the corresponding values of the trained sub-networks.Some experiments are provided to evaluate the proposed methods.The results show that the sub-network transfer learning strategy can indeed provide good initialization and accelerate the training of the global network.Additionally, after further training, the transfer learning strategy shows almost no loss of accuracy and sometimes the accuracy is higher than if the network is initialized randomly.

Efficient Incremental Training for Deep Convolutional Neural Networks

DCCD: Reducing Neural Network Redundancy Via Distillation

A Practical Incremental Method to Train Deep CTR Models

Incremental Deep Neural Network for Visual Categories

Error-Driven Incremental Learning in Deep Convolutional Neural Network for Large-Scale Image Classification

Tree-CNN: A Hierarchical Deep Convolutional Neural Network for Incremental Learning

An efficient approach to escalate the speed of training convolution neural networks

Decomposition and Composition of Deep Convolutional Neural Networks and Training Acceleration Via Sub-Network Transfer Learning

An Efficient Strategy for Catastrophic Forgetting Reduction in Incremental Learning

Incremental learning with neural networks for computer vision: a survey

Incremental Training and Group Convolution Pruning for Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms

Efficient Incremental Learning Using Dynamic Correction Vector

EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones.

Face detection based on deep convolutional neural networks exploiting incremental facial part learning

Adaptive knowledge transfer for class incremental learning

CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks

Incremental Learning In Online Scenario

Faster ILOD: Incremental learning for object detectors based on faster RCNN

Harnessing Neural Unit Dynamics for Effective and Scalable Class-Incremental Learning

An Incremental Pruning Strategy for Fast Training of CNN Models

Energy-efficient and Robust Cumulative Training with Net2Net Transformation