Abstract:Deep neural networks provide state-of-the-art accuracy for vision tasks but they require significant resources for training. Thus, they are trained on cloud servers far from the edge devices that acquire the data. This issue increases communication cost, runtime and privacy concerns. In this study, a novel hierarchical training method for deep neural networks is proposed that uses early exits in a divided architecture between edge and cloud workers to reduce the communication cost, training runtime and privacy concerns. The method proposes a brand-new use case for early exits to separate the backward pass of neural networks between the edge and the cloud during the training phase. We address the issues of most available methods that due to the sequential nature of the training phase, cannot train the levels of hierarchy simultaneously or they do it with the cost of compromising privacy. In contrast, our method can use both edge and cloud workers simultaneously, does not share the raw input data with the cloud and does not require communication during the backward pass. Several simulations and on-device experiments for different neural network architectures demonstrate the effectiveness of this method. It is shown that the proposed method reduces the training runtime for VGG-16 and ResNet-18 architectures by 29% and 61% in CIFAR-10 classification and by 25% and 81% in Tiny ImageNet classification when the communication with the cloud is done over a low bit rate channel. This gain in the runtime is achieved whilst the accuracy drop is negligible. This method is advantageous for online learning of high-accuracy deep neural networks on sensor-holding low-resource devices such as mobile phones or robots as a part of an edge-cloud system, making them more flexible in facing new tasks and classes of data.

QuickNets: Saving Training and Preventing Overconfidence in Early-Exit Neural Architectures

Elastic DNN Inference with Unpredictable Exit in Edge Computing

Why should we add early exits to neural networks?

Shallow-Deep Networks: Understanding and Mitigating Network Overthinking

RAPNet: Resolution-Adaptive and Predictive Early Exit Network for Efficient Image Recognition

Rethinking the Usage of Batch Normalization and Dropout in the Training of Deep Neural Networks

Early-Exit with Class Exclusion for Efficient Inference of Neural Networks

Fast yet Safe: Early-Exiting with Risk Control

Hierarchical Training of Deep Neural Networks Using Early Exiting

Zero Time Waste: Recycling Predictions in Early Exit Neural Networks

Improving the Accuracy of Early Exits in Multi-Exit Architectures Via Curriculum Learning.

Pruning Early Exit Networks

Accelerating Neural Network Training: A Brief Review

Consistency Training of Multi-exit Architectures for Sensor Data

Joint or Disjoint: Mixing Training Regimes for Early-Exit Models

Binary Early-Exit Network for Adaptive Inference on Low-Resource Devices

Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNN

DynExit: A Dynamic Early-Exit Strategy for Deep Residual Networks

NetCut: Real-Time DNN Inference Using Layer Removal

Dynamic nsNet2: Efficient Deep Noise Suppression with Early Exiting

Energy-efficient and Robust Cumulative Training with Net2Net Transformation