Abstract:With the rapid development of Deep Learning, more and more applications on the cloud and edge tend to utilize large DNN (Deep Neural Network) models for improved task execution efficiency as well as decision-making quality. Due to memory constraints, models are commonly optimized using compression, pruning, and partitioning algorithms to become deployable onto resource-constrained devices. As the conditions in the computational platform change dynamically, the deployed optimization algorithms should accordingly adapt their solutions. To perform frequent evaluations of these solutions in a timely fashion, RMs (Regression Models) are commonly trained to predict the relevant solution quality metrics, such as the resulted DNN module inference latency, which is the focus of this paper. Existing prediction frameworks specify different RM training workflows, but none of them allow flexible configurations of the input parameters (e.g., batch size, device utilization rate) and of the selected RMs for different modules. In this paper, a deep learning module inference latency prediction framework is proposed, which i) hosts a set of customizable input parameters to train multiple different RMs per DNN module (e.g., convolutional layer) with self-generated datasets, and ii) automatically selects a set of trained RMs leading to the highest possible overall prediction accuracy, while keeping the prediction time / space consumption as low as possible. Furthermore, a new RM, namely MEDN (Multi-task Encoder-Decoder Network), is proposed as an alternative solution. Comprehensive experiment results show that MEDN is fast and lightweight, and capable of achieving the highest overall prediction accuracy and R-squared value. The Time/Space-efficient Auto-selection algorithm also manages to improve the overall accuracy by 2.5% and R-squared by 0.39%, compared to the MEDN single-selection scheme.

Towards A Flexible Accuracy-Oriented Deep Learning Module Inference Latency Prediction Framework for Adaptive Optimization Algorithms

Edge Collaborative Learning Acceleration Based on Latency Prediction

A Fine-Grained End-to-End Latency Optimization Framework for Wireless Collaborative Inference

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Accelerating Deep Neural Network Tasks Through Edge-Device Adaptive Inference

An Adaptive DNN Inference Acceleration Framework with End–edge–cloud Collaborative Computing

Accurate Deep Learning Inference Latency Prediction over Dynamic Running Mobile Devices

Enabling Latency-Sensitive DNN Inference Via Joint Optimization of Model Surgery and Resource Allocation in Heterogeneous Edge

nn-METER: Towards Accurate Latency Prediction of DNN Inference on Diverse Edge Devices.

nn-Meter: towards accurate latency prediction of deep-learning model inference on diverse edge devices

EdgeAdaptor: Online Configuration Adaption, Model Selection and Resource Provisioning for Edge DNN Inference Serving at Scale

Adaptive Deep Inference Framework for Cloud-Edge Collaboration

CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs

Energy-Aware Dynamic Neural Inference

Accelerate Intermittent Deep Inference

nn-METER

On Latency Predictors for Neural Architecture Search

Triple Wins: Boosting Accuracy, Robustness and Efficiency Together by Enabling Input-Adaptive Inference

A Progressive Subnetwork Searching Framework for Dynamic Inference

Toward the Predictability of Dynamic Real-Time DNN Inference

Collaborative Deep Neural Network Inference via Mobile Edge Computing