Abstract:With the rapid development of Deep Learning, more and more applications on the cloud and edge tend to utilize large DNN (Deep Neural Network) models for improved task execution efficiency as well as decision-making quality. Due to memory constraints, models are commonly optimized using compression, pruning, and partitioning algorithms to become deployable onto resource-constrained devices. As the conditions in the computational platform change dynamically, the deployed optimization algorithms should accordingly adapt their solutions. To perform frequent evaluations of these solutions in a timely fashion, RMs (Regression Models) are commonly trained to predict the relevant solution quality metrics, such as the resulted DNN module inference latency, which is the focus of this paper. Existing prediction frameworks specify different RM training workflows, but none of them allow flexible configurations of the input parameters (e.g., batch size, device utilization rate) and of the selected RMs for different modules. In this paper, a deep learning module inference latency prediction framework is proposed, which i) hosts a set of customizable input parameters to train multiple different RMs per DNN module (e.g., convolutional layer) with self-generated datasets, and ii) automatically selects a set of trained RMs leading to the highest possible overall prediction accuracy, while keeping the prediction time / space consumption as low as possible. Furthermore, a new RM, namely MEDN (Multi-task Encoder-Decoder Network), is proposed as an alternative solution. Comprehensive experiment results show that MEDN is fast and lightweight, and capable of achieving the highest overall prediction accuracy and R-squared value. The Time/Space-efficient Auto-selection algorithm also manages to improve the overall accuracy by 2.5% and R-squared by 0.39%, compared to the MEDN single-selection scheme.

Sectum: Accurate Latency Prediction for TEE-hosted Deep Learning Inference

nn-Meter: towards accurate latency prediction of deep-learning model inference on diverse edge devices

Edge Collaborative Learning Acceleration Based on Latency Prediction

nn-METER: Towards Accurate Latency Prediction of DNN Inference on Diverse Edge Devices.

Accurate Deep Learning Inference Latency Prediction over Dynamic Running Mobile Devices

nn-METER

Memory-Efficient and Secure DNN Inference on TrustZone-enabled Consumer IoT Devices

Towards A Flexible Accuracy-Oriented Deep Learning Module Inference Latency Prediction Framework for Adaptive Optimization Algorithms

PerfSAGE: Generalized Inference Performance Predictor for Arbitrary Deep Learning Models on Edge Devices

SecoInfer: Secure DNN End-Edge Collaborative Inference Framework Optimizing Privacy and Latency

Occlumency

No Privacy Left Outside: on the (In-)Security of TEE-Shielded DNN Partition for On-Device ML

CDMPP: A Device-Model Agnostic Framework for Latency Prediction of Tensor Programs

Penetralium: Privacy-preserving and memory-efficient neural network inference at the edge

STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining

Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloads

An Adaptive DNN Inference Acceleration Framework with End–edge–cloud Collaborative Computing

TEESlice: Slicing DNN Models for Secure and Efficient Deployment

Latency-aware Spatial-wise Dynamic Networks.

A Causal Long Short-Term Memory Sequence to Sequence Model for TEC Prediction Using GNSS Observations

Interference-Aware Latency Prediction with Kernels for Deep Neural Network