Abstract:Deep learning inference is increasingly run at the edge. As the programming and system stack support becomes mature, it enables acceleration opportunities within a mobile system, where the system performance envelope is scaled up with a plethora of programmable co-processors. Thus, intelligent services designed for mobile users can choose between running inference on the CPU or any of the co-processors on the mobile system, or exploiting connected systems, such as the cloud or a nearby, locally connected system. By doing so, the services can scale out the performance and increase the energy efficiency of edge mobile systems. This gives rise to a new challenge - deciding when inference should run where. Such execution scaling decision becomes more complicated with the stochastic nature of mobile-cloud execution, where signal strength variations of the wireless networks and resource interference can significantly affect real-time inference performance and system energy efficiency. To enable accurate, energy-efficient deep learning inference at the edge, this paper proposes AutoScale. AutoScale is an adaptive and light-weight execution scaling engine built upon the custom-designed reinforcement learning algorithm. It continuously learns and selects the most energy-efficient inference execution target by taking into account characteristics of neural networks and available systems in the collaborative cloud-edge execution environment while adapting to the stochastic runtime variance. Real system implementation and evaluation, considering realistic execution scenarios, demonstrate an average of 9.8 and 1.6 times energy efficiency improvement for DNN edge inference over the baseline mobile CPU and cloud offloading, while meeting the real-time performance and accuracy requirement.

EdgeAdaptor: Online Configuration Adaption, Model Selection and Resource Provisioning for Edge DNN Inference Serving at Scale

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

Accelerating Deep Neural Network Tasks Through Edge-Device Adaptive Inference

Provisioning Edge Inference As a Service Via Online Learning.

Joint Multi-User DNN Partitioning and Computational Resource Allocation for Collaborative Edge Intelligence

Enabling Latency-Sensitive DNN Inference Via Joint Optimization of Model Surgery and Resource Allocation in Heterogeneous Edge

Scaling Up Deep Neural Network Optimization for Edge Inference

An Online Approach for DNN Model Caching and Processor Allocation in Edge Computing

Joint Optimization With DNN Partitioning and Resource Allocation in Mobile Edge Computing

Multi-Compression Scale DNN Inference Acceleration based on Cloud-Edge-End Collaboration

AutoScale: Optimizing Energy Efficiency of End-to-End Edge Inference under Stochastic Variance

On-demand Edge Inference Scheduling with Accuracy and Deadline Guarantee.

Collaborative Cloud-Edge Service Cognition Framework for DNN Configuration toward Smart IIoT

Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing

Joint Optimization of Model Partitioning and Resource Allocation for Edge Computing with Intermittently Operating Devices

ElasticDNN: On-device Neural Network Remodeling for Adapting Evolving Vision Domains at Edge

When CPN Meets AI: Resource Provisioning for Inference Query Upon Computing Power Network

Online Optimization of DNN Inference Network Utility in Collaborative Edge Computing

Hastening Stream Offloading of Inference Via Multi-Exit DNNs in Mobile Edge Computing

ADDA: Adaptive Distributed DNN Inference Acceleration in Edge Computing Environment

Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy