Abstract:As a key technology of enabling Artificial Intelligence (AI) applications in 5G era, Deep Neural Networks (DNNs) have quickly attracted widespread attention. However, it is challenging to run computation-intensive DNN-based tasks on mobile devices due to the limited computation resources. What's worse, traditional cloud-assisted DNN inference is heavily hindered by the significant wide-area network latency, leading to poor real-time performance as well as low quality of user experience. To address these challenges, in this paper, we propose Edgent, a framework that leverages edge computing for DNN collaborative inference through device-edge synergy. Edgent exploits two design knobs: (1) DNN partitioning that adaptively partitions computation between device and edge for purpose of coordinating the powerful cloud resource and the proximal edge resource for real-time DNN inference; (2) DNN right-sizing that further reduces computing latency via early exiting inference at an appropriate intermediate DNN layer. In addition, considering the potential network fluctuation in real-world deployment, Edgent is properly design to specialize for both static and dynamic network environment. Specifically, in a static environment where the bandwidth changes slowly, Edgent derives the best configurations with the assist of regression-based prediction models, while in a dynamic environment where the bandwidth varies dramatically, Edgent generates the best execution plan through the online change point detection algorithm that maps the current bandwidth state to the optimal configuration. We implement Edgent prototype based on the Raspberry Pi and the desktop PC and the extensive experimental evaluations demonstrate Edgent's effectiveness in enabling on-demand low-latency edge intelligence.

Online Scheduling of CPU-NPU Co-inference for Edge AI Tasks.

AI-oriented Workload Allocation for Cloud-Edge Computing.

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

Online Resource Provisioning and Batch Scheduling for AIoT Inference Serving in an XPU Edge Cloud

EdgeNN: Efficient Neural Network Inference for CPU-GPU Integrated Edge Devices.

Multi-user Co-inference with Batch Processing Capable Edge Server

CoEdge: Cooperative DNN Inference With Adaptive Workload Partitioning Over Heterogeneous Edge Devices

Ace-Sniper: Cloud-Edge Collaborative Scheduling Framework With DNN Inference Latency Modeling on Heterogeneous Devices

On-demand Edge Inference Scheduling with Accuracy and Deadline Guarantee.

Accelerating Deep Neural Network Tasks Through Edge-Device Adaptive Inference

An Online Approach for DNN Model Caching and Processor Allocation in Edge Computing

Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing

Joint Optimization of Device Placement and Model Partitioning for Cooperative DNN Inference in Heterogeneous Edge Computing

Adaptive Scheduling for Edge-Assisted DNN Serving

A Novel Adaptive Computation Offloading Strategy for Collaborative DNN Inference over Edge Devices.

A DNN inference acceleration algorithm combining model partition and task allocation in heterogeneous edge computing system

CPN Meets Learning: Online Scheduling for Inference Service in Computing Power Network

Learning to Optimize Workflow Scheduling for an Edge–Cloud Computing Environment

Optimized co-scheduling of mixed-precision neural network accelerator for real-time multitasking applications

Dynamic Batching and Early-Exiting for Accurate and Timely Edge Inference

Conflict-Resilient Incremental Offloading of Deep Neural Networks to the Edge of Smart Environment