Abstract:Mobile Edge Computing (MEC) has emerged as a promising paradigm catering to overwhelming explosions of mobile applications, by offloading compute-intensive tasks to MEC networks for processing. The surging of deep learning brings new vigor and vitality to shape the prospect of intelligent Internet of Things (IoT), and edge intelligence arises to provision real-time deep neural network (DNN) inference services for users. To accelerate the processing of the DNN inference of a user request in an MEC network, the DNN inference model usually can be partitioned into two connected parts: one part is processed in the local IoT device of the request, and another part is processed in a cloudlet (edge server) in the MEC network. Also, the DNN inference can be further accelerated by allocating multiple threads of the cloudlet to which the request is assigned. In this paper, we study a novel delay-aware DNN inference throughput maximization problem with the aim to maximize the number of delay-aware DNN service requests admitted, by accelerating each DNN inference through jointly exploring DNN partitioning and multi-thread execution parallelism. Specifically, we consider the problem under both offline and online request arrival settings: a set of DNN inference requests is given in advance, and a sequence of DNN inference requests arrives one by one without the knowledge of future arrivals, respectively. We first show that the defined problems are NP-hard. We then devise a novel constant approximation algorithm for the problem under the offline setting. We also propose an online algorithm with a provable competitive ratio for the problem under the online setting. We finally evaluate the performance of the proposed algorithms through experimental simulations. Experimental results demonstrate that the proposed algorithms are promising

On-demand Edge Inference Scheduling with Accuracy and Deadline Guarantee.

Dynamic Batching and Early-Exiting for Accurate and Timely Edge Inference

Adaptive Scheduling for Edge-Assisted DNN Serving

Scheduling Real-time Deep Learning Services as Imprecise Computations

BCEdge: SLO-Aware DNN Inference Services with Adaptive Batching on Edge Platforms

A DNN inference acceleration algorithm combining model partition and task allocation in heterogeneous edge computing system

BCEdge: SLO-Aware DNN Inference Services With Adaptive Batch-Concurrent Scheduling on Edge Devices

Computation Offloading and Task Scheduling for DNN-Based Applications in Cloud-Edge Computing

Online Scheduling of CPU-NPU Co-inference for Edge AI Tasks.

Distributed Assignment With Load Balancing for DNN Inference at the Edge

Online Approximation Scheme for Scheduling Heterogeneous Utility Jobs in Edge Computing

A QoE-Aware Service-Enhancement Strategy for Edge Artificial Intelligence Applications

When CPN Meets AI: Resource Provisioning for Inference Query Upon Computing Power Network

Enabling Latency-Sensitive DNN Inference Via Joint Optimization of Model Surgery and Resource Allocation in Heterogeneous Edge

Accelerating Deep Neural Network Tasks Through Edge-Device Adaptive Inference

Reliability-Aware Online Scheduling for DNN Inference Tasks in Mobile-Edge Computing

Throughput Maximization of Delay-Aware DNN Inference in Edge Computing by Exploring DNN Model Partitioning and Inference Parallelism

Provisioning Edge Inference As a Service Via Online Learning.

DNN Inference Task Offloading Based on Distributed Soft Actor-Critic in Mobile Edge Computing.

Hastening Stream Offloading of Inference Via Multi-Exit DNNs in Mobile Edge Computing