Abstract:Mobile Edge Computing (MEC) has emerged as a promising paradigm catering to overwhelming explosions of mobile applications, by offloading compute-intensive tasks to MEC networks for processing. The surging of deep learning brings new vigor and vitality to shape the prospect of intelligent Internet of Things (IoT), and edge intelligence arises to provision real-time deep neural network (DNN) inference services for users. To accelerate the processing of the DNN inference of a user request in an MEC network, the DNN inference model usually can be partitioned into two connected parts: one part is processed in the local IoT device of the request, and another part is processed in a cloudlet (edge server) in the MEC network. Also, the DNN inference can be further accelerated by allocating multiple threads of the cloudlet to which the request is assigned. In this paper, we study a novel delay-aware DNN inference throughput maximization problem with the aim to maximize the number of delay-aware DNN service requests admitted, by accelerating each DNN inference through jointly exploring DNN partitioning and multi-thread execution parallelism. Specifically, we consider the problem under both offline and online request arrival settings: a set of DNN inference requests is given in advance, and a sequence of DNN inference requests arrives one by one without the knowledge of future arrivals, respectively. We first show that the defined problems are NP-hard. We then devise a novel constant approximation algorithm for the problem under the offline setting. We also propose an online algorithm with a provable competitive ratio for the problem under the online setting. We finally evaluate the performance of the proposed algorithms through experimental simulations. Experimental results demonstrate that the proposed algorithms are promising

An Online Approach for DNN Model Caching and Processor Allocation in Edge Computing

Efficient Partitioning and Communication Scheme-Based Distributed Edge Computing to Accelerate Deep Neural Network

A DNN inference acceleration algorithm combining model partition and task allocation in heterogeneous edge computing system

Energy-Efficient Offloading for DNN-Based Smart IoT Systems in Cloud-Edge Environments.

Enabling Latency-Sensitive DNN Inference Via Joint Optimization of Model Surgery and Resource Allocation in Heterogeneous Edge

Cost Efficient Offloading Strategy for DNN-Based Applications in Edge-Cloud Environment

Online Optimization of DNN Inference Network Utility in Collaborative Edge Computing

Throughput Maximization of Delay-Aware DNN Inference in Edge Computing by Exploring DNN Model Partitioning and Inference Parallelism

Joint Optimization With DNN Partitioning and Resource Allocation in Mobile Edge Computing

Computation Offloading and Task Scheduling for DNN-Based Applications in Cloud-Edge Computing

Delay-Aware DNN Inference Throughput Maximization in Edge Computing Via Jointly Exploring Partitioning and Parallelism

ADDA: Adaptive Distributed DNN Inference Acceleration in Edge Computing Environment

Model Parallelism Optimization for Distributed DNN Inference on Edge Devices.

Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing

Joint DNN Partition and Resource Allocation Optimization for Energy-Constrained Hierarchical Edge-Cloud Systems

Modeling of Deep Neural Network (DNN) Placement and Inference in Edge Computing

An Adaptive DNN Inference Acceleration Framework with End–edge–cloud Collaborative Computing

Accelerating Deep Neural Network Tasks Through Edge-Device Adaptive Inference

Hastening Stream Offloading of Inference Via Multi-Exit DNNs in Mobile Edge Computing

DECC: Delay-Aware Edge-Cloud Collaboration for Accelerating DNN Inference

Multi-user Co-inference with Batch Processing Capable Edge Server