Abstract:International Journal of Software Engineering and Knowledge Engineering, Ahead of Print. In mobile edge computing environment, intelligent inference services driven by DNN are highly sensitive to latency. Recently, collaborative inference between User Devices and Edge Servers (ESs) based on Deep Neural Networks (DNN) partition has achieved success in service acceleration. However, most of the existing collaborative acceleration schemes are partitioned for a single DNN inference task, which cannot quickly make partition decisions for a set of concurrent inference tasks, and often sacrifice inference accuracy. In addition, due to the limited resources of ESs, there is resource competition among concurrent requests, which makes the partitioned tasks cannot be offloaded to ESs in time for processing. Therefore, designing an efficient offloading scheme becomes essential. The task offloading schemes based on deep reinforcement learning can solve complex decision-making problems in high-dimensional state space, but they have problems such as insufficient sample diversity and easily falling into local optimum. In this paper, a Collaborative Inference Acceleration Scheme integrating DNN Partitioning and Task Offloading (CIAS-PnO) is proposed. First, while ensuring inference accuracy, the Collaborative DNN Layer Partitioning (CDLP) algorithm is designed with the goal of optimal latency. CDLP can reduce the problem scale of concurrent inference tasks partition by pruning operation and determine the partition decisions in time. Then, the Distributed Soft Actor-Critic (SAC)-based Partition Task Offloading algorithm (DSACO) is designed. DSACO supports SAC Agents to explore samples in parallel and share learning experiences, and uses the automatic entropy adjustment mechanism to improve the exploration efficiency of Agents, so as to avoid falling into local optimum and achieve efficient offloading of partition tasks. Experimental results on DNN benchmarks show that compared with the baseline acceleration schemes, CIAS-PnO achieves more than 19.8% acceleration performance improvement, and has higher convergence performance and task success rate.

CPN Meets Learning: Online Scheduling for Inference Service in Computing Power Network

When CPN Meets AI: Resource Provisioning for Inference Query Upon Computing Power Network

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

Online Optimization of DNN Inference Network Utility in Collaborative Edge Computing

Provisioning Edge Inference As a Service Via Online Learning.

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Joint Energy-Computation Management for Electric Vehicles under Coordination of Power Distribution Networks and Computing Power Networks

Heterogeneous Resource Scheduling in Computing Power Networks: A Method Based on Improved D3QN

An Online Approach for DNN Model Caching and Processor Allocation in Edge Computing

Collaborative Inference Acceleration Integrating DNN Partitioning and Task Offloading in Mobile Edge Computing

QCQP-Net: Reliably Learning Feasible Alternating Current Optimal Power Flow Solutions Under Constraints

On-demand Edge Inference Scheduling with Accuracy and Deadline Guarantee.

Collaborative Inference for Deep Neural Networks in Edge Environments

Online Learning for Orchestration of Inference in Multi-User End-Edge-Cloud Networks

Automated Runtime-Aware Scheduling for Multi-Tenant DNN Inference on GPU

Learning to Dynamically Select Cost Optimal Schedulers in Cloud Computing Environments

Dynamic Distributed Edge Resource Provisioning Via Online Learning Across Timescales.

CoFB: latency-constrained co-scheduling of flows and batches for deep learning inference service on the CPU–GPU system

CAP: Communication-aware Automated Parallelization for Deep Learning Inference on CMP Architectures

Ultra-Low-Latency Distributed Deep Neural Network over Hierarchical Mobile Networks

The Effect of Scheduling and Preemption on the Efficiency of LLM Inference Serving