Abstract:As deep neural networks (DNNs) are increasingly used in a broad spectrum of edge intelligent applications, it is often necessary to provide multi-DNN model inference services, and it is nontrivial for edge inference platforms to simultaneously deliver high-throughput and low-latency. Such edge devices with multi-DNN model pose new challenges for scheduler designs. First, edge devices should be capable of efficiently scheduling multiple heterogeneous DNN models in order to optimize system utilization. Second, each inference request may have different service level objectives (SLOs) to improve quality of service (QoS). To address these challenges, this paper proposes BCEdge, a novel learning-based scheduling framework that incorporates adaptive batching and concurrent execution of DNN inference services on edge devices. We first propose a shared memory policy to reduce the memory contention among multiple DNN models. Afterwards, a utility function is defined to evaluate the trade-off between throughput and latency. The scheduler in BCEdge leverages branch-based deep reinforcement learning (DRL) to maximize utility by 1) optimizing batch size, 2) automatically identifying the number of concurrent instances for multiple DNN models, and 3) determining the shared memory configuration among multiple DNN models. Besides, the lightweight DNN-based prediction model in BCEdge can achieve SLO awareness by reducing the performance interference among multiple DNN models. Our prototype implemented on various edge devices illustrates that BCEdge enhances utility by up to 37.6% and reduces memory usage by up to 38% on average, compared to state-of-the-art schemes, while maintaining the SLO violation rate within 5%.

BCEdge: SLO-Aware DNN Inference Services With Adaptive Batch-Concurrent Scheduling on Edge Devices

BCEdge: SLO-Aware DNN Inference Services with Adaptive Batching on Edge Platforms

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

Octopus: SLO-Aware Progressive Inference Serving via Deep Reinforcement Learning in Multi-tenant Edge Cluster

Ace-Sniper: Cloud-Edge Collaborative Scheduling Framework With DNN Inference Latency Modeling on Heterogeneous Devices

Adaptive Scheduling for Edge-Assisted DNN Serving

Dynamic Batching and Early-Exiting for Accurate and Timely Edge Inference

An Adaptive Task Migration Scheduling Approach for Edge-Cloud Collaborative Inference

On-demand Edge Inference Scheduling with Accuracy and Deadline Guarantee.

Enabling Latency-Sensitive DNN Inference Via Joint Optimization of Model Surgery and Resource Allocation in Heterogeneous Edge

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy

Hastening Stream Offloading of Inference Via Multi-Exit DNNs in Mobile Edge Computing

CoEdge: Cooperative DNN Inference With Adaptive Workload Partitioning Over Heterogeneous Edge Devices

Scheduling Inference Workloads on Distributed Edge Clusters with Reinforcement Learning

An Online Approach for DNN Model Caching and Processor Allocation in Edge Computing

ECSRL: A Learning-Based Scheduling Framework for AI Workloads in Heterogeneous Edge-Cloud Systems

Collaborative Inference for Deep Neural Networks in Edge Environments

Joint Optimization of Device Placement and Model Partitioning for Cooperative DNN Inference in Heterogeneous Edge Computing

A Co-Scheduling Framework for DNN Models on Mobile and Edge Devices with Heterogeneous Hardware

Niagara: Scheduling DNN Inference Services on Heterogeneous Edge Processors.