Abstract:As deep neural networks (DNNs) are increasingly used in a broad spectrum of edge intelligent applications, it is often necessary to provide multi-DNN model inference services, and it is nontrivial for edge inference platforms to simultaneously deliver high-throughput and low-latency. Such edge devices with multi-DNN model pose new challenges for scheduler designs. First, edge devices should be capable of efficiently scheduling multiple heterogeneous DNN models in order to optimize system utilization. Second, each inference request may have different service level objectives (SLOs) to improve quality of service (QoS). To address these challenges, this paper proposes BCEdge, a novel learning-based scheduling framework that incorporates adaptive batching and concurrent execution of DNN inference services on edge devices. We first propose a shared memory policy to reduce the memory contention among multiple DNN models. Afterwards, a utility function is defined to evaluate the trade-off between throughput and latency. The scheduler in BCEdge leverages branch-based deep reinforcement learning (DRL) to maximize utility by 1) optimizing batch size, 2) automatically identifying the number of concurrent instances for multiple DNN models, and 3) determining the shared memory configuration among multiple DNN models. Besides, the lightweight DNN-based prediction model in BCEdge can achieve SLO awareness by reducing the performance interference among multiple DNN models. Our prototype implemented on various edge devices illustrates that BCEdge enhances utility by up to 37.6% and reduces memory usage by up to 38% on average, compared to state-of-the-art schemes, while maintaining the SLO violation rate within 5%.

Kalmia: A Heterogeneous QoS-aware Scheduling Framework for DNN Tasks on Edge Servers

On-demand Edge Inference Scheduling with Accuracy and Deadline Guarantee.

BCEdge: SLO-Aware DNN Inference Services With Adaptive Batch-Concurrent Scheduling on Edge Devices

Adaptive Scheduling for Edge-Assisted DNN Serving

BCEdge: SLO-Aware DNN Inference Services with Adaptive Batching on Edge Platforms

Intelligent Task Dispatching and Scheduling Using a Deep Q-Network in a Cluster Edge Computing System

Reinforcement Learning Based Online Scheduling of Multiple Workflows in Edge Environment

Joint Task Offloading and Resource Allocation for Quality-Aware Edge-Assisted Machine Learning Task Inference

Dynamic Batching and Early-Exiting for Accurate and Timely Edge Inference

Hybrid Workflow Scheduling on Edge Cloud Computing Systems

Ace-Sniper: Cloud-Edge Collaborative Scheduling Framework With DNN Inference Latency Modeling on Heterogeneous Devices

Joint Batching and Scheduling for High-Throughput Multiuser Edge AI with Asynchronous Task Arrivals

QoE-driven computation offloading for Edge Computing

Preemptive Scheduling for Distributed Machine Learning Jobs in Edge-Cloud Networks

A Co-Scheduling Framework for DNN Models on Mobile and Edge Devices with Heterogeneous Hardware

QoS-Aware Task Scheduling in Cloud-Edge Environment

MCDS: AI Augmented Workflow Scheduling in Mobile Edge Cloud Computing Systems

Niagara: Scheduling DNN Inference Services on Heterogeneous Edge Processors.

Cooperative Computation Offloading and Dynamic Task Scheduling in Edge Computing

Optimizing Task-Specific Timeliness With Edge-Assisted Scheduling for Status Update

A Dynamic Distributed Scheduler for Computing on the Edge