Abstract:Nowadays, the significance of multi-tenant deep neural networks (DNNs) has grown exponentially, particularly for cloud providers who execute multiple DNN models on one server to fulfill the users' requirements while reducing the computational overhead. To satisfy the heavy computation requirement of multi-tenant DNNs, a feasible approach is to establish a multi-core accelerator housing multiple sub-accelerators. Although many researchers have achieved a certain success by designing either offline schedulers for heterogeneous accelerators or real-time schedulers for homogeneous accelerators, they fail to schedule multi-tenant DNNs to both homogeneous and heterogeneous multi-core accelerators in real time given the large search space and restricted overhead constraint. In this paper, we propose RealArch, a novel real-time scheduler that efficiently schedules multi-tenant DNNs to both homogeneous and heterogeneous multi-core accelerators in real-time. The key idea of RealArch is to quickly find the minimal latency of mapping multi-tenant DNNs to sub-accelerators, considering the occupation of DRAM, sub-accelerators, and buffers. To support the key idea, we first establish lightweight estimation models for multiple sub-accelerators to evaluate the Data Movement (DM) and Execution (EX) time when mapping a layer to them. Then, we design a real-time scheduling algorithm to compute and select the mapping solution with minimal latency. Finally, we build a low-cost hardware scheduler to perform the estimation models and the real-time scheduling algorithm. Extensive experiment results verify that RealArch can exceed the baseline Round Robin scheduling algorithm and two state-of-the-art schedulers AI-MT and MAGMA with acceptable hardware overhead.

Relation-oriented Resource Allocation for Multi-Accelerator Systems

Joint User Scheduling and Resource Allocation for Millimeter Wave Systems Relying on Adaptive-Resolution ADCs

A Dynamic Resource Allocation Optimization for SMT Processors

Multi-Amdahl: Optimal Resource Sharing with Multiple Program Execution Segments

MultiAmdahl: Optimal Resource Allocation in Heterogeneous Architectures

A2: Towards Accelerator Level Parallelism for Autonomous Micromobility Systems

SCAR: Scheduling Multi-Model AI Workloads on Heterogeneous Multi-Chiplet Module Accelerators

Priority-Aware Near-Optimal Scheduling for Heterogeneous Multi-Core Systems with Specialized Accelerators

Mitigating inefficient task mappings with an Adaptive Resource-Moldable Scheduler (ARMS)

Automatic Mapping of Heterogeneous DNN Models on Adaptive Multi-Accelerator Systems

MARS: Exploiting Multi-Level Parallelism for DNN Workloads on Adaptive Multi-Accelerator Systems

Decentralized Resource Allocation for Multi-Radar Systems Based on Quality of Service Framework

Resource Partitioning and Application Scheduling with Module Merging on Dynamically and Partially Reconfigurable FPGAs

A probabilistic-approach based resource allocation algorithm in pervasive computing systems

Evaluation Mappings of Spatial Accelerator Based On Data Placement

RealArch: A Real-Time Scheduler for Mapping Multi-Tenant DNNs on Multi-Core Accelerators

AuRORA: A Full-Stack Solution for Scalable and Virtualized Accelerator Integration

BARM: A Batch-Aware Resource Manager for Boosting Multiple Neural Networks Inference on GPUs With Memory Oversubscription

UltraShare: FPGA-based Dynamic Accelerator Sharing and Allocation

Holistic Resource Allocation Under Federated Scheduling for Parallel Real-time Tasks

A Resourceful Coordination Approach for Multilevel Scheduling