Abstract:The relentless expansion of deep learning applications in recent years has prompted a pivotal shift toward on-device execution, driven by the urgent need for real-time processing, heightened privacy concerns, and reduced latency across diverse domains. This article addresses the challenges inherent in optimising the execution of deep neural networks (DNNs) on mobile devices, with a focus on device heterogeneity, multi-DNN execution, and dynamic runtime adaptation. We introduce CARIn, a novel framework designed for the optimised deployment of both single- and multi-DNN applications under user-defined service-level objectives. Leveraging an expressive multi-objective optimisation framework and a runtime-aware sorting and search algorithm (RASS) as the MOO solver, CARIn facilitates efficient adaptation to dynamic conditions while addressing resource contention issues associated with multi-DNN execution. Notably, RASS generates a set of configurations, anticipating subsequent runtime adaptation, ensuring rapid, low-overhead adjustments in response to environmental fluctuations. Extensive evaluation across diverse tasks, including text classification, scene recognition, and face analysis, showcases the versatility of CARIn across various model architectures, such as Convolutional Neural Networks and Transformers, and realistic use cases. We observe a substantial enhancement in the fair treatment of the problem's objectives, reaching 1.92x when compared to single-model designs and up to 10.69x in contrast to the state-of-the-art OODIn framework. Additionally, we achieve a significant gain of up to 4.06x over hardware-unaware designs in multi-DNN applications. Finally, our framework sustains its performance while effectively eliminating the time overhead associated with identifying the optimal design in response to environmental challenges.

COS: Cross-Processor Operator Scheduling for Multi-Tenant Deep Learning Inference

MOC: Multi-Objective Mobile CPU-GPU Co-Optimization for Power-Efficient DNN Inference

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

An Optimization Toolchain Design Of Deep Learning Deployment Based On Heterogeneous Computing Platform

Automated Runtime-Aware Scheduling for Multi-Tenant DNN Inference on GPU

A Co-Scheduling Framework for DNN Models on Mobile and Edge Devices with Heterogeneous Hardware

POS: an Operator Scheduling Framework for Multi-model Inference on Edge Intelligent Computing

IOS: Inter-Operator Scheduler for CNN Acceleration

CoDL: efficient CPU-GPU co-execution for deep learning inference on mobile devices

Joint Device Scheduling and Resource Allocation for ISCC-Based Multi-View-Multi-Task Inference

Octopus: SLO-Aware Progressive Inference Serving via Deep Reinforcement Learning in Multi-tenant Edge Cluster

MultiTASC: A Multi-Tenancy-Aware Scheduler for Cascaded DNN Inference at the Consumer Edge

A Survey of Multi-Tenant Deep Learning Inference on GPU

VELTAIR: towards high-performance multi-tenant deep learning services via adaptive compilation and scheduling

Orchestrated Co-scheduling, Resource Partitioning, and Power Capping on CPU-GPU Heterogeneous Systems via Machine Learning

GACER: Granularity-Aware ConcurrEncy Regulation for Multi-Tenant Deep Learning

Perseus: Characterizing Performance and Cost of Multi-Tenant Serving for CNN Models

COAC: Cross-layer Optimization of Accelerator Configurability for Efficient CNN Processing

Inter-Layer Scheduling Space Exploration for Multi-model Inference on Heterogeneous Chiplets

Multi-Model Running Latency Optimization in an Edge Computing Paradigm

CARIn: Constraint-Aware and Responsive Inference on Heterogeneous Devices for Single- and Multi-DNN Workloads