Abstract:Mobile devices are becoming increasingly capable of delivering intelligent services by leveraging deep learning architectures such as deep neural networks (DNNs). However, due to the compute-intensive nature of these tasks, mobile devices often struggle to handle them independently, leading to the exploration of collaborative inference as a promising solution for achieving low-latency mobile intelligence. Despite its potential benefits, many challenges need to be addressed in realizing the full potential of inference acceleration. This paper presents a collaborative device-edge inference optimization framework as a promising solution to inference acceleration. The framework comprises fundamental modules, including the Parameters Generator, Accuracy Predictor, Delay Calculator, and Optimizer, which are specifically designed to identify the optimal set of parameters for Model Compression, DNN Partition, and Feature Compression. To illustrate its implementation, an example of a deep CNN network is introduced, and the collaborative inference latency optimization is formulated as a mixed-integer programming problem. The implementation of a specific algorithm instance using a quantum-inspired optimizer within the optimization framework is then presented. A multiple regression-based inference accuracy prediction model is proposed to maintain inference accuracy close to that of the original network while significantly reducing the time consumption during the offline phase. Through various simulation scenarios involving inference tasks of AlexNet and RegNet on CIFAR-10, incorporating diverse hardware computation specifications and wireless communication link conditions, the proposed framework demonstrates superior performance in terms of inference acceleration compared to the compared methods.

Coarse-to-Fine: A hierarchical DNN inference framework for edge computing

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

Adaptive Distributed Convolutional Neural Network Inference at the Network Edge with ADCNN

End-Edge Collaborative Inference of Convolutional Fuzzy Neural Networks for Big Data-Driven Internet of Things

DECC: Delay-Aware Edge-Cloud Collaboration for Accelerating DNN Inference

Joint Optimization of Data Transfer and Co-Execution for DNN in Edge Computing

CCIED: Cache-aided Collaborative Intelligence Between Edge Devices.

CoEdge: Cooperative DNN Inference With Adaptive Workload Partitioning Over Heterogeneous Edge Devices

EdgeCI: Distributed Workload Assignment and Model Partitioning for CNN Inference on Edge Clusters

Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy

ADDA: Adaptive Distributed DNN Inference Acceleration in Edge Computing Environment

Automated Exploration and Implementation of Distributed CNN Inference at the Edge

DVFO: Learning-Based DVFS for Energy-Efficient Edge-Cloud Collaborative Inference

A Fine-Grained End-to-End Latency Optimization Framework for Wireless Collaborative Inference

Adaptive Deep Inference Framework for Cloud-Edge Collaboration

Accelerating DNN Inference by Edge-Cloud Collaboration

DeepZoning: Re-accelerate CNN Inference with Zoning Graph for Heterogeneous Edge Cluster

Design and Implementation of Deep Neural Network for Edge Computing

EdgeKE: An On-Demand Deep Learning IoT System for Cognitive Big Data on Industrial Edge Devices

An Adaptive DNN Inference Acceleration Framework with End–edge–cloud Collaborative Computing

SHA-CNN: Scalable Hierarchical Aware Convolutional Neural Network for Edge AI