Abstract:In recent years, deep neural networks (DNNs) have witnessed a booming of artificial intelligence Internet of Things applications with stringent demands across high accuracy and low latency. A widely adopted solution is to process such computation-intensive DNNs inference tasks with edge computing. Nevertheless, existing edge-based DNN processing methods still cannot achieve acceptable performance due to the intensive transmission data and unnecessary computation. To address the above limitations, we take the advantage of Multi-exit DNNs (ME-DNNs) that allows the tasks to exit early at different depths of the DNN during inference, based on the input complexity. However, naively deploying ME-DNNs in edge still fails to deliver fast and consistent inference in the wild environment. Specifically, 1) at the model-level, unsuitable exit settings will increase additional computational overhead and will lead to excessive queuing delay; 2) at the computation-level, it is hard to sustain high performance consistently in the dynamic edge computing environment. In this paper, we present a Low Latency Edge Intelligence Scheme based on Multi-Exit DNNs (LEIME) to tackle the aforementioned problem. At the model-level, we propose an exit setting algorithm to automatically build optimal ME-DNNs with lower time complexity; At the computation-level, we present a distributed offloading mechanism to fine-tune the task dispatching at runtime to sustain high performance in the dynamic environment, which has the property of close-to-optimal performance guarantee. Finally, we implement a prototype system and extensively evaluate it through testbed and large-scale simulation experiments. Experimental results demonstrate that LEIME significantly improves applications' performance, achieving 1.1–18.7 × speedup in different situations.

MMExit: Enabling Fast and Efficient Multi-modal DNN Inference with Adaptive Network Exits

Unlocking the Non-deterministic Computing Power with Memory-Elastic Multi-Exit Neural Networks

Multi-Exit DNN Inference Acceleration Based on Multi-Dimensional Optimization for Edge Intelligence

Elastic DNN Inference with Unpredictable Exit in Edge Computing

MOC: Multi-Objective Mobile CPU-GPU Co-Optimization for Power-Efficient DNN Inference

Multi-exit DNN inference acceleration for intelligent terminal with heterogeneous processors

Enabling Low Latency Edge Intelligence Based on Multi-exit DNNs in the Wild

You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Model

Early-Exit with Class Exclusion for Efficient Inference of Neural Networks

CMCI: A Robust Multimodal Fusion Method for Spiking Neural Networks

Dynamic Path Based DNN Synergistic Inference Acceleration in Edge Computing Environment.

DCMN: Double Core Memory Network for Patient Outcome Prediction with Multimodal Data.

Energy-Aware Adaptive Multi-Exit Neural Network Inference Implementation for a Millimeter-Scale Sensing System

Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exiting

MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks

M2SN: Adaptive and Dynamic Multi-modal Shortcut Network Architecture for Latency-Aware Applications

Deep Feature Surgery: Towards Accurate and Efficient Multi-Exit Networks

Dynamic Multi-path Neural Network

The Balanced Multi-Modal Spiking Neural Networks with Online Loss Adjustment and Time Alignment

Jointly-Learned Exit and Inference for a Dynamic Neural Network : JEI-DNN

Enabling Latency-Sensitive DNN Inference Via Joint Optimization of Model Surgery and Resource Allocation in Heterogeneous Edge