Abstract:As a key technology of enabling Artificial Intelligence (AI) applications in 5G era, Deep Neural Networks (DNNs) have quickly attracted widespread attention. However, it is challenging to run computation-intensive DNN-based tasks on mobile devices due to the limited computation resources. What's worse, traditional cloud-assisted DNN inference is heavily hindered by the significant wide-area network latency, leading to poor real-time performance as well as low quality of user experience. To address these challenges, in this paper, we propose Edgent, a framework that leverages edge computing for DNN collaborative inference through device-edge synergy. Edgent exploits two design knobs: (1) DNN partitioning that adaptively partitions computation between device and edge for purpose of coordinating the powerful cloud resource and the proximal edge resource for real-time DNN inference; (2) DNN right-sizing that further reduces computing latency via early exiting inference at an appropriate intermediate DNN layer. In addition, considering the potential network fluctuation in real-world deployment, Edgent is properly design to specialize for both static and dynamic network environment. Specifically, in a static environment where the bandwidth changes slowly, Edgent derives the best configurations with the assist of regression-based prediction models, while in a dynamic environment where the bandwidth varies dramatically, Edgent generates the best execution plan through the online change point detection algorithm that maps the current bandwidth state to the optimal configuration. We implement Edgent prototype based on the Raspberry Pi and the desktop PC and the extensive experimental evaluations demonstrate Edgent's effectiveness in enabling on-demand low-latency edge intelligence.

Task-aware Swapping for Efficient DNN Inference on DRAM-constrained Edge Systems

SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

Unlocking the Non-deterministic Computing Power with Memory-Elastic Multi-Exit Neural Networks

DaDianNao: A Machine-Learning Supercomputer

FlexNN: Efficient and Adaptive DNN Inference on Memory-Constrained Edge Devices.

RNC: Efficient RRAM-aware NAS and Compilation for DNNs on Resource-Constrained Edge Devices

An Application-oblivious Memory Scheduling System for DNN Accelerators

Adaptive Distributed Convolutional Neural Network Inference at the Network Edge with ADCNN

CoEdge: Cooperative DNN Inference With Adaptive Workload Partitioning Over Heterogeneous Edge Devices

A Task-Adaptive In-Situ ReRAM Computing for Graph Convolutional Networks

A generic deep learning architecture optimization method for edge device based on start-up latency reduction

EdgeDRNN: Enabling Low-latency Recurrent Neural Network Edge Inference

Enabling Large Neural Networks on Tiny Microcontrollers with Swapping

Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing

Accelerating Neural Network Inference with Processing-in-DRAM: From the Edge to the Cloud

Incremental Training and Group Convolution Pruning for Runtime DNN Performance Scaling on Heterogeneous Embedded Platforms

DT-SCNN: dual-threshold spiking convolutional neural network with fewer operations and memory access for edge applications

Dynamic DNN Decomposition for Lossless Synergistic Inference

A Hybrid RRAM-SRAM Computing-In-Memory Architecture for Deep Neural Network Inference-Training Edge Acceleration

SNrram: an Efficient Sparse Neural Network Computation Architecture Based on Resistive Random-Access Memory.