Abstract:For model inference of convolutional neural networks (CNNs), we nowadays witness a shift from the Cloud to the Edge. Unfortunately, deploying and inferring large, compute- and memory-intensive CNNs on Internet of Things devices at the Edge is challenging as they typically have limited resources. One approach to address this challenge is to leverage all available resources across multiple edge devices to execute a large CNN by properly partitioning it and running each CNN partition on a separate edge device. However, there currently does not exist a design and programming framework that takes a trained CNN model as input and subsequently allows for efficiently exploring and automatically implementing a range of different CNN partitions on multiple edge devices to facilitate distributed CNN inference. Therefore, in this article, we propose a novel framework that automates the splitting of a CNN model into a set of submodels as well as the code generation needed for the distributed and collaborative execution of these submodels on multiple, possibly heterogeneous, edge devices, while supporting the exploitation of parallelism among and within the edge devices. In addition, since the number of different CNN mapping possibilities on multiple edge devices is vast, our framework also features a multistage and hierarchical design space exploration methodology to efficiently search for (near-)optimal distributed CNN inference implementations. Our experimental results demonstrate that our work allows for rapidly finding and realizing distributed CNN inference implementations with reduced energy consumption and memory usage per edge device, and under certain conditions, with improved system throughput as well.

Towards Efficient Inference: Adaptively Cooperate in Heterogeneous IoT Edge Cluster

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

Efficient Partitioning and Communication Scheme-Based Distributed Edge Computing to Accelerate Deep Neural Network

Researching the CNN Collaborative Inference Mechanism for Heterogeneous Edge Devices

DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices

CoEdge: Cooperative DNN Inference With Adaptive Workload Partitioning Over Heterogeneous Edge Devices

Automated Exploration and Implementation of Distributed CNN Inference at the Edge

EdgeCI: Distributed Workload Assignment and Model Partitioning for CNN Inference on Edge Clusters

DeepThings: Distributed Adaptive Deep Learning Inference on Resource-Constrained IoT Edge Clusters

Enhancing Distributed In-Situ CNN Inference in the Internet of Things

Collaborative Inference for Deep Neural Networks in Edge Environments

Adaptive Distributed Convolutional Neural Network Inference at the Network Edge with ADCNN

DeeperThings: Fully Distributed CNN Inference on Resource-Constrained Edge Devices

Opt-CoInfer: Optimal Collaborative Inference Across IoT and Cloud for Fast and Accurate CNN Inference

Low Latency Deep Learning Inference Model for Distributed Intelligent IoT Edge Clusters

Edge AI Inference in Heterogeneous Constrained Computing: Feasibility and Opportunities

Self-aware Collaborative Edge Inference with Embedded Devices for Task-oriented IIoT

Collaborative Inference for AI-Empowered IoT Devices

Communication-Efficient Edge AI Inference Over Wireless Networks

Toward Collaborative Inferencing of Deep Neural Networks on Internet-of-Things Devices