Abstract:We have witnessed an increasing usage of multi-modal DNNs with multi-task heads on edge computing scenarios. These networks typically process inputs of different modalities first, then extract features for unified fusion, and finally input the fused features into multi-task heads. Such networks are often used to determine pose and navigate movement direction via multi-modal data obtained from diverse sensory equipment, therefore necessitating low inference latency. An edge device cluster with high-speed interconnection can be employed to support such DNN workload for scaled-out performance.For accelerating model inference on edge devices, previous researchers have proposed methods including model pruning, quantization, etc. However, these methods failed to take advantage of the structural features of multi-modal DNNs with multi-task heads and may impair the model’s prediction accuracy.Based on the intrinsic structure of multi-modal DNNs with multi-task heads, we propose Sub-model Parallelism to achieve scalable execution speedup. Sub-model Parallelism is a scale-out deployment method that first assigns preprocessing tasks of different modalities to different edge devices, then delivers them to a device for modality feature fusion, and finally distributes the fused features to other devices responsible for different task head computations. We run experiments on BEVFusion network and achieve an approximately 30% reduction in latency using two Jetson Orin devices connected by Remote Direct Memory Access (RDMA). Furthermore, we conduct a series of simulation experiments to cover scale-out scenarios and also achieve a good level of latency reduction. We hope that our proposed method can provide valuable experience for the optimized scale-out deployment of large multi-modal DNNs with multi-task heads on multiple edge devices.

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Model Parallelism Optimization for Distributed DNN Inference on Edge Devices.

Accelerating DNN Inference by Edge-Cloud Collaboration

Adaptive Deep Inference Framework for Cloud-Edge Collaboration

Joint Optimization of Device Placement and Model Partitioning for Cooperative DNN Inference in Heterogeneous Edge Computing

Pipeline Parallelism for Inference on Heterogeneous Edge Computing

Collaborative DNNs Inference with Joint Model Partition and Compression in Mobile Edge-Cloud Computing Networks

DECC: Delay-Aware Edge-Cloud Collaboration for Accelerating DNN Inference

Sub-model Parallelism: A Scale-out Deployment Method for Large Multi-modal DNNs

Multi-Compression Scale DNN Inference Acceleration based on Cloud-Edge-End Collaboration

Resource-efficient Parallel Split Learning in Heterogeneous Edge Computing

A Novel Adaptive Computation Offloading Strategy for Collaborative DNN Inference over Edge Devices.

Context-Aware Compilation of DNN Training Pipelines across Edge and Cloud

DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices

Collaborative Inference for Deep Neural Networks in Edge Environments

Distributed DNN Inference with Fine-grained Model Partitioning in Mobile Edge Computing Networks

EdgeSP: Scalable Multi-device Parallel DNN Inference on Heterogeneous Edge Clusters

Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices

FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training Approach for Heterogeneous Edge Devices

Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy