Abstract:We have witnessed an increasing usage of multi-modal DNNs with multi-task heads on edge computing scenarios. These networks typically process inputs of different modalities first, then extract features for unified fusion, and finally input the fused features into multi-task heads. Such networks are often used to determine pose and navigate movement direction via multi-modal data obtained from diverse sensory equipment, therefore necessitating low inference latency. An edge device cluster with high-speed interconnection can be employed to support such DNN workload for scaled-out performance.For accelerating model inference on edge devices, previous researchers have proposed methods including model pruning, quantization, etc. However, these methods failed to take advantage of the structural features of multi-modal DNNs with multi-task heads and may impair the model’s prediction accuracy.Based on the intrinsic structure of multi-modal DNNs with multi-task heads, we propose Sub-model Parallelism to achieve scalable execution speedup. Sub-model Parallelism is a scale-out deployment method that first assigns preprocessing tasks of different modalities to different edge devices, then delivers them to a device for modality feature fusion, and finally distributes the fused features to other devices responsible for different task head computations. We run experiments on BEVFusion network and achieve an approximately 30% reduction in latency using two Jetson Orin devices connected by Remote Direct Memory Access (RDMA). Furthermore, we conduct a series of simulation experiments to cover scale-out scenarios and also achieve a good level of latency reduction. We hope that our proposed method can provide valuable experience for the optimized scale-out deployment of large multi-modal DNNs with multi-task heads on multiple edge devices.

Sub-model Parallelism: A Scale-out Deployment Method for Large Multi-modal DNNs

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

Model Parallelism Optimization for Distributed DNN Inference on Edge Devices.

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

EdgeSP: Scalable Multi-device Parallel DNN Inference on Heterogeneous Edge Clusters

Scalable Heterogeneous Scheduling Based Model Parallelism for Real-Time Inference of Large-Scale Deep Neural Networks

Multi-Model Running Latency Optimization in an Edge Computing Paradigm

Accelerating Deep Learning Inference via Model Parallelism and Partial Computation Offloading

Multi-Compression Scale DNN Inference Acceleration based on Cloud-Edge-End Collaboration

Fused-Layer-based DNN Model Parallelism and Partial Computation Offloading

Multi-Exit DNN Inference Acceleration Based on Multi-Dimensional Optimization for Edge Intelligence

Enabling Latency-Sensitive DNN Inference Via Joint Optimization of Model Surgery and Resource Allocation in Heterogeneous Edge

Model Parallelism Optimization for Distributed Inference Via Decoupled CNN Structure

Joint Optimization of Device Placement and Model Partitioning for Cooperative DNN Inference in Heterogeneous Edge Computing

Hastening Stream Offloading of Inference Via Multi-Exit DNNs in Mobile Edge Computing

NN-Stretch: Automatic Neural Network Branching for Parallel Inference on Heterogeneous Multi-Processors.

Delay-Aware DNN Inference Throughput Maximization in Edge Computing Via Jointly Exploring Partitioning and Parallelism

Distributed DNN Inference with Fine-grained Model Partitioning in Mobile Edge Computing Networks

EdgeLD: Locally Distributed Deep Learning Inference on Edge Device Clusters

Resource-efficient Parallel Split Learning in Heterogeneous Edge Computing