Abstract:We have witnessed an increasing usage of multi-modal DNNs with multi-task heads on edge computing scenarios. These networks typically process inputs of different modalities first, then extract features for unified fusion, and finally input the fused features into multi-task heads. Such networks are often used to determine pose and navigate movement direction via multi-modal data obtained from diverse sensory equipment, therefore necessitating low inference latency. An edge device cluster with high-speed interconnection can be employed to support such DNN workload for scaled-out performance.For accelerating model inference on edge devices, previous researchers have proposed methods including model pruning, quantization, etc. However, these methods failed to take advantage of the structural features of multi-modal DNNs with multi-task heads and may impair the model’s prediction accuracy.Based on the intrinsic structure of multi-modal DNNs with multi-task heads, we propose Sub-model Parallelism to achieve scalable execution speedup. Sub-model Parallelism is a scale-out deployment method that first assigns preprocessing tasks of different modalities to different edge devices, then delivers them to a device for modality feature fusion, and finally distributes the fused features to other devices responsible for different task head computations. We run experiments on BEVFusion network and achieve an approximately 30% reduction in latency using two Jetson Orin devices connected by Remote Direct Memory Access (RDMA). Furthermore, we conduct a series of simulation experiments to cover scale-out scenarios and also achieve a good level of latency reduction. We hope that our proposed method can provide valuable experience for the optimized scale-out deployment of large multi-modal DNNs with multi-task heads on multiple edge devices.

FastPTM: Fast weights loading of pre-trained models for parallel inference service provisioning

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

FastTuning: Enabling Fast and Efficient Hyper-Parameter Tuning with Partitioning and Parallelism of Search Space

Parallel Training of Pre-Trained Models Via Chunk-Based Dynamic Memory Management

TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices

PetS: A Unified Framework for Parameter-Efficient Transformers Serving

Perseus: Characterizing Performance and Cost of Multi-Tenant Serving for CNN Models

DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing

Efficient Federated Prompt Tuning for Black-box Large Pre-trained Models

PETPS: Supporting Huge Embedding Models with Persistent Memory

Fast-iTPN: Integrally Pre-Trained Transformer Pyramid Network with Token Migration

FastFold: Reducing AlphaFold Training Time from 11 Days to 67 Hours

Angel-PTM: A Scalable and Economical Large-scale Pre-training System in Tencent

A Multi-Level Framework for Accelerating Training Transformer Models

Sub-model Parallelism: A Scale-out Deployment Method for Large Multi-modal DNNs

PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation

An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUs

FASTERMOE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models

Exploiting Student Parallelism for Low-latency GPU Inference of BERT-like Models in Online Services

FedPT: Federated Proxy-Tuning of Large Language Models on Resource-Constrained Edge Devices