Abstract:Transformer-based models have unlocked a plethora of powerful intelligent applications at the edge, such as voice assistant in smart home. Traditional deployment approaches offload the inference workloads to the remote cloud server, which would induce substantial pressure on the backbone network as well as raise users' privacy concerns. To address that, in-situ inference has been recently recognized for edge intelligence, but it still confronts significant challenges stemming from the conflict between intensive workloads and limited on-device computing resources. In this paper, we leverage our observation that many edge environments usually comprise a rich set of accompanying trusted edge devices with idle resources and propose Galaxy, a collaborative edge AI system that breaks the resource walls across heterogeneous edge devices for efficient Transformer inference acceleration. Galaxy introduces a novel hybrid model parallelism to orchestrate collaborative inference, along with a heterogeneity-aware parallelism planning for fully exploiting the resource potential. Furthermore, Galaxy devises a tile-based fine-grained overlapping of communication and computation to mitigate the impact of tensor synchronizations on inference latency under bandwidth-constrained edge environments. Extensive evaluation based on prototype implementation demonstrates that Galaxy remarkably outperforms state-of-the-art approaches under various edge environment setups, achieving up to 2.5x end-to-end latency reduction.

What problem does this paper attempt to address?

### Problems Addressed by the Paper The paper aims to address the resource constraints faced by Transformer model inference on edge devices. Specifically, the traditional approach of offloading inference tasks to remote cloud servers not only increases the burden on backbone networks but also raises user privacy concerns. To overcome these issues, the paper proposes a collaborative edge AI system named Galaxy, which accelerates Transformer model inference through the following methods: 1. **Utilizing Multi-Device Collaboration**: It is observed that in common edge environments (such as smart homes), there are usually multiple trusted edge devices with idle resources. Galaxy leverages these devices for distributed computing to break the resource barriers between different heterogeneous devices. 2. **Hybrid Model Parallelism (HMP)**: A new hybrid model parallelism architecture is introduced, combining the advantages of Tensor Parallelism (TP) and Sequence Parallelism (SP) to optimize the distributed inference process across devices. 3. **Resource-Aware Workload Planning**: A workload planning algorithm is designed that comprehensively considers device heterogeneity and memory budgets, ensuring balanced workload distribution and full utilization of computational resources across devices. 4. **Fine-Grained Communication Optimization**: A tile-based fine-grained communication optimization scheme is proposed, which reduces the latency caused by tensor synchronization in bandwidth-constrained environments by decoupling the tight dependency between continuous computation and communication operations. Through the above methods, Galaxy significantly enhances inference performance in edge environments. Experimental results show that it can achieve up to 2.5 times reduction in end-to-end latency compared to existing methods.

Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inference

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

Transformer Inference Acceleration in Edge Computing Environment

ED-ViT: Splitting Vision Transformer for Distributed Inference on Edge Devices

EdgeCI: Distributed Workload Assignment and Model Partitioning for CNN Inference on Edge Clusters

Implementation of Big AI Models for Wireless Networks with Collaborative Edge Computing

Edge AI: On-Demand Accelerating Deep Neural Network Inference via Edge Computing

MDP: Model Decomposition and Parallelization of Vision Transformer for Distributed Edge Inference

EdgeTran: Co-designing Transformers for Efficient Inference on Mobile Edge Platforms

Exploring Approximation and Dataflow Co-Optimization for Scalable Transformer Inference Architecture on the Edge

AI on the Edge: Rethinking AI-based IoT Applications Using Specialized Edge Architectures

Edge Intelligence: On-Demand Deep Learning Model Co-Inference with Device-Edge Synergy

CoEdge: Cooperative DNN Inference With Adaptive Workload Partitioning Over Heterogeneous Edge Devices

Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs

Asteroid: Resource-Efficient Hybrid Pipeline Parallelism for Collaborative DNN Training on Heterogeneous Edge Devices

Adaptive Device-Edge Collaboration on DNN Inference in AIoT: A Digital Twin-Assisted Approach

Edge-PRUNE: Flexible Distributed Deep Learning Inference

Collaborative Inference for Deep Neural Networks in Edge Environments

Researching the CNN Collaborative Inference Mechanism for Heterogeneous Edge Devices

Boosting Edge Intelligence with Collaborative Cross-Edge Analytics.

Communication-Efficient Separable Neural Network for Distributed Inference on Edge Devices