Abstract:Deploying Large Language Models (LLMs) locally on mobile devices presents a significant challenge due to their extensive memory requirements. In this paper, we introduce LinguaLinked, a system for decentralized, distributed LLM inference on mobile devices. LinguaLinked enables collaborative execution of the inference task across multiple trusted devices. LinguaLinked ensures data privacy by processing information locally. LinguaLinked uses three key strategies. First, an optimized model assignment technique segments LLMs and uses linear optimization to align segments with each device's capabilities. Second, an optimized data transmission mechanism ensures efficient and structured data flow between model segments while also maintaining the integrity of the original model structure. Finally, LinguaLinked incorporates a runtime load balancer that actively monitors and redistributes tasks among mobile devices to prevent bottlenecks, enhancing the system's overall efficiency and responsiveness. We demonstrate that LinguaLinked facilitates efficient LLM inference while maintaining consistent throughput and minimal latency through extensive testing across various mobile devices, from high-end to low-end Android devices. In our evaluations, compared to the baseline, LinguaLinked achieves an inference performance acceleration of $1.11\times$ to $1.61\times$ in single-threaded settings, $1.73\times$ to $2.65\times$ with multi-threading. Additionally, runtime load balancing yields an overall inference acceleration of $1.29\times$ to $1.32\times$.

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the challenges of local deployment of large - language models (LLMs) on mobile devices. Specifically, due to the large memory requirements of LLMs, it is very difficult to directly run these models on resource - constrained mobile devices. This involves not only technical problems, such as how to efficiently perform model inference within limited computing power and memory, but also privacy and data security issues, because traditional server - side inference requires sending data to remote servers, increasing the risk of data leakage. To address these challenges, the paper proposes the LinguaLinked system, which is a decentralized system for distributed LLM inference on mobile devices. LinguaLinked solves these problems through the following three key technical strategies: 1. **Optimized model allocation**: The LLM is divided into multiple smaller model segments by linear optimization techniques, and these segments are allocated to different devices according to the capabilities of each device. This can reduce the computing and storage burden of each device while maintaining the overall performance of the model. 2. **Optimized data transfer mechanism**: Ensure that the data flow between model segments is both efficient and structured while maintaining the integrity of the original model structure. This method reduces the latency during data transfer and improves the response speed of the system. 3. **Runtime load balancing**: By real - time monitoring and re - allocating tasks, prevent any single device from becoming a bottleneck, thereby improving the overall efficiency and responsiveness of the system. This helps to dynamically adapt to the workload changes of different devices and ensure the stability and reliability of the system. In summary, LinguaLinked aims to enable large - language models to run efficiently and securely on mobile devices through distributed computing while maintaining low latency and a high level of data privacy protection.

LinguaLinked: A Distributed Large Language Model Inference System for Mobile Devices

Large Language Models on Mobile Devices: Measurements, Analysis, and Insights

LLMCad: Fast and Scalable On-device Large Language Model Inference

Distributed Inference and Fine-tuning of Large Language Models Over The Internet

PowerInfer-2: Fast Large Language Model Inference on a Smartphone

Large Language Model Performance Benchmarking on Mobile Platforms: A Thorough Evaluation

MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases

Large Language Models (llms) Inference Offloading and Resource Allocation in Cloud-Edge Networks: an Active Inference Approach

ELMS: Elasticized Large Language Models On Mobile Devices

BlueLM-V-3B: Algorithm and System Co-Design for Multimodal Large Language Models on Mobile Devices

Mobile Edge Intelligence for Large Language Models: A Contemporary Survey

Distributed Inference Performance Optimization for LLMs on CPUs

Efficient and Economic Large Language Model Inference with Attention Offloading

On-Device Language Models: A Comprehensive Review

Transformer-Lite: High-efficiency Deployment of Large Language Models on Mobile Phone GPUs

Efficient Deployment of Large Language Model Across Cloud-Device Systems

A Contemporary Overview: Trends and Applications of Large Language Models on Mobile Devices

CoLLM: A Collaborative LLM Inference Framework for Resource-Constrained Devices

Porting Large Language Models to Mobile Devices for Question Answering

Enhancing On-Device LLM Inference with Historical Cloud-Based LLM Interactions

FwdLLM: Efficient Federated Finetuning of Large Language Models with Perturbed Inferences.