Abstract:Large Language Models (LLMs) are currently pre-trained and fine-tuned on large cloud servers. The next frontier is LLM personalization, where a foundation model can be fine-tuned with user/task-specific data. Given the sensitive nature of such private data, it is desirable to fine-tune these models on edge devices to improve user trust. However, fine-tuning on resource-constrained edge devices presents significant challenges due to substantial memory and computational demands, as well as limited infrastructure support. We observe that inference engines (e.g., ExecuTorch) can be repurposed for fine-tuning by leveraging zeroth-order (ZO) optimization, which uses multiple forward passes to approximate gradients. However, directly applying ZO methods on edge devices is impractical due to the high computational cost of multiple model perturbations required to achieve accuracy improvements. Based on these observations, we propose a memory- and computation-efficient LLM fine-tuning method for edge devices. Our approach has three key innovations: (1) We introduce a parallelized randomized gradient estimation (P-RGE) technique that achieves high parallel efficiency by leveraging outer-loop and inner-loop parallelization. This enables multiple function queries and forward passes to be executed in parallel, reducing training time. (2) We integrate P-RGE with parameter-efficient fine-tuning methods (e.g. LoRA) to further reduce computational and memory overhead. (3) We implement a P-RGE LoRA-FA module that fully supports fine-tuning with ExecuTorch. Our approach requires no modifications to ExecuTorch's runtime code, as it can be implemented with server-side code changes only. Experiments demonstrate that P-RGE achieves substantial runtime speedups and memory savings while improving fine-tuning accuracy, paving the way for practical deployment of LLMs in real-time, on-device applications.

Device-Edge Cooperative Fine-Tuning of Foundation Models as a 6G Service

Resource Management for Low-latency Cooperative Fine-tuning of Foundation Models at the Network Edge

Condense: A Framework for Device and Frequency Adaptive Neural Network Models on the Edge.

FedPEAT: Convergence of Federated Learning, Parameter-Efficient Fine Tuning, and Emulator Assisted Tuning for Artificial Intelligence Foundation Models with Mobile Edge Computing

Foundation Model Based Native AI Framework in 6G with Cloud-Edge-End Collaboration

FedPFT: Federated Proxy Fine-Tuning of Foundation Models

Beyond Fine-Tuning: Efficient and Effective Fed-Tuning for Mobile/Web Users

Federated Fine-Tuning for Pre-Trained Foundation Models Over Wireless Networks

See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition

EdgeFM: Leveraging Foundation Model for Open-set Learning on the Edge

EdgeTA: Neuron-grained Scaling of Foundation Models in Edge-side Retraining

Private Edge Computing Resource Allocation and Communication Optimization Based on Federated Learning

Enabling Efficient On-Device Fine-Tuning of LLMs Using Only Inference Engines

One Communication Round is All It Needs for Federated Fine-Tuning Foundation Models

A Fine-Grained End-to-End Latency Optimization Framework for Wireless Collaborative Inference

Pluto and Charon: A Time and Memory Efficient Collaborative Edge AI Framework for Personal LLMs Fine-Tuning

Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applications

Mobile Foundation Model as Firmware

Exploring Edge-driven Collaborative Fine-tuning Towards Customized AIGC Services

SFPrompt: Communication-Efficient Split Federated Fine-Tuning for Large Pre-Trained Models over Resource-Limited Devices

Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies