Abstract:Large Language Models (LLMs) have revolutionized natural language processing, but deploying them in resource-constrained environments and privacy-sensitive domains remains challenging. This paper introduces the Distributed Large Language Model (DisLLM), a novel distributed learning framework that addresses privacy preservation and computational efficiency issues in LLM fine-tuning and inference. DisLLM leverages the Splitfed Learning (SFL) approach, combining Federated Learning (FL) and Split Learning (SL) benefits for privacy-preserving and computationally efficient LLM deployment. It splits the LLM into client-side and server-side components, allowing sensitive data to remain on client devices while distributing the computational load. An efficient fine-tuning technique called Low-Rank Adaptation (LoRA) is incorporated to reduce trainable parameters and memory footprint. Additionally, Local Differential Privacy (LDP) is used to add controlled noise to the sensitive data, further enhancing privacy preservation. Experimental evaluations on various text classification datasets demonstrate DisLLM’s effectiveness in model accuracy, convergence rate, and computational efficiency. Factors like the number of clients, cut layer position, and model variants are thoroughly investigated. Results show that DisLLM achieves comparable accuracy to centralized fine-tuning while preserving data privacy and optimizing resource utilization. Moreover, its ability to handle multiclass classification tasks with many classes emphasizes its robustness and versatility. Therefore, DisLLM represents a significant step forward in the practical deployment of LLMs, ensuring efficiency and privacy preservation.

Analyzing constrained LLM through PDFA-learning

LLMs as Probabilistic Minimally Adequate Teachers for DFA Learning

Combining Constraint Programming Reasoning with Large Language Model Predictions

Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference

LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language

Alignment Between the Decision-Making Logic of LLMs and Human Cognition: A Case Study on Legal LLMs

Few-Shot Fairness: Unveiling LLM's Potential for Fairness-Aware Classification

Understanding and Mitigating Language Confusion in LLMs

LLM-Generated Natural Language Meets Scaling Laws: New Explorations and Data Augmentation Methods

Unused information in token probability distribution of generative LLM: improving LLM reading comprehension through calculation of expected values

Finetuning LLMs for Comparative Assessment Tasks

PDFA Distillation via String Probability Queries

DisLLM: Distributed LLMs for Privacy Assurance in Resource-Constrained Environments

LLMs learn governing principles of dynamical systems, revealing an in-context neural scaling law

Conformal Language Modeling

Incoherent Probability Judgments in Large Language Models

Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE

Language Models with Conformal Factuality Guarantees

Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment

Learning The Structure Of Probabilistic Sentential Decision Diagrams