Abstract:Large Language Models (LLMs) have received much recent attention due to their human-level accuracy. While existing works mostly focus on either improving accuracy or testing accuracy robustness, the computation efficiency of LLMs, which is of paramount importance due to often vast generation demands and real-time requirements, has surprisingly received little attention. In this article, we make the first attempt to understand and test potential computation efficiency robustness in state-of-the-art LLMs. By analyzing the working mechanism and implementation of 20,543 public-accessible LLMs, we observe a fundamental property in LLMs that could be manipulated in an adversarial manner to reduce computation efficiency significantly. Our interesting observation is that the output length determines the computation efficiency of LLMs instead of the input, where the output length depends on two factors: an often sufficiently large yet pessimistic pre-configured threshold controlling the max number of iterations and a runtime-generated end of sentence (EOS) token. Our key motivation is to generate test inputs that could sufficiently delay the generation of EOS such that LLMs would have to go through enough iterations to satisfy the pre-configured threshold. We present LLMEffiChecker, which can work under both white-box setting and black-box setting. In the white-box scenario, LLMEffiChecker develops a gradient-guided technique that searches for a minimal and unnoticeable perturbation at character-level, token-level, and structure-level. In the black-box scenario, LLMEffiChecker employs a causal inference-based approach to find critical tokens and similarly applies three levels of imperceptible perturbation to them. Both the white-box and black-box settings effectively delay the appearance of EOS, compelling these inputs to reach the naturally unreachable threshold. To demonstrate the effectiveness of LLMEffiChecker, we conduct a systematic evaluation on nine publicly available LLMs: Google T5, AllenAI WMT14, Helsinki-NLP translator, Facebook FairSeq, UNICAMP-DL translator, MarianMT, Google FLAN-T5, MBZUAI LaMini-GPT, and Salesforce CodeGen. Experimental results show that LLMEffiChecker can increase on average LLMs' response latency and energy consumption by 325% to 3,244% and 344% to 3,616%, respectively, by perturbing just one character or token in the input sentence. Our case study shows that inputs generated by LLMEffiChecker significantly affect the battery power in real-world mobile devices (i.e., drain more than 30 times battery power than normal inputs).

Reducing the Energy Dissipation of Large Language Models (llms) with Approximate Memories

Enabling Energy-Efficient Deployment of Large Language Models on Memristor Crossbar: A Synergy of Large and Small

Fine-Tuning and Deploying Large Language Models Over Edges: Issues and Approaches

Full Parameter Fine-tuning for Large Language Models with Limited Resources

SoftLMs: Efficient Adaptive Low-Rank Approximation of Language Models using Soft-Thresholding Mechanism

MEMORYLLM: Towards Self-Updatable Large Language Models

Schrodinger's Memory: Large Language Models

MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory

Efficient and Economic Large Language Model Inference with Attention Offloading

LLMem: Estimating GPU Memory Usage for Fine-Tuning Pre-Trained LLMs

LLMEffiChecker: : Understanding and Testing Efficiency Degradation of Large Language Models

LLMEffiChecker:Understanding and Testing Efficiency Degradation of Large Language Models

Search for Efficient Large Language Models

Efficient Memory Management for Large Language Model Serving with PagedAttention

Efficiently Training 7B LLM with 1 Million Sequence Length on 8 GPUs

Augmenting Language Models with Long-Term Memory

New Solutions on LLM Acceleration, Optimization, and Application

$\text{Memory}^3$: Language Modeling with Explicit Memory

Mitigating Memorization In Language Models

Aggressive Post-Training Compression on Extremely Large Language Models