Abstract:Large Language Models (LLMs) have received much recent attention due to their human-level accuracy. While existing works mostly focus on either improving accuracy or testing accuracy robustness, the computation efficiency of LLMs, which is of paramount importance due to often vast generation demands and real-time requirements, has surprisingly received little attention. In this article, we make the first attempt to understand and test potential computation efficiency robustness in state-of-the-art LLMs. By analyzing the working mechanism and implementation of 20,543 public-accessible LLMs, we observe a fundamental property in LLMs that could be manipulated in an adversarial manner to reduce computation efficiency significantly. Our interesting observation is that the output length determines the computation efficiency of LLMs instead of the input, where the output length depends on two factors: an often sufficiently large yet pessimistic pre-configured threshold controlling the max number of iterations and a runtime-generated end of sentence (EOS) token. Our key motivation is to generate test inputs that could sufficiently delay the generation of EOS such that LLMs would have to go through enough iterations to satisfy the pre-configured threshold. We present LLMEffiChecker, which can work under both white-box setting and black-box setting. In the white-box scenario, LLMEffiChecker develops a gradient-guided technique that searches for a minimal and unnoticeable perturbation at character-level, token-level, and structure-level. In the black-box scenario, LLMEffiChecker employs a causal inference-based approach to find critical tokens and similarly applies three levels of imperceptible perturbation to them. Both the white-box and black-box settings effectively delay the appearance of EOS, compelling these inputs to reach the naturally unreachable threshold. To demonstrate the effectiveness of LLMEffiChecker, we conduct a systematic evaluation on nine publicly available LLMs: Google T5, AllenAI WMT14, Helsinki-NLP translator, Facebook FairSeq, UNICAMP-DL translator, MarianMT, Google FLAN-T5, MBZUAI LaMini-GPT, and Salesforce CodeGen. Experimental results show that LLMEffiChecker can increase on average LLMs' response latency and energy consumption by 325% to 3,244% and 344% to 3,616%, respectively, by perturbing just one character or token in the input sentence. Our case study shows that inputs generated by LLMEffiChecker significantly affect the battery power in real-world mobile devices (i.e., drain more than 30 times battery power than normal inputs).

Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly

Long-context LLMs Struggle with Long In-context Learning

Systematic Evaluation of Long-Context LLMs on Financial Concepts

InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory

Why Does the Effective Context Length of LLMs Fall Short?

XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference

LLMEffiChecker:Understanding and Testing Efficiency Degradation of Large Language Models

LLMEffiChecker: : Understanding and Testing Efficiency Degradation of Large Language Models

Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models

Large Language Models Can Self-Improve in Long-context Reasoning

LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning

E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning

Make Your LLM Fully Utilize the Context

LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Models

LLM×MapReduce: Simplified Long-Sequence Processing Using Large Language Models

LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models

Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell

LongIns: A Challenging Long-context Instruction-based Exam for LLMs

QuickLLaMA: Query-aware Inference Acceleration for Large Language Models

Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?