Abstract:Despite their remarkable achievements, modern Large Language Models (LLMs) face exorbitant computational and memory footprints. Recently, several works have shown significant success in training-free and data-free compression (pruning and quantization) of LLMs that achieve 50 - 60% sparsity and reduce the bit width to 3 or 4 bits per weight, with negligible degradation of perplexity over the uncompressed baseline. As recent research efforts are focused on developing increasingly sophisticated compression methods, our work takes a step back and re-evaluates the effectiveness of existing SoTA compression methods, which rely on a fairly simple and widely questioned metric, perplexity (even for dense LLMs). We introduce Knowledge-Intensive Compressed LLM BenchmarK (LLM-KICK), a collection of carefully curated tasks to redefine the evaluation protocol for compressed LLMs, which have significant alignment with their dense counterparts and perplexity fail to capture subtle change in their true capabilities. LLM-KICK unveils many favorable merits and unfortunate plights of current SoTA compression methods: all pruning methods suffer significant performance degradation, sometimes at trivial sparsity ratios (e.g., 25-30%), and fail for N:M sparsity in knowledge-intensive tasks; current quantization methods are more successful than pruning; yet, pruned LLMs even at $\geq 50$% sparsity are robust in-context retrieval and summarization systems; among others. LLM-KICK is designed to holistically access compressed LLMs' ability for language understanding, reasoning, generation, in-context retrieval, in-context summarization, etc. We hope our study can foster the development of better LLM compression methods. The reproduced codes are available at <a class="link-external link-https" href="https://github.com/VITA-Group/llm-kick" rel="external noopener nofollow">this https URL</a>.

Using First-Order Logic to Compress Sentences.

Sentence Compression with a Markov Logic Network

An Operation Network for Abstractive Sentence Compression.

Compressive Summarization with Plausibility and Salience Modeling

Query-focused Sentence Compression in Linear Time

Compression Represents Intelligence Linearly

Sentence Compression for Aspect-Based Sentiment Analysis

Learning Semantic Structure through First-Order-Logic Translation

Sentence compression with natural language generation

With Measured Words: Simple Sentence Selection for Black-Box Optimization of Sentence Compression Algorithms

Translating First-Order Theories into Logic Programs.

Divide and Translate: Compositional First-Order Logic Translation and Verification for Complex Logical Reasoning

Ranking LLMs by compression

Greedy Flipping for Constrained Word Deletion.

FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression

500xCompressor: Generalized Prompt Compression for Large Language Models

First-Order vs. Second-Order Encodings for LTLf-to-Automata Translation

Compressing LLMs: The Truth is Rarely Pure and Never Simple

A lightweight Chinese semantic dependency parsing model based on sentence compression

Exploring Neural Models for Parsing Natural Language into First-Order Logic

LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models