Abstract:Evaluating LLMs and text-to-image models is a computationally intensive task often overlooked. Efficient evaluation is crucial for understanding the diverse capabilities of these models and enabling comparisons across a growing number of new models and benchmarks. To address this, we introduce SubLIME, a data-efficient evaluation framework that employs adaptive sampling techniques, such as clustering and quality-based methods, to create representative subsets of benchmarks. Our approach ensures statistically aligned model rankings compared to full datasets, evidenced by high Pearson correlation coefficients. Empirical analysis across six NLP benchmarks reveals that: (1) quality-based sampling consistently achieves strong correlations (0.85 to 0.95) with full datasets at a 10\% sampling rate such as Quality SE and Quality CPD (2) clustering methods excel in specific benchmarks such as MMLU (3) no single method universally outperforms others across all metrics. Extending this framework, we leverage the HEIM leaderboard to cover 25 text-to-image models on 17 different benchmarks. SubLIME dynamically selects the optimal technique for each benchmark, significantly reducing evaluation costs while preserving ranking integrity and score distribution. Notably, a minimal sampling rate of 1% proves effective for benchmarks like MMLU. Additionally, we demonstrate that employing difficulty-based sampling to target more challenging benchmark segments enhances model differentiation with broader score distributions. We also combine semantic search, tool use, and GPT-4 review to identify redundancy across benchmarks within specific LLM categories, such as coding benchmarks. This allows us to further reduce the number of samples needed to maintain targeted rank preservation. Overall, SubLIME offers a versatile and cost-effective solution for the robust evaluation of LLMs and text-to-image models.

Sample Design Engineering: An Empirical Study of What Makes Good Downstream Fine-Tuning Samples for LLMs

More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineering

APEER: Automatic Prompt Engineering Enhances Large Language Model Reranking

SEED: Customize Large Language Models with Sample-Efficient Adaptation for Code Generation

Mitigating Tail Narrowing in LLM Self-Improvement via Socratic-Guided Sampling

SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection

From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning

Exploring Parameter-Efficient Fine-Tuning Techniques for Code Generation with Large Language Models

LLMRefine: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback

Fine-grained LLM Agent: Pinpointing and Refining Large Language Models via Fine-Grained Actionable Feedback

APE: Active Learning-based Tooling for Finding Informative Few-shot Examples for LLM-based Entity Matching

DesignProbe: A Graphic Design Benchmark for Multimodal Large Language Models

An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models

Fine-Tuning and Prompt Engineering for Large Language Models-based Code Review Automation

Evolutionary Prompt Design for LLM-Based Post-ASR Error Correction

LLM as a Complementary Optimizer to Gradient Descent: A Case Study in Prompt Tuning

LLMs are Imperfect, Then What? An Empirical Study on LLM Failures in Software Engineering

Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Sampling

Sample-Efficient Alignment for LLMs

Experimental Design for Active Transductive Inference in Large Language Models