Abstract:Precise estimation of downstream performance in large language models (LLMs) prior to training is essential for guiding their development process. Scaling laws analysis utilizes the statistics of a series of significantly smaller sampling language models (LMs) to predict the performance of the target LLM. For downstream performance prediction, the critical challenge lies in the emergent abilities in LLMs that occur beyond task-specific computational thresholds. In this work, we focus on the pre-training loss as a more computation-efficient metric for performance estimation. Our two-stage approach consists of first estimating a function that maps computational resources (e.g., FLOPs) to the pre-training Loss using a series of sampling models, followed by mapping the pre-training loss to downstream task Performance after the critical "emergent phase". In preliminary experiments, this FLP solution accurately predicts the performance of LLMs with 7B and 13B parameters using a series of sampling LMs up to 3B, achieving error margins of 5% and 10%, respectively, and significantly outperforming the FLOPs-to-Performance approach. This motivates FLP-M, a fundamental approach for performance prediction that addresses the practical need to integrate datasets from multiple sources during pre-training, specifically blending general corpora with code data to accurately represent the common necessity. FLP-M extends the power law analytical function to predict domain-specific pre-training loss based on FLOPs across data sources, and employs a two-layer neural network to model the non-linear relationship between multiple domain-specific loss and downstream performance. By utilizing a 3B LLM trained on a specific ratio and a series of smaller sampling LMs, FLP-M can effectively forecast the performance of 3B and 7B LLMs across various data mixtures for most benchmarks within 10% error margins.

Towards More Fine-grained and Reliable NLP Performance Prediction

A PERFORMANCE PREDICTION METHOD BASED ON NAIVE BAYES CLASSIFICATION

Collaborative Performance Prediction for Large Language Models

Toward Human-Like Evaluation for Natural Language Generation with Error Analysis

Towards CPU Performance Prediction: New Challenge Benchmark Dataset and Novel Approach

Scaling Laws for Predicting Downstream Performance in LLMs

Towards Efficient NLP: A Standard Evaluation and A Strong Baseline

How Powerful are Performance Predictors in Neural Architecture Search?

Query Performance Prediction using Relevance Judgments Generated by Large Language Models

Can We Predict Performance of Large Models across Vision-Language Tasks?

Research on Constructing Online Learning Performance Prediction Model Combining Feature Selection and Neural Network

Predicting performance in recommender systems

Query Performance Prediction for Neural IR: Are We There Yet?

LLM Performance Predictors are good initializers for Architecture Search

Third-Party Language Model Performance Prediction from Instruction

Predicting Fine-Tuning Performance with Probing

Performative Prediction with Neural Networks

Predicting Performance During Tutoring with Models of Recent Performance

Eliciting Informative Text Evaluations with Large Language Models

Imporved Model for Pavement Performance Prediction Based on Recurrent Neural Network Using LTPP Database