Abstract:Scaling up neural models has yielded significant advancements in a wide array of tasks, particularly in language generation. Previous studies have found that the performance of neural models frequently adheres to predictable scaling laws, correlated with factors such as training set size and model size. This insight is invaluable, especially as large-scale experiments grow increasingly resource-intensive. Yet, such scaling law has not been fully explored in dense retrieval due to the discrete nature of retrieval metrics and complex relationships between training data and model sizes in retrieval tasks. In this study, we investigate whether the performance of dense retrieval models follows the scaling law as other neural models. We propose to use contrastive log-likelihood as the evaluation metric and conduct extensive experiments with dense retrieval models implemented with different numbers of parameters and trained with different amounts of annotated data. Results indicate that, under our settings, the performance of dense retrieval models follows a precise power-law scaling related to the model size and the number of annotations. Additionally, we examine scaling with prevalent data augmentation methods to assess the impact of annotation quality, and apply the scaling law to find the best resource allocation strategy under a budget constraint. We believe that these insights will significantly contribute to understanding the scaling effect of dense retrieval models and offer meaningful guidance for future research endeavors.

What problem does this paper attempt to address?

The paper primarily explores the issue of scaling laws in Dense Retrieval models. Specifically, the research aims to investigate the following key questions: 1. **How does model size affect dense retrieval performance?** - Researchers observed the impact of model size on performance by adjusting the number of parameters in the dense retrieval model. 2. **How does the amount of annotated training data affect dense retrieval performance?** - The paper also considers the impact of different scales of annotated datasets on the performance of dense retrieval models. 3. **Do different types of annotated data lead to different scaling effects in dense retrieval models?** - To comprehensively understand the performance of dense retrieval models, the research also examines the impact of different types of data annotation methods on model performance. ### Main Findings - **Relationship between model size and performance**: The paper shows that as the model size increases, the performance of the dense retrieval model improves, following a precise power-law scaling rule. The authors propose a formula (Eq. 6) to fit this scaling rule and provide specific parameter values (as shown in Table 1). - **Contrastive entropy as an evaluation metric**: To better evaluate the performance of dense retrieval models, the paper introduces a new continuous evaluation metric—contrastive entropy. Experiments show a strong positive correlation between contrastive entropy and traditional ranking metrics (such as MAP@10, NDCG@10, etc.), thus effectively reflecting the overall retrieval capability of the model. - **Application of scaling laws**: The paper demonstrates that these scaling laws help optimize the training process of dense retrieval models, such as optimizing resource allocation strategies under budget constraints. In summary, this paper provides valuable insights for understanding and optimizing dense retrieval models. Especially when dealing with large-scale datasets, these scaling laws can help researchers design and implement model training strategies more efficiently.

Scaling Laws For Dense Retrieval

ScalingNote: Scaling up Retrievers with Large Language Models for Real-World Dense Retrieval

Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments

Scaling Law of Large Sequential Recommendation Models

Scaling Laws for Neural Language Models

Towards Neural Scaling Laws on Graphs

Scaling Laws for Online Advertisement Retrieval

A Resource Model For Neural Scaling Law

Scaling Retrieval-Based Language Models with a Trillion-Token Datastore

Observational Scaling Laws and the Predictability of Language Model Performance

Revisiting Neural Scaling Laws in Language and Vision

A Solvable Model of Neural Scaling Laws

Scaling Laws for Linear Complexity Language Models

Scaling Laws for Transfer

Explaining Neural Scaling Laws

Neural Scaling Laws Rooted in the Data Distribution

A Hitchhiker's Guide to Scaling Law Estimation

ScaleNet: Searching for the Model to Scale.

Unified Neural Network Scaling Laws and Scale-time Equivalence

Large Language Models as Foundations for Next-Gen Dense Retrieval: A Comprehensive Empirical Assessment

Scaling Graph Neural Networks for Large-Scale Power Systems Analysis: Empirical Laws for Emergent Abilities