Is Your LLM Outdated? Evaluating LLMs at Temporal Generalization

Chenghao Zhu,Nuo Chen,Yufei Gao,Yunyi Zhang,Prayag Tiwari,Benyou Wang

2024-07-11

Abstract:The rapid advancement of Large Language Models (LLMs) highlights the urgent need for evolving evaluation methodologies that keep pace with improvements in language comprehension and information processing. However, traditional benchmarks, which are often static, fail to capture the continually changing information landscape, leading to a disparity between the perceived and actual effectiveness of LLMs in ever-changing real-world scenarios. Our study examines temporal generalization, which includes the ability to understand, predict, and generate text relevant to past, present, and future contexts, revealing significant temporal biases in LLMs. We propose an evaluation framework, for dynamically generating benchmarks from recent real-world predictions. Experiments demonstrate that LLMs struggle with temporal generalization, showing performance decline over time. These findings highlight the necessity for improved training and updating processes to enhance adaptability and reduce biases. Our code, dataset and benchmark are available at <a class="link-external link-https" href="https://github.com/FreedomIntelligence/FreshBench" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the issue of temporal generalization capabilities in large language models (LLMs). Specifically, with the rapid development of LLMs, traditional evaluation benchmarks can no longer effectively capture the performance of these models in an ever-changing information environment. The paper proposes a dynamic generation benchmark method to evaluate the ability of LLMs to handle past, present, and future texts, revealing significant biases in the temporal generalization of existing models.

The main contributions of the paper include:
1. Defining and quantifying temporal generalization and bias, providing a foundation for understanding the temporal adaptability of LLMs.
2. Proposing the FreshBench benchmark testing framework, which reflects the latest data in a dynamically updated manner, ensuring the accuracy and relevance of evaluation results.
3. Demonstrating through experiments the shortcomings of existing LLMs in temporal generalization, emphasizing the importance of improving training and updating processes to enhance model adaptability and reduce bias.