Abstract:Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate, where to evaluate, and how to evaluate. Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, educations, natural and social sciences, agent applications, and other areas. Secondly, we answer the `where' and `how' questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: <a class="link-external link-https" href="https://github.com/MLGroupJLU/LLM-eval-survey" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

The paper primarily explores the evaluation methods for Large Language Models (LLMs) and provides a comprehensive review of existing evaluation research. Its core objective is to address the following issues: 1. **Clarify the dimensions of evaluation**: The paper proposes three key dimensions to consider when evaluating LLMs—"what to evaluate," "where to evaluate," and "how to evaluate." This helps to systematically understand all aspects of the evaluation process. 2. **Cover a wide range of evaluation tasks**: Evaluate LLMs from various aspects including natural language processing tasks, reasoning abilities, multilingual support, to ethical considerations. These tasks encompass the performance of LLMs in different application scenarios. 3. **Summarize existing evaluation datasets and benchmarks**: By summarizing the available datasets and benchmark tests, the paper provides researchers with reference standards for assessing the performance of LLMs. 4. **Explore evaluation methods**: Discusses different evaluation criteria and techniques, including automatic and human evaluations, to ensure the accuracy and reliability of the evaluation results. 5. **Distill successful and unsuccessful cases**: By summarizing the performance of LLMs on different tasks, the paper reveals their strengths and limitations, providing valuable lessons for future research and development. 6. **Look forward to future challenges**: Based on current research progress, the paper also points out future challenges in the field of LLM evaluation, such as designing benchmarks for Artificial General Intelligence (AGI), assessing behavioral integrity, and other issues. In summary, this paper aims to provide a comprehensive framework for the evaluation of LLMs and to offer guidance for researchers, engineers, and professionals in related fields, to promote the further development and refinement of LLM technology.

A Survey on Evaluation of Large Language Models

A Survey on Evaluation of Large Language Models

A Survey on Evaluation of Large Language ModelsJust Accepted

Evaluating Large Language Models: A Comprehensive Survey

Evaluating large language models in medical applications: a survey

LLMEval: A Preliminary Study on How to Evaluate Large Language Models

A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations

A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry

Through the Lens of Core Competency: Survey on Evaluation of Large Language Models

A Survey on Evaluating Large Language Models in Code Generation Tasks

Efficient Large Language Models: A Survey

A Survey on Evaluation of Multimodal Large Language Models

History, Development, and Principles of Large Language Models-An Introductory Survey

A Survey of Large Language Models

A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

Exploring Advanced Methodologies in Security Evaluation for LLMs

Leveraging Large Language Models for NLG Evaluation: Advances and Challenges

A Survey on Large Language Models from General Purpose to Medical Applications: Datasets, Methodologies, and Evaluations

A Survey on Benchmarks of Multimodal Large Language Models

A Survey of Large Language Models in Medicine: Progress, Application, and Challenge

What is the best model? Application-driven Evaluation for Large Language Models