Abstract:Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate , where to evaluate , and how to evaluate . Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, education, natural and social sciences, agent applications, and other areas. Secondly, we answer the ‘where’ and ‘how’ questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing the performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey.

A Closer Look into Using Large Language Models for Automatic Evaluation

Can Large Language Models Be an Alternative to Human Evaluations?

LLMEval: A Preliminary Study on How to Evaluate Large Language Models

Leveraging Large Language Models for NLG Evaluation: Advances and Challenges

A Comprehensive Analysis of the Effectiveness of Large Language Models As Automatic Dialogue Evaluators

Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study

Large Language Model as an Assignment Evaluator: Insights, Feedback, and Challenges in a 1000+ Student Course

CriticEval: Evaluating Large Language Model as Critic

Large Language Models Are State-of-the-Art Evaluator for Grammatical Error Correction

Rethinking Generative Large Language Model Evaluation for Semantic Comprehension

Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysis

A Survey on Evaluation of Large Language Models

Large Language Models Are Active Critics in NLG Evaluation

Evaluating Large Language Models at Evaluating Instruction Following

Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Tasks

A Survey on Evaluation of Large Language ModelsJust Accepted

A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks

Collaborative Evaluation: Exploring the Synergy of Large Language Models and Humans for Open-ended Generation Evaluation

Style Over Substance: Evaluation Biases for Large Language Models

Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization