Abstract:Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue to play a vital role in both research and daily use, their evaluation becomes increasingly critical, not only at the task level, but also at the society level for better understanding of their potential risks. Over the past years, significant efforts have been made to examine LLMs from various perspectives. This paper presents a comprehensive review of these evaluation methods for LLMs, focusing on three key dimensions: what to evaluate , where to evaluate , and how to evaluate . Firstly, we provide an overview from the perspective of evaluation tasks, encompassing general natural language processing tasks, reasoning, medical usage, ethics, education, natural and social sciences, agent applications, and other areas. Secondly, we answer the ‘where’ and ‘how’ questions by diving into the evaluation methods and benchmarks, which serve as crucial components in assessing the performance of LLMs. Then, we summarize the success and failure cases of LLMs in different tasks. Finally, we shed light on several future challenges that lie ahead in LLMs evaluation. Our aim is to offer invaluable insights to researchers in the realm of LLMs evaluation, thereby aiding the development of more proficient LLMs. Our key point is that evaluation should be treated as an essential discipline to better assist the development of LLMs. We consistently maintain the related open-source materials at: https://github.com/MLGroupJLU/LLM-eval-survey.

A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems

MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues

Recent Advances in Speech Language Models: A Survey

Leveraging LLMs for Dialogue Quality Measurement

Advances in Multi-turn Dialogue Comprehension: A Survey

Large Language Model based Situational Dialogues for Second Language Learning

A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontiers

MM-LLMs: Recent Advances in MultiModal Large Language Models

A Survey on Speech Large Language Models

Exploring the Dialogue Comprehension Ability of Large Language Models

Attacks, Defenses and Evaluations for LLM Conversation Safety: A Survey

Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions

Speak Out of Turn: Safety Vulnerability of Large Language Models in Multi-turn Dialogue

Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

A Survey on Evaluation of Large Language ModelsJust Accepted

A Survey on Evaluation of Large Language Models

Balancing Accuracy and Efficiency in Multi-Turn Intent Classification for LLM-Powered Dialog Systems in Production

Are LLMs All You Need for Task-Oriented Dialogue?