DialogBench: Evaluating LLMs as Human-like Dialogue Systems

Jiao Ou,Junda Lu,Che Liu,Yihong Tang,Fuzheng Zhang,Di Zhang,Kun Gai

2024-03-29

Abstract:Large language models (LLMs) have achieved remarkable breakthroughs in new dialogue capabilities by leveraging instruction tuning, which refreshes human impressions of dialogue systems. The long-standing goal of dialogue systems is to be human-like enough to establish long-term connections with users. Therefore, there has been an urgent need to evaluate LLMs as human-like dialogue systems. In this paper, we propose DialogBench, a dialogue evaluation benchmark that contains 12 dialogue tasks to probe the capabilities of LLMs as human-like dialogue systems should have. Specifically, we prompt GPT-4 to generate evaluation instances for each task. We first design the basic prompt based on widely used design principles and further mitigate the existing biases to generate higher-quality evaluation instances. Our extensive tests on English and Chinese DialogBench of 26 LLMs show that instruction tuning improves the human likeness of LLMs to a certain extent, but most LLMs still have much room for improvement as human-like dialogue systems. Interestingly, results also show that the positioning of assistant AI can make instruction tuning weaken the human emotional perception of LLMs and their mastery of information about human daily life.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem addressed in this paper is the evaluation of large-scale language models (LLMs) as human-like dialogue systems. Existing evaluation benchmarks mainly focus on the task completion capability of LLMs, rather than their ability to establish long-term user connections with sufficient human-likeness. Therefore, the researchers propose a dialogue evaluation benchmark called DialogBench, which contains 12 tasks to probe the human-like dialogue capabilities that LLMs should possess. The experiments show that while fine-tuning instructions can improve the human-likeness of LLMs, there is still significant room for improvement for most models, particularly in emotional perception and everyday knowledge understanding. DialogBench provides in-depth evaluation to enhance the human similarity capability of LLMs.

DialogBench: Evaluating LLMs as Human-like Dialogue Systems

Dialogue Learning with Human-in-the-Loop.

MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues

SimulBench: Evaluating Language Models with Creative Simulation Tasks

On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation

FB-Bench: A Fine-Grained Multi-Task Benchmark for Evaluating LLMs' Responsiveness to Human Feedback

VoiceBench: Benchmarking LLM-Based Voice Assistants

Exploring the Dialogue Comprehension Ability of Large Language Models

Real or Robotic? Assessing Whether LLMs Accurately Simulate Qualities of Human Responses in Dialogue

Leveraging LLMs for Dialogue Quality Measurement

A Comprehensive Analysis of the Effectiveness of Large Language Models As Automatic Dialogue Evaluators

Benchmarking Open-ended Audio Dialogue Understanding for Large Audio-Language Models

HLB: Benchmarking LLMs' Humanlikeness in Language Use

Large Language Model based Situational Dialogues for Second Language Learning

Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue

Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems

Evaluating Human-Language Model Interaction

Benchmarking LLMs for Mimicking Child-Caregiver Language in Interaction

CharacterBench: Benchmarking Character Customization of Large Language Models

DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models

Judging LLM-as-a-judge with MT-Bench and Chatbot Arena