Multi-dimensional Evaluation of Empathetic Dialog Responses

Zhichao Xu,Jiepu Jiang

2024-10-12

Abstract:Empathy is critical for effective and satisfactory conversational communication. Prior efforts to measure conversational empathy mostly focus on expressed communicative intents -- that is, the way empathy is expressed. Yet, these works ignore the fact that conversation is also a collaboration involving both speakers and listeners. In contrast, we propose a multi-dimensional empathy evaluation framework to measure both \emph{expressed intents from the speaker's perspective} and \emph{perceived empathy from the listener's perspective}. We apply our analytical framework to examine internal customer-service dialogues. We find the two dimensions (expressed intent types and perceived empathy) are inter-connected, while perceived empathy has high correlations with dialogue satisfaction levels. To reduce the annotation cost, we explore different options to automatically measure conversational empathy: prompting LLMs and training language model-based classifiers. Our experiments show that prompting methods with even popular models like GPT-4 and Flan family models perform relatively poorly on both public and our internal datasets. In contrast, instruction-finetuned classifiers based on Flan-T5 family models outperform prior works and competitive baselines. We conduct a detailed ablation study to give more insights into instruction finetuning method's strong performance.

Computation and Language

What problem does this paper attempt to address?

The problem this paper attempts to address is: how to comprehensively evaluate empathy in conversations. Existing research mostly focuses on expressed empathy, i.e., how the speaker expresses empathy, while neglecting that conversation is a collaborative process involving both parties. Therefore, this paper proposes a multidimensional empathy evaluation framework aimed at measuring empathy from two aspects: the speaker's expressed communicative intents and the listener's perceived empathy.

Specifically, the framework includes four dimensions of perceived empathy:
1. Perceived Engagement
2. Perceived Understanding
3. Perceived Sympathy
4. Perceived Helpfulness

Through this framework, the authors hope to more comprehensively evaluate empathy in multi-turn conversations and explore the relationship between these dimensions and conversation satisfaction. Additionally, to reduce the cost of manual annotation, the authors also explore different automated methods to measure empathy in conversations, including prompting large language models (LLMs) and training language model-based classifiers. Experimental results show that instruction-tuned classifiers (especially those based on the Flan-T5 model) perform the best, while popular LLMs such as GPT-4 and the Flan series models perform relatively poorly in automatically measuring empathy.