MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation

Junqing He,Liang Zhu,Rui Wang,Xi Wang,Reza Haffari,Jiaxing Zhang

2024-10-24

Abstract:Long-term memory is important for chatbots and dialogue systems (DS) to create consistent and human-like conversations, evidenced by numerous developed memory-augmented DS (MADS). To evaluate the effectiveness of such MADS, existing commonly used evaluation metrics, like retrieval accuracy and perplexity (PPL), mainly focus on query-oriented factualness and language quality assessment. However, these metrics often lack practical value. Moreover, the evaluation dimensions are insufficient for human-like assessment in DS. Regarding memory-recalling paradigms, current evaluation schemes only consider passive memory retrieval while ignoring diverse memory recall with rich triggering factors, e.g., emotions and surroundings, which can be essential in emotional support scenarios. To bridge the gap, we construct a novel Memory-Augmented Dialogue Benchmark (MADail-Bench) covering various memory-recalling paradigms based on cognitive science and psychology theories. The benchmark assesses two tasks separately: memory retrieval and memory recognition with the incorporation of both passive and proactive memory recall data. We introduce new scoring criteria to the evaluation, including memory injection, emotion support (ES) proficiency, and intimacy, to comprehensively assess generated responses. Results from cutting-edge embedding models and large language models on this benchmark indicate the potential for further advancement. Extensive testing further reveals correlations between memory injection, ES proficiency, and intimacy.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the limitations in evaluating the performance of current Memory - Augmented Dialogue Systems (MADS). Specifically, the existing evaluation methods mainly focus on indicators such as query accuracy and language quality. These indicators often lack practical application value and are not sufficient to comprehensively evaluate the human - like performance of dialogue systems. In addition, the existing evaluation schemes only consider passive memory retrieval and ignore the diverse memory recalls triggered by various factors such as emotions and environments, which is especially important in emotional support scenarios.

To solve these problems, the author constructs a new Memory - Augmented Dialogue Benchmark (MADial - Bench), which covers various memory recall paradigms based on cognitive science and psychology theories. MADial - Bench aims to evaluate two tasks: memory retrieval and memory recognition, and combines passive and active memory recall data. The author also introduces new scoring criteria, including memory injection, emotional support ability and intimacy, to comprehensively evaluate the generated responses. Through this benchmark, researchers hope to more accurately evaluate the actual performance of MADS and promote the development of related technologies.