Modeling User Satisfaction Dynamics in Dialogue via Hawkes Process

Fanghua Ye,Zhiyuan Hu,Emine Yilmaz

2023-05-22

Abstract:Dialogue systems have received increasing attention while automatically evaluating their performance remains challenging. User satisfaction estimation (USE) has been proposed as an alternative. It assumes that the performance of a dialogue system can be measured by user satisfaction and uses an estimator to simulate users. The effectiveness of USE depends heavily on the estimator. Existing estimators independently predict user satisfaction at each turn and ignore satisfaction dynamics across turns within a dialogue. In order to fully simulate users, it is crucial to take satisfaction dynamics into account. To fill this gap, we propose a new estimator ASAP (sAtisfaction eStimation via HAwkes Process) that treats user satisfaction across turns as an event sequence and employs a Hawkes process to effectively model the dynamics in this sequence. Experimental results on four benchmark dialogue datasets demonstrate that ASAP can substantially outperform state-of-the-art baseline estimators.

Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to more effectively evaluate User Satisfaction Estimation (USE) in dialogue systems. Specifically, existing user satisfaction estimation methods usually predict user satisfaction for each dialogue turn independently, ignoring the dynamic changes in user satisfaction during the dialogue process. This omission results in existing methods being unable to fully simulate the actual user experience, thus affecting the accurate evaluation of dialogue system performance.

To fill this gap, the authors propose a new user satisfaction estimator - ASAP (Satisfaction Estimation via Hawkes Process). This method views the user's satisfaction throughout the entire dialogue process as a series of events and uses the Hawkes process to effectively model the dynamic changes in these event sequences, thereby more accurately capturing the trend of changes in user satisfaction. The Hawkes process is a self - exciting point process that can well describe the mutual excitation relationship between events and is particularly suitable for modeling the pattern of changes in user satisfaction with the progress of the dialogue.

The main contributions of the paper include:
- Proposing a basic satisfaction estimator to predict user satisfaction only based on the dialogue context.
- Introducing a Hawkes process module to model the dynamic changes in user satisfaction by regarding the satisfaction scores of different turns as an event sequence.
- Discretizing the continuous Hawkes process to adapt to the USE task and implementing this module using the Transformer architecture.
- Conducting extensive experiments on four benchmark dialogue datasets, and the results show that ASAP significantly outperforms existing baseline methods.

Through this method, the paper not only improves the accuracy of user satisfaction estimation but also provides a new perspective for the automatic evaluation of dialogue systems, which helps developers to timely detect the defects of dialogue systems and continuously optimize system performance according to user feedback.