Machine Learning Facial Emotion Recognition in Psychotherapy Research. A useful approach?

Martin Steppan,Ronan Zimmermann,Lukas Fürer,Nathalie Schenk,Klaus Schmeck
DOI: https://doi.org/10.31234/osf.io/wpa5e
2020-04-20
Abstract:Background: Manual coding of facial emotion expression, e.g.using the Facial Action Coding System (FACS), can be very time consuming. For psychotherapy research the change in emotion expressionand microexpressions over time is relevant. Hence, automatic emotionrecognition may be a promising tool. Method: We apply a Convolutionary Neural Network (CNN) for emotion recognition to video material from 389 psychotherapy sessions of 23 patients with borderline personality pathology. We cross-validate the findings with human ratings accordingto the Clients Emotional Arousal Scale (CEAS) and the outcome of psychotherapy. Results: Overall, machine learning ratings show substantial,however, numerically low agreement with human ratings, particularly with overall (non-specific) emotional arousal. Machine learning emotion recognition shows substantial predictive validity for therapy outcome, in particular the display of positive emotions (smiling and happiness). Discussion: Machine learning is a highly promising resource for tracking change in emotional expression over time. The results highlight thedifferential association of displayed positive and negative feelings to the treatment outcome.
What problem does this paper attempt to address?