A Bayesian Deep Learning Framework for End-To-End Prediction of Emotion from Heartbeat

Ross Harper,Joshua Southern

DOI: https://doi.org/10.1109/TAFFC.2020.2981610

2020-04-19

Abstract:Automatic prediction of emotion promises to revolutionise human-computer interaction. Recent trends involve fusion of multiple data modalities - audio, visual, and physiological - to classify emotional state. However, in practice, collection of physiological data `in the wild' is currently limited to heartbeat time series of the kind generated by affordable wearable heart monitors. Furthermore, real-world applications of emotion prediction often require some measure of uncertainty over model output, in order to inform downstream decision-making. We present here an end-to-end deep learning model for classifying emotional valence from unimodal heartbeat time series. We further propose a Bayesian framework for modelling uncertainty over these valence predictions, and describe a probabilistic procedure for choosing to accept or reject model output according to the intended application. We benchmarked our framework against two established datasets and achieved peak classification accuracy of 90%. These results lay the foundation for applications of affective computing in real-world domains such as healthcare, where a high premium is placed on non-invasive collection of data, and predictive certainty.

Machine Learning,Human-Computer Interaction

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: **Automatically predict emotional states from unimodal heart rate time series (IBI, i.e., inter - beat interval) and provide uncertainty measures for model outputs**. Specifically, the paper proposes an end - to - end deep - learning framework, aiming to classify emotional valence through heart rate data and introducing a Bayesian framework to model and evaluate the uncertainty of predictions.

### Problem Background
1. **Importance of Emotional Prediction**:
- Automated emotional prediction is expected to revolutionize human - computer interaction.
- Emotion is closely related to cognition, attention, memory, perception, and decision - making.
- Emotional prediction has great potential for application in fields such as healthcare, especially with high requirements for non - invasive data collection and prediction accuracy.

2. **Limitations of Existing Methods**:
- Although multi - modal data fusion (audio, visual, physiological signals) has improved the performance of emotional classification, in practical applications, wearable devices can usually only collect unimodal heart rate time - series data.
- Most of the existing heart - rate - based emotional prediction methods ignore the complex dynamic structures in the time series.
- In many practical application scenarios, the confidence of model outputs is crucial for downstream decision - making, but traditional neural networks lack a description of uncertainty.

### Main Contributions of the Paper
1. **Proposing an End - to - End Deep - Learning Model**:
- Use convolutional neural networks (CNN) and bidirectional long - short - term memory networks (Bi - LSTM) to extract local and global features in the heart rate time series.
- Model the uncertainty of predictions through the Bayesian framework, allowing the criteria for accepting or rejecting predictions to be adjusted according to application requirements.

2. **Solving Challenges in Practical Applications**:
- In view of the fact that only unimodal heart rate data can be obtained in reality, the model input is limited to the IBI time series, simulating the data form of consumer - level wearable devices.
- Provide a statistically meaningful method to adjust the acceptance boundary, ensuring that the confidence of predictions can be flexibly adjusted in different application scenarios.

3. **Experimental Verification**:
- Verification was carried out on two established datasets, AMIGOS and DREAMER, achieving a maximum classification accuracy of 90%.
- By introducing the confidence threshold α, the performance of the model at different confidence levels was demonstrated, proving the clear relationship between model confidence and prediction accuracy.

### Conclusion
This research has developed an emotional prediction model based on the Bayesian deep - learning framework, which can classify emotional valence from unimodal heart rate time series and provide an uncertainty assessment of prediction results. This framework not only improves the accuracy of emotional prediction but also provides reliable confidence support for decision - making in practical applications, especially in fields such as healthcare.