Emotion Recognition Under Sleep Deprivation Using A Multimodal Residual Lstm Network

Le-Yan Tao,Bao-Liang Lu
DOI: https://doi.org/10.1109/IJCNN48605.2020.9206957
2020-01-01
Abstract:Emotion recognition under sleep deprivation is instructive for the study of mental disorders such as major depressive disorder. Previous studies on emotion recognition under sleep deprivation have been mainly based on psychological research techniques. In this paper, we introduce a multilayer weight sharing multimodal residual LSTM network for emotion recognition under sleep deprivation. The advantage of our proposed method is that it allows for the combination of three different features: the electroencephalography (EEG) single-channel differential entropy (DE) features, EEG functional strength features with topological correlation connectivity, and eye movement features. The experiments under the conditions of sleep deprivation, sleep recovery and baseline are designed and conducted. The experimental results demonstrate that the proposed method significantly enhances the performance compared with the simple concatenation of the features of different modalities, and the best mean accuracies of 86.86% and 82.03% are achieved for four emotions (happiness, sadness, fear, and neutral) in subject dependent and cross-subject emotion recognition tasks under 30 hours of sleep deprivation, respectively. The classification accuracy of the happiness emotion is obviously impaired under sleep deprivation, indicating that sleep deprivation impairs the stimulation of the happiness emotion, and one night of sleep recovery can reactivate the elicitation of the happiness emotion to the baseline level. Furthermore, we study the brain neural patterns of the four emotional states. The prefrontal area becomes less activated for the happiness emotion and sadness emotion in the gamma band under sleep deprivation, while the neural pattern of the fear emotion is highly robust with respect to sleep deprivation.
What problem does this paper attempt to address?