Abstract:BackgroundIn recent years, more and more people suffer from depression and anxiety. These symptoms are hard to be spotted and can be very dangerous. Currently, the Self-Reported Anxiety Scale (SAS) and Self-Reported Depression Scale (SDS) are commonly used for initial screening for depression and anxiety disorders. However, the information contained in these two scales is limited, while the symptoms of subjects are various and complex, which results in the inconsistency between the questionnaire evaluation results and the clinician's diagnosis results. To fully mine the scale data, we propose a method to extract the features from the facial expression and movements, which are generated from the video recorded simultaneously when subjects fill in the scale. Then we collect the facial expression, movements and scale information to establish a multimodal framework for improving the accuracy and robustness of the diagnosis of depression and anxiety.MethodsWe collect the scale results of the subjects and the videos when filling in the scales. Given the two scales, SAS and SDS, we construct a model with two branches, where each branch processes the multimodal data of SAS and SDS, respectively. In the branch, we first build a convolutional neural network (CNN) to extracts the facial expression features in each frame of images. Secondly, we establish a long short-term memory (LSTM) network to further embedding the facial expression feature and build the connections between various frames, so that the movement feature in the video can be generated. Thirdly, we transform the scale scores into one-hot format, and feed them into the corresponding branch of the network to further mining the information of the multimodal data. Finally, we fuse the embeddings of these two branches to generate inference results of depression and anxiety.Results and conclusionsBased on the score results of SAS and SDS, our multimodal model further mines the video information, and can reach the accuracy of 0.946 in diagnosing depression and anxiety. This study demonstrates the feasibility of using our CNN-LSTM-based multimodal model for initial screening and diagnosis of depression and anxiety disorders with high diagnostic performance.

A Deep Multiscale Spatiotemporal Network for Assessing Depression from Facial Dynamics

Facial Expression Analysis Using Decomposed Multiscale Spatiotemporal Networks

MDN: A Deep Maximization-Differentiation Network for Spatio-Temporal Depression Detection

Automatic Depression Prediction Via Cross-Modal Attention-Based Multi-Modal Fusion in Social Networks

Dual‐task enhanced global–local temporal–spatial network for depression recognition from facial videos

Depressioner: Facial dynamic representation for automatic depression level prediction

Multi-Scale and Multi-Region Facial Discriminative Representation for Automatic Depression Level Prediction.

TCEDN: A Lightweight Time-Context Enhanced Depression Detection Network

Deep Neural Networks for Depression Recognition Based on 2D and 3D Facial Expressions Under Emotional Stimulus Tasks

Deep 3D-CNN for Depression Diagnosis with Facial Video Recording of Self-Rating Depression Scale Questionnaire

A Multi-Frame Rate Network with Attention Mechanism for Depression Severity Estimation.

Multimodal fusion diagnosis of depression and anxiety based on CNN-LSTM model

FacialPulse: An Efficient RNN-based Depression Detection via Temporal Facial Landmarks

Measuring depression severity based on facial expression and body movement using deep convolutional neural network

DepNet: An automated industrial intelligent system using deep learning for video‐based depression analysis

Visually Interpretable Representation Learning for Depression Recognition from Facial Images

Two-stage Temporal Modelling Framework for Video-based Depression Recognition using Graph Representation

MTDAN: A Lightweight Multi-Scale Temporal Difference Attention Networks for Automated Video Depression Detection

MAST-GCN: Multi-Scale Adaptive Spatial-Temporal Graph Convolutional Network for EEG-Based Depression Recognition

Automated Depression Diagnosis Based on Facial Expression and Deep Dual-Stream Convolutional Neural Networks

Multimodal Measurement of Depression Using Deep Learning Models