Abstract:Introduction: Real-time evaluations of the severity of depressive symptoms are of great significance for the diagnosis and treatment of patients with major depressive disorder (MDD). In clinical practice, the evaluation approaches are mainly based on psychological scales and doctor-patient interviews, which are time-consuming and labor-intensive. Also, the accuracy of results mainly depends on the subjective judgment of the clinician. With the development of artificial intelligence (AI) technology, more and more machine learning methods are used to diagnose depression by appearance characteristics. Most of the previous research focused on the study of single-modal data; however, in recent years, many studies have shown that multi-modal data has better prediction performance than single-modal data. This study aimed to develop a measurement of depression severity from expression and action features and to assess its validity among the patients with MDD. Methods: We proposed a multi-modal deep convolutional neural network (CNN) to evaluate the severity of depressive symptoms in real-time, which was based on the detection of patients' facial expression and body movement from videos captured by ordinary cameras. We established behavioral depression degree (BDD) metrics, which combines expression entropy and action entropy to measure the depression severity of MDD patients. Results: We found that the information extracted from different modes, when integrated in appropriate proportions, can significantly improve the accuracy of the evaluation, which has not been reported in previous studies. This method presented an over 74% Pearson similarity between BDD and self-rating depression scale (SDS), self-rating anxiety scale (SAS), and Hamilton depression scale (HAMD). In addition, we tracked and evaluated the changes of BDD in patients at different stages of a course of treatment and the results obtained were in agreement with the evaluation from the scales. Discussion: The BDD can effectively measure the current state of patients' depression and its changing trend according to the patient's expression and action features. Our model may provide an automatic auxiliary tool for the diagnosis and treatment of MDD.

Automated Depression Diagnosis Based on Facial Expression and Deep Dual-Stream Convolutional Neural Networks

Automatic Diagnosis of Depression Based on Facial Expression Information and Deep Convolutional Neural Network

Dynamic Facial Features in Positive-Emotional Speech for Identification of Depressive Tendencies

Automatic Depression Prediction Via Cross-Modal Attention-Based Multi-Modal Fusion in Social Networks

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Multi-Modal and Multi-Task Depression Detection with Sentiment Assistance

Towards Automatic Depression Detection: A BiLSTM/1D CNN-Based Model

Automatic diagnosis of depression based on attention mechanism and feature pyramid model

Measuring depression severity based on facial expression and body movement using deep convolutional neural network

Automatic identification of depressive symptoms in college students: an application of deep learning-based CNN (Convolutional Neural Network)

Non-verbal Facial Action Units-based Automatic Depression Classification

Depressioner: Facial dynamic representation for automatic depression level prediction

DepNet: An automated industrial intelligent system using deep learning for video‐based depression analysis

Dual‐task enhanced global–local temporal–spatial network for depression recognition from facial videos

Visually Interpretable Representation Learning for Depression Recognition from Facial Images

Depression Recognition Based on Dynamic Facial and Vocal Expression Features Using Partial Least Square Regression

Deep 3D-CNN for Depression Diagnosis with Facial Video Recording of Self-Rating Depression Scale Questionnaire

Facial action units guided graph representation learning for multimodal depression detection

Encoding Visual Behaviors with Attentive Temporal Convolution for Depression Prediction

Dual Attention and Element Recalibration Networks for Automatic Depression Level Prediction

Multi-Scale and Multi-Region Facial Discriminative Representation for Automatic Depression Level Prediction.