Abstract:Introduction: Real-time evaluations of the severity of depressive symptoms are of great significance for the diagnosis and treatment of patients with major depressive disorder (MDD). In clinical practice, the evaluation approaches are mainly based on psychological scales and doctor-patient interviews, which are time-consuming and labor-intensive. Also, the accuracy of results mainly depends on the subjective judgment of the clinician. With the development of artificial intelligence (AI) technology, more and more machine learning methods are used to diagnose depression by appearance characteristics. Most of the previous research focused on the study of single-modal data; however, in recent years, many studies have shown that multi-modal data has better prediction performance than single-modal data. This study aimed to develop a measurement of depression severity from expression and action features and to assess its validity among the patients with MDD. Methods: We proposed a multi-modal deep convolutional neural network (CNN) to evaluate the severity of depressive symptoms in real-time, which was based on the detection of patients' facial expression and body movement from videos captured by ordinary cameras. We established behavioral depression degree (BDD) metrics, which combines expression entropy and action entropy to measure the depression severity of MDD patients. Results: We found that the information extracted from different modes, when integrated in appropriate proportions, can significantly improve the accuracy of the evaluation, which has not been reported in previous studies. This method presented an over 74% Pearson similarity between BDD and self-rating depression scale (SDS), self-rating anxiety scale (SAS), and Hamilton depression scale (HAMD). In addition, we tracked and evaluated the changes of BDD in patients at different stages of a course of treatment and the results obtained were in agreement with the evaluation from the scales. Discussion: The BDD can effectively measure the current state of patients' depression and its changing trend according to the patient's expression and action features. Our model may provide an automatic auxiliary tool for the diagnosis and treatment of MDD.

A Robust Method of Bipolar Mental Illness Detection from Facial Micro Expressions Using Machine Learning Methods

DISTINGUISHING BIPOLAR DEPRESSION FROM MAJOR DEPRESSIVE DISORDER USING FNIRS AND DEEP NEURAL NETWORK

A Hybrid Model for Bipolar Disorder Classification from Visual Information

A Deep Quantum Convolutional Neural Network Based Facial Expression Recognition For Mental Health Analysis

A convolutional neural network-based diagnostic method using resting-state electroencephalograph signals for major depressive and bipolar disorders

Convolutional Neural Network-Based Digital Diagnostic Tool for the Identification of Psychosomatic Illnesses

Mental Health State Classification Using Facial Emotion Recognition and Detection

Depression Analysis Using Face Recognition System

Tracking and Detecting Depression Level using Facial Recognition and PEN&IQ Test

Diagnosis of depression level using multimodal approaches using deep learning techniques with multiple selective features

Automatic Assessment Method and Device for Depression Symptom Severity Based on Emotional Facial Expression and Pupil-Wave

Arabic Speech Analysis for Classification and Prediction of Mental Illness due to Depression Using Deep Learning

Emotion Recognition System via Facial Expressions and Speech Using Machine Learning and Deep Learning Techniques

A Mental Illness Detection Model For College Students Based On Body Behavior And Facial Expression Features

Breaking New Ground in Affective Computing: Enhanced Facial Expression Classification via CNN-SVM Integration

Catching Elusive Depression via Facial Micro-Expression Recognition

Measuring depression severity based on facial expression and body movement using deep convolutional neural network

Analysis of facial expressions recorded from patients during psychiatric interviews

FacePsy: An Open-Source Affective Mobile Sensing System -- Analyzing Facial Behavior and Head Gesture for Depression Detection in Naturalistic Settings

Facial emotion recognition in adolescents with bipolar disorder

A Multimodal Approach for Automatic Mania Assessment in Bipolar Disorder