Abstract:Background and objective: Major Depressive Disorder is a highly prevalent and disabling mental health condition. Numerous studies explored multimodal fusion systems combining visual, audio, and textual features via deep learning architectures for clinical depression recognition. Yet, no comparative analysis for multimodal depression analysis has been proposed in the literature. Methods: In this paper, an up-to-date literature overview of multimodal depression recognition is presented and an extensive comparative analysis of different deep learning architectures for depression recognition is performed. First, audio features based Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) are studied. Then, early-level and model-level fusion of deep audio features with visual and textual features through LSTM and CNN architectures are investigated. Results: The performance of the proposed architectures using an hold-out strategy on the DAIC-WOZ dataset (80% training, 10% validation, 10% test split) for binary and severity levels of depression recognition is tested. Using this strategy, a set of experiments have been performed and they have demonstrated: (1) LSTM-based audio features perform slightly better than CNN ones with an accuracy of 66.25% versus 65.60% for binary depression classes. (2) the model level fusion of deep audio and visual features using LSTM network performed the best with an accuracy of 77.16%, a precision of 53% for the depressed class, and a precision of 83% for the non-depressed class. The given network obtained a normalized Root Mean Square Error (RMSE) of 0.15 for depression severity level prediction. Using a Leave-One-Subject-Out strategy, this network achieved an accuracy of 95.38% for binary depression detection, and a normalized RMSE of 0.1476 for depression severity level prediction. Our best-performing architecture outperforms all state-of-the-art approaches on DAIC-WOZ dataset. Conclusions: The obtained results show that the proposed LSTM-based surpass the proposed CNN-based architectures allowing to learn temporal dynamics representations of multimodal features. Furthermore, model-level fusion of audio and visual features using an LSTM network leads to the best performance. Our best-performing architecture successfully detects depression using a speech segment of less than 8 seconds, and an average prediction computation time of less than 6ms; making it suitable for real-world clinical applications.

Multimodal temporal machine learning for Bipolar Disorder and Depression Recognition

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

DISTINGUISHING BIPOLAR DEPRESSION FROM MAJOR DEPRESSIVE DISORDER USING FNIRS AND DEEP NEURAL NETWORK

A Novel Audio-Visual Information Fusion System for Mental Disorders Detection

Multimodal Deep Learning for Mental Disorders Prediction from Audio Speech Samples

Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

End-to-end multimodal clinical depression recognition using deep neural networks: A comparative analysis

Multimodal Physiological Signals Representation Learning via Multiscale Contrasting for Depression Recognition

Multimodal Mental Health Digital Biomarker Analysis from Remote Interviews using Facial, Vocal, Linguistic, and Cardiovascular Patterns

Multimodal Measurement of Depression Using Deep Learning Models

Multimodal Fusion of EEG and Audio Spectrogram for Major Depressive Disorder Recognition Using Modified DenseNet121

Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

A Hybrid Model for Bipolar Disorder Classification from Visual Information

Multimodal Spatiotemporal Representation for Automatic Depression Level Detection

A Multimodal Approach for Automatic Mania Assessment in Bipolar Disorder

Reading Between the Frames: Multi-Modal Depression Detection in Videos from Non-Verbal Cues

End-to-end multimodal system for depression detection from online recordings

A machine learning based depression screening framework using temporal domain features of the electroencephalography signals

Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI

Mental Disorder Classification via Temporal Representation of Text

Unaligned Multimodal Sequences for Depression Assessment From Speech