Abstract:Background: The rise of depression, anxiety, and suicide rates has led to increased demand for telemedicine-based mental health screening and remote patient monitoring (RPM) solutions to alleviate the burden on, and enhance the efficiency of, mental health practitioners. Multimodal dialog systems (MDS) that conduct on-demand, structured interviews offer a scalable and cost-effective solution to address this need. Objective: This study evaluates the feasibility of a cloud based MDS agent, Tina, for mental state characterization in participants with depression, anxiety, and suicide risk. Method: Sixty-eight participants were recruited through an online health registry and completed 73 sessions, with 15 (20.6%), 21 (28.8%), and 26 (35.6%) sessions screening positive for depression, anxiety, and suicide risk, respectively using conventional screening instruments. Participants then interacted with Tina as they completed a structured interview designed to elicit calibrated, open-ended responses regarding the participants' feelings and emotional state. Simultaneously, the platform streamed their speech and video recordings in real-time to a HIPAA-compliant cloud server, to compute speech, language, and facial movement-based biomarkers. After their sessions, participants completed user experience surveys. Machine learning models were developed using extracted features and evaluated with the area under the receiver operating characteristic curve (AUC). Results: For both depression and suicide risk, affected individuals tended to have a higher percent pause time, while those positive for anxiety showed reduced lip movement relative to healthy controls. In terms of single-modality classification models, speech features performed best for depression (AUC = 0.64; 95% CI = 0.51-0.78), facial features for anxiety (AUC = 0.57; 95% CI = 0.43-0.71), and text features for suicide risk (AUC = 0.65; 95% CI = 0.52-0.78). Best overall performance was achieved by decision fusion of all models in identifying suicide risk (AUC = 0.76; 95% CI = 0.65-0.87). Participants reported the experience comfortable and shared their feelings. Conclusion: MDS is a feasible, useful, effective, and interpretable solution for RPM in real-world clinical depression, anxiety, and suicidal populations. Facial information is more informative for anxiety classification, while speech and language are more discriminative of depression and suicidality markers. In general, combining speech, language, and facial information improved model performance on all classification tasks.

Real Time Multimodal Depression Analysis

Multi-Modal and Multi-Task Depression Detection with Sentiment Assistance

Automatic Assessment of Depression from Speech Via a Hierarchical Attention Transfer Network and Attention Autoencoders

Multimodal Depression Severity Prediction from medical bio-markers using Machine Learning Tools and Technologies

End-to-end multimodal system for depression detection from online recordings

A Multimodal Approach for Detection and Assessment of Depression Using Text, Audio and Video

D-ResNet-PVKELM: deep neural network and paragraph vector based kernel extreme machine learning model for multimodal depression analysis

Diagnosis of depression level using multimodal approaches using deep learning techniques with multiple selective features

Depression Scale Recognition from Audio, Visual and Text Analysis

Multimodal Mental Health Digital Biomarker Analysis from Remote Interviews using Facial, Vocal, Linguistic, and Cardiovascular Patterns

Cross-Silo, Privacy-Preserving, and Lightweight Federated Multimodal System for the Identification of Major Depressive Disorder Using Audio and Electroencephalogram

Multimodal Measurement of Depression Using Deep Learning Models

Depression Status Estimation by Deep Learning based Hybrid Multi-Modal Fusion Model

Multimodal Prediction of Affective Dimensions and Depression in Human-Computer Interactions

Unaligned Multimodal Sequences for Depression Assessment From Speech

Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

A multimodal dialog approach to mental state characterization in clinically depressed, anxious, and suicidal populations

Using Multimodal Data Collection System as a Research Tool in the Major Depressive Disorder Analysis: a cross-sectional study protocol

Textual-dominated Multimodal Depression Detection

Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalities

Automated Detection of Depression using Wavelet Scattering Networks