Machine Learning for Multimodal Mental Health Detection: A Systematic Review of Passive Sensing Approaches

Lin Sze Khoo,Mei Kuan Lim,Chun Yong Chong,Roisin McNaney

DOI: https://doi.org/10.3390/s24020348

IF: 3.9

2024-01-07

Sensors

Abstract:As mental health (MH) disorders become increasingly prevalent, their multifaceted symptoms and comorbidities with other conditions introduce complexity to diagnosis, posing a risk of underdiagnosis. While machine learning (ML) has been explored to mitigate these challenges, we hypothesized that multiple data modalities support more comprehensive detection and that non-intrusive collection approaches better capture natural behaviors. To understand the current trends, we systematically reviewed 184 studies to assess feature extraction, feature fusion, and ML methodologies applied to detect MH disorders from passively sensed multimodal data, including audio and video recordings, social media, smartphones, and wearable devices. Our findings revealed varying correlations of modality-specific features in individualized contexts, potentially influenced by demographics and personalities. We also observed the growing adoption of neural network architectures for model-level fusion and as ML algorithms, which have demonstrated promising efficacy in handling high-dimensional features while modeling within and cross-modality relationships. This work provides future researchers with a clear taxonomy of methodological approaches to multimodal detection of MH disorders to inspire future methodological advancements. The comprehensive analysis also guides and supports future researchers in making informed decisions to select an optimal data source that aligns with specific use cases based on the MH disorder of interest.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is the deficiencies in the current diagnosis of mental health disorders, especially the diagnostic difficulties and the risk of missed diagnosis caused by complex symptoms and comorbidities. Specifically, the author focuses on the following points:

1. **The necessity of multi - modal data**: Since the symptoms of mental health disorders are multifaceted, involving multiple levels such as emotion, behavior, and body, single - modal data may not be sufficient to comprehensively capture these complex symptoms. Therefore, the author hypothesizes that multi - modal data can provide a more comprehensive detection effect.

2. **The advantages of non - invasive data collection**: Traditional diagnostic methods rely on patients' subjective recall and clinicians' judgment, which may lead to inaccurate diagnosis. Non - invasive data collection methods can naturally capture users' behaviors through commonly used devices in daily life (such as smart phones, wearable devices, etc.), reducing interference in daily life, thereby improving the authenticity and reliability of data.

3. **The application of machine learning in multi - modal mental health detection**: Although previous studies have attempted to use machine learning techniques to improve the detection of mental health disorders, most of these studies have focused on single - modal data. The author hopes to systematically review the existing literature, evaluate the application of multi - modal data in machine learning, especially feature extraction, feature fusion, and the selection of machine learning algorithms, in order to propose more effective detection methods.

In summary, this paper aims to systematically review the existing multi - modal machine learning methods, evaluate the effectiveness of different data sources and methods, thereby providing guidance for future research and promoting the development of more accurate mental health disorder detection methods.