Multi-modal Interactive Fusion Method for Detecting Teenagers Psychological Stress.

Huijun Zhang,Lei Cao,Ling Feng,Mo Yang

DOI: https://doi.org/10.1016/j.jbi.2020.103427

IF: 8

2020-01-01

Journal of Biomedical Informatics

Abstract:•Learning and generation of multi-modal feature representations.•Multi-modal interactive fusion for adolescent stress detection.•Multi-modalities enable better performance than singular modality.

What problem does this paper attempt to address?

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Shiyu Teng,Jiaqing Liu,Yue Huang,Shurong Chai,Tomoko Tateyama,Xinyin Huang,Lanfen Lin,Yen-Wei Chen

DOI: https://doi.org/10.1587/transinf.2023hcp0006

2024-01-01

IEICE Transactions on Information and Systems

Abstract:Depression is a prevalent mental disorder affecting a significant portion of the global population, leading to considerable disability and contributing to the overall burden of disease. Consequently, designing efficient and robust automated methods for depression detection has become imperative. Recently, deep learning methods, especially multimodal fusion methods, have been increasingly used in computer-aided depression detection. Importantly, individuals with depression and those without respond differently to various emotional stimuli, providing valuable information for detecting depression. Building on these observations, we propose an intra- and inter-emotional stimulus transformer-based fusion model to effectively extract depression-related features. The intra-emotional stimulus fusion framework aims to prioritize different modalities, capitalizing on their diversity and complementarity for depression detection. The inter-emotional stimulus model maps each emotional stimulus onto both invariant and specific subspaces using individual invariant and specific encoders. The emotional stimulus-invariant subspace facilitates efficient information sharing and integration across different emotional stimulus categories, while the emotional stimulus specific subspace seeks to enhance diversity and capture the distinct characteristics of individual emotional stimulus categories. Our proposed intra- and inter-emotional stimulus fusion model effectively integrates multimodal data under various emotional stimulus categories, providing a comprehensive representation that allows accurate task predictions in the context of depression detection. We evaluate the proposed model on the Chinese Soochow University students dataset, and the results outperform state-of-the-art models in terms of concordance correlation coefficient (CCC), root mean squared error (RMSE) and accuracy.
Automatic Depression Prediction Via Cross-Modal Attention-Based Multi-Modal Fusion in Social Networks

Lidong Wang,Yin Zhang,Bin Zhou,Shihua Cao,Keyong Hu,Yunfei Tan

DOI: https://doi.org/10.1016/j.compeleceng.2024.109413

IF: 4.152

2024-01-01

Computers & Electrical Engineering

Abstract:Depression has become a worldwide public health concern in recent years. The online social network can be served as a tool to predict latent depressed users. However, existing studies primarily use machine learning or deep learning methods on multi-modal information with simple concatenation, resulting in unsatisfactory prediction. Additionally, the lack of interpretability in extracted features is a challenge. Furthermore, the majority of these studies primarily concentrate on English social networks. To address these issues, we propose a novel unified multi-tasking training model named CrossAMF (Cross-modal Attention-based Multi-modal Fusion). CrossAMF extracts crucial multi-modal features with explainability and effectively fuses representations of different modalities. Our approach employs a sentiment filtering-based XLNet-CNN-BiGRU model and Reduce-VGGNet to extract textual and image features, respectively. We also introduce innovative user behavior metrics previously unexplored. We collect a dataset of Chinese depressed users from Sina microblog with their texts and images, including 5692 depressed users and 9121 normal users. Experimental results demonstrate the significance of each modality to our task, with the combination yielding the best performance. CrossAMF outperforms other state-of-the-art methods with 0.9097 in accuracy and 0.8915 in F1, showing the potential of application in screening out early depression automatically.
Multimodal Educational Data Fusion for Students' Mental Health Detection

Teng Guo,Wenhong Zhao,Mubarak Alrashoud,Amr Tolba,Selena Firmin,Feng Xia

DOI: https://doi.org/10.1109/access.2022.3187502

IF: 3.9

2022-07-12

IEEE Access

Abstract:Mental health issues can lead to serious consequences like depression, self-mutilation, and worse, especially for university students who are not physically and mentally mature. Not all students with poor mental health are aware of their situation and actively seek help. Proactive detection of mental problems is a critical step in addressing this issue. However, accurate detections are hard to achieve due to the inherent complexity and heterogeneity of unstructured multi-modal data generated by campus life. Against this background, we propose a detection framework for detecting students' mental health, named CASTLE (educational data fusion for mental health detection). Three parts are involved in this framework. First, we utilize representation learning to fuse data on social life, academic performance, and physical appearance. An algorithm, named MOON (multi-view social network embedding), is proposed to represent students' social life in a comprehensive way by fusing students' heterogeneous social relations effectively. Second, a synthetic minority oversampling technique algorithm (SMOTE) is applied to the label imbalance issue. Finally, a DNN (deep neural network) model is utilized for the final detection. The extensive results demonstrate the promising performance of the proposed methods in comparison to an extensive range of state-of-the-art baselines.

computer science, information systems,telecommunications,engineering, electrical & electronic
Multimodal Stress Detection Using Facial Landmarks and Biometric Signals

Majid Hosseini,Morteza Bodaghi,Ravi Teja Bhupatiraju,Anthony Maida,Raju Gottumukkala

2023-11-07

Abstract:The development of various sensing technologies is improving measurements of stress and the well-being of individuals. Although progress has been made with single signal modalities like wearables and facial emotion recognition, integrating multiple modalities provides a more comprehensive understanding of stress, given that stress manifests differently across different people. Multi-modal learning aims to capitalize on the strength of each modality rather than relying on a single signal. Given the complexity of processing and integrating high-dimensional data from limited subjects, more research is needed. Numerous research efforts have been focused on fusing stress and emotion signals at an early stage, e.g., feature-level fusion using basic machine learning methods and 1D-CNN Methods. This paper proposes a multi-modal learning approach for stress detection that integrates facial landmarks and biometric signals. We test this multi-modal integration with various early-fusion and late-fusion techniques to integrate the 1D-CNN model from biometric signals and 2-D CNN using facial landmarks. We evaluate these architectures using a rigorous test of models' generalizability using the leave-one-subject-out mechanism, i.e., all samples related to a single subject are left out to train the model. Our findings show that late-fusion achieved 94.39\% accuracy, and early-fusion surpassed it with a 98.38\% accuracy rate. This research contributes valuable insights into enhancing stress detection through a multi-modal approach. The proposed research offers important knowledge in improving stress detection using a multi-modal approach.

Computer Vision and Pattern Recognition,Artificial Intelligence
Combining Multiple Sensor Features for Stress Detection Using Combinatorial Fusion

Yong Deng,D. Frank Hsu,Zhonghai Wu,Chao-Hsien Chu

DOI: https://doi.org/10.1142/s0219265912500089

2012-01-01

Journal of Interconnection Networks

Abstract:Physiological sensors have been used to detect different stress levels in order to improve human health and well-being. When analyzing these sensor data, sensor features are generated in the experiment and a subset of the features are selected and then combined using a host of informatics techniques (machine learning, data mining, or information fusion). Our previous work studied feature selection using correlation and diversity as well as feature combination using five methods C4.5, Naïve Bayes, Linear Discriminant Function, Support Vector Machine, and k-Nearest Neighbors. In this paper, we use combinatorial fusion, based on performance criterion (CF-P) and cognitive diversity (CF-CD), to combine those multiple sensor features. Our results showed that: (a) sensor feature combination method is distinctly much better than CF-CD and other algorithms, and (b) CF-CD is as good as other five feature combination methods, and is better in most of the cases.
Sensor Feature Selection and Combination for Stress Identification Using Combinatorial Fusion

Yong Deng,Zhonghai Wu,Chao-Hsien Chu,Qixun Zhang,D. Frank Hsu

DOI: https://doi.org/10.5772/56344

IF: 1.714

2013-01-01

International Journal of Advanced Robotic Systems

Abstract:The identification of stressfulness under certain driving condition is an important issue for safety, security and health. Sensors and systems have been placed or implemented as wearable devices for drivers. Features are extracted from the data collected and combined to predict symptoms. The challenge is to select the feature set most relevant for stress. In this paper, we propose a feature selection method based on the performance and the diversity between two features. The feature sets selected are then combined using a combinatorial fusion. We also compare our results with other combination methods such as naïve Bayes, support vector machine, C4.5, linear discriminant function (LDF), and k-nearest neighbour (kNN). Our experimental results demonstrate that combinatorial fusion is an efficient approach for feature selection and feature combination. It can also improve the stress recognition rate.
Integrating Human Mobility and Social Media for Adolescent Psychological Stress Detection.

Jin Li,Xue Yuanyuan,Li Qi,Feng Ling

DOI: https://doi.org/10.1007/978-3-319-32049-6_23

2016-01-01

Abstract:Leveraging social media to detect psychological stress is an emerging research topic, as it addresses one of the most common mental health issues. One of the notable challenges in this area, however, is data sparsity: users with high stress level tend to reduce their activities on social networks. While teenagers�?mobility behavior always appears some outliers for stress release in physical world, a question arises: can we identify the stress-related outlier features from daily trajectories to facilitate stress detection? In this paper, we propose a co-training-based semi-supervised learning approach that consists of two separated classifiers. One classifier is conditional random field (CRF), which takes outlier features from GPS trajectories as input to model the daily moving behavior correlation of stress. The other classifier is deep neural network (DNN) involving tweet features to model the social media behavior correlation of stress. We evaluate our approach with an over 6-month user study on 57 teenagers from Beijing, and demonstrate effectiveness of the proposed model compared to state-of-the-art methods. © Springer International Publishing Switzerland 2016.
Driver stress detection via multimodal fusion using attention-based CNN-LSTM

Luntian Mou,Chao Zhou,Pengfei Zhao,Bahareh Nakisa,Mohammad Naim Rastgoo,Ramesh Jain,Wen Gao

DOI: https://doi.org/10.1016/j.eswa.2021.114693

IF: 8.5

2021-07-01

Expert Systems with Applications

Abstract:<p>Stress has been identified as one of major contributing factors in car crashes due to its negative impact on driving performance. It is in urgent need that the stress levels of drivers can be detected in real time with high accuracy so that intervening or navigating measures can be taken in time to mitigate the situation. Existing driver stress detection models mainly rely on traditional machine learning techniques to fuse multimodal data. However, due to the non-linear correlations among modalities, it is still challenging for traditional multimodal fusion methods to handle the real-time influx of complex multimodal and high dimensional data, and report drivers' stress levels accurately. To solve this issue, a framework of driver stress detection through multimodal fusion using attention based deep learning techniques is proposed in this paper. Specifically, an attention based convolutional neural networks (CNN) and long short-term memory (LSTM) model is proposed to fuse non-invasive data, including eye data, vehicle data, and environmental data. Then, the proposed model can automatically extract features separately from each modality and give different levels of attention to features from different modalities through self-attention mechanism. To verify the validity of the proposed method, extensive experiments have been carried out on our dataset collected using an advanced driving simulator. Experimental results demonstrate that the performance of the proposed method on driver stress detection outperforms the state-of-the-art models with an average accuracy of 95.5%.</p>

computer science, artificial intelligence,engineering, electrical & electronic,operations research & management science
Multimodal Hierarchical CNN Feature Fusion for Stress Detection

Radhika Kuttala,Ramanathan Subramanian,Venkata Ramana Murthy Oruganti

DOI: https://doi.org/10.1109/access.2023.3237545

IF: 3.9

2023-01-27

IEEE Access

Abstract:Stress is one of the most severe concerns in modern life. High-level stress can create various diseases or loss of focus and productivity at work. Being under stress prevents people from recognizing their stress levels, so early stress detection is essential. Recently, multimodal fusion has enhanced the performance of stress detection models using Deep Learning (DL) techniques. The low, mid, and high-level features of a Convolutional Neural Network (CNN) are discriminative. A comprehensive feature representation can be obtained by fusing all three levels of CNN's features. This study mainly focuses on detecting stress by exploiting these advantages using a multimodal hierarchical CNN feature fusion. The two multimodal physiological signals used in this study are Electrodermal activity (EDA) and Electrocardiogram (ECG). We develop a hierarchical feature set by concatenating multi-level CNN features for each modality. Multimodal fusion on both hierarchical feature sets is performed using the Multimodal Transfer Module (MMTM). The experiments are carried out with raw frequency domain data and the features from the frequency bands to study the effectiveness of both. The model's performance is compared to the different combinations of hierarchical features from low, mid, and high levels. To verify the generalizability, the proposed approach has been evaluated on four benchmark datasets - ASCERTAIN, CLAS, MAUS, and WAUC. The proposed method showed its effectiveness by outperforming existing models by 1-2%, respectively, on frequency band features. It is observed that the hierarchical feature set from all three levels performed better than all other combinations by 2-4%. As a result, this strategy can be a useful addition to stress detection.

computer science, information systems,telecommunications,engineering, electrical & electronic
Adolescent Depression Detection Model Based on Multimodal Data of Interview Audio and Text.

Lei Zhang,Yuanxiao Fan,Jingwen Jiang,Yuchen Li,Wei Zhang

DOI: https://doi.org/10.1142/s0129065722500459

IF: 6.325

2022-01-01

International Journal of Neural Systems

Abstract:Depression is a common mental disease that has a tendency to develop at a younger age. Early detection of depression with psychological intervention may effectively prevent youth suicide. The establishment of the computer-aided model may be efficient for early detection. However, the existing methods of automatic detection for depression mostly rely on unimodal data. Clinical research shows that patients with depression have specificity in speech, text, expression, and other modal data. Multimodal machine learning is emerging but not yet widely used for the detection of psychiatric disorders. The problem of existing multimodal detection models is that only global or local information is considered in feature fusion, which leads to the low accuracy of the depression detection model. Therefore, this study constructs an automatic detection model based on multimodal machine learning for adolescent depression. The proposed method first extracted four features from audio and text globally and locally; then construct a coarse-grained fusion model and fine-grained fusion model base on these four features; and fuse the coarse-grained and the fine-grained fusion model finally. Experiments on the real-world dataset demonstrate that the proposed method could improve the accuracy of depression detection automatically.
Multi-modal Depression Estimation based on Sub-attentional Fusion

Ping-Cheng Wei,Kunyu Peng,Alina Roitberg,Kailun Yang,Jiaming Zhang,Rainer Stiefelhagen

DOI: https://doi.org/10.48550/arXiv.2207.06180

2022-07-13

Computer Vision and Pattern Recognition

Abstract:Failure to timely diagnose and effectively treat depression leads to over 280 million people suffering from this psychological disorder worldwide. The information cues of depression can be harvested from diverse heterogeneous resources, e.g., audio, visual, and textual data, raising demand for new effective multi-modal fusion approaches for automatic estimation. In this work, we tackle the task of automatically identifying depression from multi-modal data and introduce a sub-attention mechanism for linking heterogeneous information while leveraging Convolutional Bidirectional LSTM as our backbone. To validate this idea, we conduct extensive experiments on the public DAIC-WOZ benchmark for depression assessment featuring different evaluation modes and taking gender-specific biases into account. The proposed model yields effective results with 0.89 precision and 0.70 F1-score in detecting major depression and 4.92 MAE in estimating the severity. Our attention-based fusion module consistently outperforms conventional late fusion approaches and achieves competitive performance compared to the previously published depression estimation frameworks, while learning to diagnose the disorder end-to-end and relying on far fewer preprocessing steps.
Dynamic Alignment and Fusion of Multimodal Physiological Patterns for Stress Recognition

Xiaowei Zhang,Xiangyu Wei,Zhongyi Zhou,Qiqi Zhao,Sipo Zhang,Yikun Yang,Rui Li,Bin Hu

DOI: https://doi.org/10.1109/taffc.2023.3290177

IF: 13.99

2023-01-01

IEEE Transactions on Affective Computing

Abstract:Stress has been identified as one of major causes of health issues. To detect the stress levels with higher accuracy, fusion of multimodal physiological signals is a promising technique. However, there is an asynchrony between physiological signals observed from different perspectives. Exploring the temporal alignment relationship between modalities is helpful to improve the quality of multimodal fusion. This paper proposes an end-to-end multimodal stress detection model based on Bidirectional Cross- and Self-modal Attention (BCSA) mechanism. Specifically, we first construct different feature extractors based on the characteristics of Blood Volume Pulse (BVP) and Electrodermal Activity (EDA) to complete automated temporal feature extraction. Secondly, cross-modal attention is used to seek the alignment relationship between the two modalities and fully fuse cross-modal information. The self-modal attention is used to attenuate noise and redundant information, highlight important information and obtain salient stress representations. Finally, the stress representations of the two modalities are processed separately, and the mean square error (MSE) is used to narrow the gap between them. Experimental results on the UBFC-Phys dataset and WESAD dataset show that the proposed model can effectively improve the accuracy of stress recognition, and outperforms several state-of-the-art methods.

computer science, cybernetics, artificial intelligence
A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention

Shengjie Li,Yinhao Xiao

2024-07-02

Abstract:Depression, a prevalent and serious mental health issue, affects approximately 3.8\% of the global population. Despite the existence of effective treatments, over 75\% of individuals in low- and middle-income countries remain untreated, partly due to the challenge in accurately diagnosing depression in its early stages. This paper introduces a novel method for detecting depression based on multi-modal feature fusion utilizing cross-attention. By employing MacBERT as a pre-training model to extract lexical features from text and incorporating an additional Transformer module to refine task-specific contextual understanding, the model's adaptability to the targeted task is enhanced. Diverging from previous practices of simply concatenating multimodal features, this approach leverages cross-attention for feature integration, significantly improving the accuracy in depression detection and enabling a more comprehensive and precise analysis of user emotions and behaviors. Furthermore, a Multi-Modal Feature Fusion Network based on Cross-Attention (MFFNC) is constructed, demonstrating exceptional performance in the task of depression identification. The experimental results indicate that our method achieves an accuracy of 0.9495 on the test dataset, marking a substantial improvement over existing approaches. Moreover, it outlines a promising methodology for other social media platforms and tasks involving multi-modal processing. Timely identification and intervention for individuals with depression are crucial for saving lives, highlighting the immense potential of technology in facilitating early intervention for mental health issues.

Computation and Language,Artificial Intelligence,Machine Learning
Multi Fine-Grained Fusion Network for Depression Detection

Li Zhou,Zhenyu Liu,Yutong Li,Yuchi Duan,Huimin Yu,Bin Hu

DOI: https://doi.org/10.1145/3665247

2024-06-29

Abstract:Depression is an illness that involves emotional and mental health. Currently, depression detection through interviews is the most popular way. With the advancement of natural language processing and sentiment analysis, automated interview-based depression detection is strongly supported. However, current multimodal depression detection models fail to adequately capture the fine-grained features of depressive behaviors, making it difficult for the models to accurately characterize the subtle changes in depressive symptoms. To address this problem, we propose a Multi Fine-Grained Fusion Network (MFFNet). The core idea of this model is to extract and fuse the information of different scale feature pairs through a Multi-Scale Fastformer (MSfastformer), and then use the Recurrent Pyramid Model to integrate the features of different resolutions, promoting the interaction of multi-level information. Through the interaction of multi-scale and multi-resolution features, it aims to explore richer feature representations. To validate the effectiveness of our proposed MFFNet model, we conduct experiments on two depression interview datasets. The experimental results show that the MFFNet model performs better in depression detection compared to other benchmark multimodal models.

computer science, information systems, theory & methods, software engineering
A Multi-modal Feature Layer Fusion Model for Assessment of Depression Based on Attention Mechanisms

Congcong Wang,Kemeng Tao,Yuefeng Zhao,Decheng Liu,Zhi Liu,Xiaoxiao Cui,Gongtang Wang

DOI: https://doi.org/10.1109/CISP-BMEI56279.2022.9979894

2022-11-05

Abstract:Depression is a common psychiatric disorder that can lead to depressed moods and even suicidal behavior. Intelligent assessment of depression from multiple physiological and behavioral data and breaking the limitations of traditional methods is the focus of this research area. In this paper, a multi-modal fusion depression assessment model based on attention mechanisms is proposed to predict the severity of depression from visual, acoustic and text modalities. By training and testing on an improved dataset, the proposed multi-modal feature layer fusion model based on attention mechanisms (MFF-Att) is validated to be superior to unimodal prediction models and achieved good results in depression assessment. The root mean square error (RMSE) and mean absolute error (MAE) of the proposed model on the development set are 4.03 and 3.05, respectively, which are better than the baseline and state-of-the-art results.

Psychology,Computer Science
A Mental Stress Classification Method Based on Feature Fusion Using Physiological Signals

Ming Sun,Xuanmeng Cao

DOI: https://doi.org/10.1142/s0218126624500166

2024-01-01

Journal of Circuits Systems and Computers

Abstract:Mental stress can cause a range of mental health issues, which makes it challenging to develop a stress classification method especially based on physiological signals. Although cutting-edge deep learning models are currently popular for recognizing mental stress, most frameworks rely solely on deep features, which may not provide a comprehensive understanding of physiological signals. In response to this concern, we propose a mental stress classification method that uses feature fusion. We integrate the squeeze-excitation attention mechanism and voting classifier technique to learn detailed and typical information about mental stress. To be more precise, the feature fusion segment consists of two steps: we first extract shallow statistic features and deep features separately from raw signal recordings, and the deep features are dimensionally reduced using principal component analysis to enable better integration with the shallow features. We then flatten both kinds of features and concatenate them by column to create a combined set that contains more salient information about physiological signals. Our experiments show that the attention mechanism and voting classifier technique improve the accuracy of stress classification. Furthermore, our proposed model based on feature fusion achieves remarkable performance compared to state-of-the-art methods.
SFF-DA: Sptialtemporal Feature Fusion for Detecting Anxiety Nonintrusively

Haimiao Mo,Yuchen Li,Shanlin Yang,Wei Zhang,Shuai Ding

DOI: https://doi.org/10.48550/arXiv.2208.06411

2023-03-09

Abstract:Early detection of anxiety is crucial for reducing the suffering of individuals with mental disorders and improving treatment outcomes. Utilizing an mHealth platform for anxiety screening can be particularly practical in improving screening efficiency and reducing costs. However, the effectiveness of existing methods has been hindered by differences in mobile devices used to capture subjects' physical and mental evaluations, as well as by the variability in data quality and small sample size problems encountered in real-world settings. To address these issues, we propose a framework with spatiotemporal feature fusion for detecting anxiety nonintrusively. We use a feature extraction network based on a 3D convolutional network and long short-term memory ("3DCNN+LSTM") to fuse the spatiotemporal features of facial behavior and noncontact physiology, which reduces the impact of uneven data quality. Additionally, we design a similarity assessment strategy to address the issue of deteriorating model accuracy due to small sample sizes. Our framework is validated with a crew dataset from the real world and two public datasets: the University of Burgundy Franche-Comté Psychophysiological (UBFC-Phys) dataset and the Smart Reasoning for Well-being at Home and at Work for Knowledge Work (SWELL-KW) dataset. The experimental results indicate that our framework outperforms the comparison methods.

Computer Vision and Pattern Recognition,Image and Video Processing
Integrating Content-Semantics-World Knowledge to Detect Stress from Videos

Yang Ding,Yi Dai,Xin Wang,Ling Feng,Lei Cao,Huijun Zhang

DOI: https://doi.org/10.1145/3664647.3680584

2024-01-01

Abstract:Stress has rapidly emerged as a significant public health concern in the contemporary society, necessitating prompt identification and effective intervention strategies. Video-based stress detection offers a non-invasive, low-cost, and mass-reaching approach for identifying stress. In this paper, we propose a three-level content-semantic-world knowledge framework, addressing three particular issues for video-based stress detection. (1) How to abstract and encode video semantics with frame contents into visual representation? (2) How to leverage general-purpose LMMs to augment task-specific visual representation? (3) To what extent could general-purpose LMMs contribute to video-based stress detection? We design a Slow-Emotion-Fast-Action scheme to encode fast temporal changes of body actions revealed from video frames, as well as subtle details of emotions per video segment, into visual representation. We augment task-specific visual representation with linguistic facial expression descriptions by prompting general-purpose Large Multimodal Models (LMMs). A knowledge retriever is designed to evaluate and select the most proper deliverable of LMMs. Experimental results on two video-based stress detection datasets show that 1) our proposed three-level framework can achieve 90.89% F1-score in UVSD dataset and 80.79% F1-score, outperforming state-of-the-art; 2) leveraging LMMs helps to improve the F1-score by 2.25% in UVSD and 3.55% in RSL, compared to using the traditional Facial Action Coding System; 3) purely relying on general-purpose LMMs is insufficient with 88.73% F1-score in UVSD dataset and 77.48% F1-score in RSL dataset, demonstrating the necessity to combine task-specific dedicated solutions with world knowledge given by LMMs.
A Multimodal Late Fusion Framework for Physiological Sensor and Audio-Signal-Based Stress Detection: An Experimental Study and Public Dataset

Vasileios-Rafail Xefteris,Monica Dominguez,Jens Grivolla,Athina Tsanousa,Francesco Zaffanela,Martina Monego,Spyridon Symeonidis,Sotiris Diplaris,Leo Wanner,Stefanos Vrochidis,Ioannis Kompatsiaris

DOI: https://doi.org/10.3390/electronics12234871

IF: 2.9

2023-12-03

Electronics

Abstract:Stress can be considered a mental/physiological reaction in conditions of high discomfort and challenging situations. The levels of stress can be reflected in both the physiological responses and speech signals of a person. Therefore the study of the fusion of the two modalities is of great interest. For this cause, public datasets are necessary so that the different proposed solutions can be comparable. In this work, a publicly available multimodal dataset for stress detection is introduced, including physiological signals and speech cues data. The physiological signals include electrocardiograph (ECG), respiration (RSP), and inertial measurement unit (IMU) sensors equipped in a smart vest. A data collection protocol was introduced to receive physiological and audio data based on alterations between well-known stressors and relaxation moments. Five subjects participated in the data collection, where both their physiological and audio signals were recorded by utilizing the developed smart vest and audio recording application. In addition, an analysis of the data and a decision-level fusion scheme is proposed. The analysis of physiological signals includes a massive feature extraction along with various fusion and feature selection methods. The audio analysis comprises a state-of-the-art feature extraction fed to a classifier to predict stress levels. Results from the analysis of audio and physiological signals are fused at a decision level for the final stress level detection, utilizing a machine learning algorithm. The whole framework was also tested in a real-life pilot scenario of disaster management, where users were acting as first responders while their stress was monitored in real time.

engineering, electrical & electronic,computer science, information systems,physics, applied
Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

Zhang,Zhang,Ni,Wei,Yang,Jin,Huang,Liang,Zhang,Li,Ding,Zhang,Wang

DOI: https://doi.org/10.3390/s24123714

IF: 3.9

2024-06-08

Sensors

Abstract:Depression is a major psychological disorder with a growing impact worldwide. Traditional methods for detecting the risk of depression, predominantly reliant on psychiatric evaluations and self-assessment questionnaires, are often criticized for their inefficiency and lack of objectivity. Advancements in deep learning have paved the way for innovations in depression risk detection methods that fuse multimodal data. This paper introduces a novel framework, the Audio, Video, and Text Fusion-Three Branch Network (AVTF-TBN), designed to amalgamate auditory, visual, and textual cues for a comprehensive analysis of depression risk. Our approach encompasses three dedicated branches—Audio Branch, Video Branch, and Text Branch—each responsible for extracting salient features from the corresponding modality. These features are subsequently fused through a multimodal fusion (MMF) module, yielding a robust feature vector that feeds into a predictive modeling layer. To further our research, we devised an emotion elicitation paradigm based on two distinct tasks—reading and interviewing—implemented to gather a rich, sensor-based depression risk detection dataset. The sensory equipment, such as cameras, captures subtle facial expressions and vocal characteristics essential for our analysis. The research thoroughly investigates the data generated by varying emotional stimuli and evaluates the contribution of different tasks to emotion evocation. During the experiment, the AVTF-TBN model has the best performance when the data from the two tasks are simultaneously used for detection, where the F1 Score is 0.78, Precision is 0.76, and Recall is 0.81. Our experimental results confirm the validity of the paradigm and demonstrate the efficacy of the AVTF-TBN model in detecting depression risk, showcasing the crucial role of sensor-based data in mental health detection.

engineering, electrical & electronic,instruments & instrumentation,chemistry, analytical

Multi-modal Interactive Fusion Method for Detecting Teenagers Psychological Stress.

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Automatic Depression Prediction Via Cross-Modal Attention-Based Multi-Modal Fusion in Social Networks

Multimodal Educational Data Fusion for Students' Mental Health Detection

Multimodal Stress Detection Using Facial Landmarks and Biometric Signals

Combining Multiple Sensor Features for Stress Detection Using Combinatorial Fusion

Sensor Feature Selection and Combination for Stress Identification Using Combinatorial Fusion

Integrating Human Mobility and Social Media for Adolescent Psychological Stress Detection.

Driver stress detection via multimodal fusion using attention-based CNN-LSTM

Multimodal Hierarchical CNN Feature Fusion for Stress Detection

Adolescent Depression Detection Model Based on Multimodal Data of Interview Audio and Text.

Multi-modal Depression Estimation based on Sub-attentional Fusion

Dynamic Alignment and Fusion of Multimodal Physiological Patterns for Stress Recognition

A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention

Multi Fine-Grained Fusion Network for Depression Detection

A Multi-modal Feature Layer Fusion Model for Assessment of Depression Based on Attention Mechanisms

A Mental Stress Classification Method Based on Feature Fusion Using Physiological Signals

SFF-DA: Sptialtemporal Feature Fusion for Detecting Anxiety Nonintrusively

Integrating Content-Semantics-World Knowledge to Detect Stress from Videos

A Multimodal Late Fusion Framework for Physiological Sensor and Audio-Signal-Based Stress Detection: An Experimental Study and Public Dataset

Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data