[Fetal phono-electrocardiography. II. Sensitivity of the fetus to some drugs at various periods of pregnancy in physiological and pathological conditions].

M. Vignali

1963-02-28

Minerva Ginecologica

Abstract:

What problem does this paper attempt to address?

Automatic Depression Prediction Via Cross-Modal Attention-Based Multi-Modal Fusion in Social Networks

Lidong Wang,Yin Zhang,Bin Zhou,Shihua Cao,Keyong Hu,Yunfei Tan

DOI: https://doi.org/10.1016/j.compeleceng.2024.109413

IF: 4.152

2024-01-01

Computers & Electrical Engineering

Abstract:Depression has become a worldwide public health concern in recent years. The online social network can be served as a tool to predict latent depressed users. However, existing studies primarily use machine learning or deep learning methods on multi-modal information with simple concatenation, resulting in unsatisfactory prediction. Additionally, the lack of interpretability in extracted features is a challenge. Furthermore, the majority of these studies primarily concentrate on English social networks. To address these issues, we propose a novel unified multi-tasking training model named CrossAMF (Cross-modal Attention-based Multi-modal Fusion). CrossAMF extracts crucial multi-modal features with explainability and effectively fuses representations of different modalities. Our approach employs a sentiment filtering-based XLNet-CNN-BiGRU model and Reduce-VGGNet to extract textual and image features, respectively. We also introduce innovative user behavior metrics previously unexplored. We collect a dataset of Chinese depressed users from Sina microblog with their texts and images, including 5692 depressed users and 9121 normal users. Experimental results demonstrate the significance of each modality to our task, with the combination yielding the best performance. CrossAMF outperforms other state-of-the-art methods with 0.9097 in accuracy and 0.8915 in F1, showing the potential of application in screening out early depression automatically.
An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Shiyu Teng,Jiaqing Liu,Yue Huang,Shurong Chai,Tomoko Tateyama,Xinyin Huang,Lanfen Lin,Yen-Wei Chen

DOI: https://doi.org/10.1587/transinf.2023hcp0006

2024-01-01

IEICE Transactions on Information and Systems

Abstract:Depression is a prevalent mental disorder affecting a significant portion of the global population, leading to considerable disability and contributing to the overall burden of disease. Consequently, designing efficient and robust automated methods for depression detection has become imperative. Recently, deep learning methods, especially multimodal fusion methods, have been increasingly used in computer-aided depression detection. Importantly, individuals with depression and those without respond differently to various emotional stimuli, providing valuable information for detecting depression. Building on these observations, we propose an intra- and inter-emotional stimulus transformer-based fusion model to effectively extract depression-related features. The intra-emotional stimulus fusion framework aims to prioritize different modalities, capitalizing on their diversity and complementarity for depression detection. The inter-emotional stimulus model maps each emotional stimulus onto both invariant and specific subspaces using individual invariant and specific encoders. The emotional stimulus-invariant subspace facilitates efficient information sharing and integration across different emotional stimulus categories, while the emotional stimulus specific subspace seeks to enhance diversity and capture the distinct characteristics of individual emotional stimulus categories. Our proposed intra- and inter-emotional stimulus fusion model effectively integrates multimodal data under various emotional stimulus categories, providing a comprehensive representation that allows accurate task predictions in the context of depression detection. We evaluate the proposed model on the Chinese Soochow University students dataset, and the results outperform state-of-the-art models in terms of concordance correlation coefficient (CCC), root mean squared error (RMSE) and accuracy.
Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

Zhang,Zhang,Ni,Wei,Yang,Jin,Huang,Liang,Zhang,Li,Ding,Zhang,Wang

DOI: https://doi.org/10.3390/s24123714

IF: 3.9

2024-06-08

Sensors

Abstract:Depression is a major psychological disorder with a growing impact worldwide. Traditional methods for detecting the risk of depression, predominantly reliant on psychiatric evaluations and self-assessment questionnaires, are often criticized for their inefficiency and lack of objectivity. Advancements in deep learning have paved the way for innovations in depression risk detection methods that fuse multimodal data. This paper introduces a novel framework, the Audio, Video, and Text Fusion-Three Branch Network (AVTF-TBN), designed to amalgamate auditory, visual, and textual cues for a comprehensive analysis of depression risk. Our approach encompasses three dedicated branches—Audio Branch, Video Branch, and Text Branch—each responsible for extracting salient features from the corresponding modality. These features are subsequently fused through a multimodal fusion (MMF) module, yielding a robust feature vector that feeds into a predictive modeling layer. To further our research, we devised an emotion elicitation paradigm based on two distinct tasks—reading and interviewing—implemented to gather a rich, sensor-based depression risk detection dataset. The sensory equipment, such as cameras, captures subtle facial expressions and vocal characteristics essential for our analysis. The research thoroughly investigates the data generated by varying emotional stimuli and evaluates the contribution of different tasks to emotion evocation. During the experiment, the AVTF-TBN model has the best performance when the data from the two tasks are simultaneously used for detection, where the F1 Score is 0.78, Precision is 0.76, and Recall is 0.81. Our experimental results confirm the validity of the paradigm and demonstrate the efficacy of the AVTF-TBN model in detecting depression risk, showcasing the crucial role of sensor-based data in mental health detection.

engineering, electrical & electronic,instruments & instrumentation,chemistry, analytical
Multi-Modal and Multi-Task Depression Detection with Sentiment Assistance

Shiyu Teng,Shurong Chai,Jiaqing Liu,Tomoko Tateyama,Lanfen Lin,Yen-Wei Chen

DOI: https://doi.org/10.1109/icce59016.2024.10444213

2024-01-01

Abstract:Depression, a multifaceted mental health disorder, is characterized by persistent feelings of sorrow, hopelessness, and a pervasive loss of interest or pleasure in once-enjoyed activities. It often manifests with physical and cognitive symptoms, including alterations in appetite and sleep patterns, overwhelming fatigue, difficulty in maintaining focus, and recurrent contemplations of death or suicide. Psychological research has unveiled a profound connection between depressive emotions, the expression of those emotions, and their perception. This intricate relationship underscores the paramount importance of comprehending how individuals with depression both undergo and convey their emotional experiences. This study enhances the precision of depression detection through a multimodal, multi-task learning approach. It combines the depression detection dataset(the AVEC 2019 Detecting Depression with AI Sub-challenge_with the sentiment analysis dataset, CMU-MOSEI. By harnessing emotional data, this method significantly augments the accuracy of depression detection. The efficacy of this proposed approach is validated using a publicly available dataset, AVEC 2019. It surpasses existing state-of-the-art methods, achieving a Concordance Correlation Coefficient (CCC) of 0.466 and a Mean Absolute Error (MAE) of 5.21 on the AVEC 2019 DDS test set. This represents a notable 5.4% improvement over the accuracy achieved by the current state-of-the-art method, which had a CCC of 0.442.
Hybrid Network Feature Extraction for Depression Assessment from Speech

Ziping Zhao,Qifei Li,Nicholas Cummins,Bin Liu,Haishuai Wang,Jianhua Tao,Bjoern W. Schuller

DOI: https://doi.org/10.21437/interspeech.2020-2396

2020-01-01

Abstract:A fast-growing area of mental health research is the search for speech-based objective markers for conditions such as depression. One vital challenge in the development of speech-based depression severity assessment systems is the extraction of depression-relevant features from speech signals. In order to deliver more comprehensive feature representation, we herein explore the benefits of a hybrid network that encodes depressionrelated characteristics in speech for the task of depression severity assessment. The proposed network leverages self-attention networks (SAN) trained on low-level acoustic features and deep convolutional neural networks (DCNN) trained on 3D Log-Mel spectrograms. The feature representations learnt in the SAN and DCNN are concatenated and average pooling is exploited to aggregate complementary segment-level features. Finally, support vector regression is applied to predict a speaker’s Beck Depression Inventory-II score. Experiments based on a subset of the Audio-Visual Depressive Language Corpus, as used in the 2013 and 2014 Audio/Visual Emotion Challenges, demonstrate the effectiveness of our proposed hybrid approach.
DepMSTAT: Multimodal Spatio-Temporal Attentional Transformer for Depression Detection

Yongfeng Tao,Minqiang Yang,Huiru Li,Yushan Wu,Bin Hu

DOI: https://doi.org/10.1109/tkde.2024.3350071

IF: 9.235

2024-01-01

IEEE Transactions on Knowledge and Data Engineering

Abstract:Depression is one of the most common mental illnesses, but few of the currently proposed in-depth models based on social media data take into account both temporal and spatial information in the data for the detection of depression. In this paper, we present an efficient, low-covariance multimodal integrated spatio-temporal converter framework called DepMSTAT, which aims to detect depression using acoustic and visual features in social media data. The framework consists of four modules: a data preprocessing module, a token generation module, a Spatial-Temporal Attentional Transformer (STAT) module, and a depression classifier module. To efficiently capture spatial and temporal correlations in multimodal social media depression data, a plug-and-play STAT module is proposed. The module is capable of extracting unimodal spatio-temporal features and fusing unimodal information, playing a key role in the analysis of acoustic and visual features in social media data. Through extensive experiments on a depression database (D-Vlog), the method in this paper shows high accuracy (71.53%) in depression detection, achieving a performance that exceeds most models. This work provides a scaffold for studies based on multimodal data that assists in the detection of depression.

computer science, information systems, artificial intelligence,engineering, electrical & electronic
A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention

Shengjie Li,Yinhao Xiao

2024-07-02

Abstract:Depression, a prevalent and serious mental health issue, affects approximately 3.8\% of the global population. Despite the existence of effective treatments, over 75\% of individuals in low- and middle-income countries remain untreated, partly due to the challenge in accurately diagnosing depression in its early stages. This paper introduces a novel method for detecting depression based on multi-modal feature fusion utilizing cross-attention. By employing MacBERT as a pre-training model to extract lexical features from text and incorporating an additional Transformer module to refine task-specific contextual understanding, the model's adaptability to the targeted task is enhanced. Diverging from previous practices of simply concatenating multimodal features, this approach leverages cross-attention for feature integration, significantly improving the accuracy in depression detection and enabling a more comprehensive and precise analysis of user emotions and behaviors. Furthermore, a Multi-Modal Feature Fusion Network based on Cross-Attention (MFFNC) is constructed, demonstrating exceptional performance in the task of depression identification. The experimental results indicate that our method achieves an accuracy of 0.9495 on the test dataset, marking a substantial improvement over existing approaches. Moreover, it outlines a promising methodology for other social media platforms and tasks involving multi-modal processing. Timely identification and intervention for individuals with depression are crucial for saving lives, highlighting the immense potential of technology in facilitating early intervention for mental health issues.

Computation and Language,Artificial Intelligence,Machine Learning
Multimodal Spatiotemporal Representation for Automatic Depression Level Detection

Mingyue Niu,Jianhua Tao,Bin Liu,Jian Huang,Zheng Lian

DOI: https://doi.org/10.1109/taffc.2020.3031345

IF: 13.99

2020-01-01

IEEE Transactions on Affective Computing

Abstract:Physiological studies have shown that there are some differences in speech and facial activities between depressive and healthy individuals. Based on this fact, we propose a novel spatio-temporal attention (STA) network and a multimodal attention feature fusion (MAFF) strategy to obtain the multimodal representation of depression cues for predicting the individual depression level. Specifically, we first divide the speech amplitude spectrum/video into fixed-length segments and input these segments into the STA network, which not only integrates the spatial and temporal information through attention mechanism, but also emphasizes the audio/video frames related to depression detection. The audio/video segment-level feature is obtained from the output of the last full connection layer of the STA network. Second, this article employs the eigen evolution pooling method to summarize the changes of each dimension of the audio/video segment-level features to aggregate them into the audio/video level feature. Third, the multimodal representation with modal complementary information is generated using the MAFF and inputs into the support vector regression predictor for estimating depression severity. Experimental results on the AVEC2013 and AVEC2014 depression databases illustrate the effectiveness of our method.
Multimodal time-aware attention networks for depression detection

Ju Chun Cheng,Arbee L. P. Chen

DOI: https://doi.org/10.1007/s10844-022-00704-w

2022-04-18

Journal of Intelligent Information Systems

Abstract:Depression is a common mental disorder, which may lead to suicide when the condition is severe. With the advancement of technology, there are billions of people who share their thoughts and feelings on social media at any time and from any location. Social media data has therefore become a valuable resource to study and detect the depression of the user. In our work, we use Instagram as the platform to study depression detection. We use hashtags to find users and label them as depressive or non-depressive according to their self-statement. Text, image, and posting time are used jointly to detect depression. Furthermore, the time interval between posts is important information when studying medical-related data. In this paper, we use time-aware LSTM to handle the irregularity of time intervals in social media data and use an attention mechanism to pay more attention to the posts that are important for detecting depression. Experiment results show that our model outperforms previous work with an F1-score of 95.6%. In addition to the good performance on Instagram, our model also outperforms state-of-the-art methods in detecting depression on Twitter with an F1-score of 90.8%. This indicates the potential of our model to be a reference for psychiatrists to assess the patient; or for users to know more about their mental health condition.

computer science, information systems, artificial intelligence
Transformer-based multimodal feature enhancement networks for multimodal depression detection integrating video, audio and remote photoplethysmograph signals

Huiting Fan,Xingnan Zhang,Yingying Xu,Jiangxiong Fang,Shiqing Zhang,Xiaoming Zhao,Jun Yu

DOI: https://doi.org/10.1016/j.inffus.2023.102161

IF: 18.6

2023-12-02

Information Fusion

Abstract:Depression stands as one of the most widespread psychological disorders and has garnered increasing attention. Currently, how to effectively achieve automatic multimodal depression detection for assisting doctors in early diagnosis of depression, has become an important and challenging issue. To address this issue, this work proposes Transformer-based feature enhancement networks for multimodal depression detection. The proposed method effectively integrates three modalities including video, audio and remote photoplethysmographic (rPPG) signals for multimodal depression detection, in which the rPPG modality is introduced as an additional modality for enhancing the effectiveness of multimodal depression detection. The proposed method consists of three key steps: multimodal feature extraction for video, audio and rPPG modalities, Transformer-based multimodal feature enhancement (TMFE), and graph fusion networks (GFN) based multimodal fusion and depression prediction. More specially, in the stage of multimodal feature extraction, for video and audio modalities we employ deep convolutional neural networks (CNN) to extract the corresponding high-level video and audio features, respectively. For rPPG modality, we adopt a short-time end-to-end rPPG estimation framework to extract the rPPG signal values.The TMFE module stacks multiple Transformers such as the inter-modal, intra-modal, and tri-modal Transformers to jointly capture the dynamics and relationships within and between modalities for each time-step of input sequences. The GFN module is designed to effectively fuse the obtained feature representations from different modalities while maintaining the interactions between them simultaneously. Finally, the obtained shared feature representations of all modalities are fed into a multilayer perceptrons (MLP) network to implement final depression detection tasks. Extensive experiments are conducted on two public datasets such as AVEC2013 and AVEC2014, and experimental results demonstrate the validity of the proposed method on depression detection tasks.

computer science, artificial intelligence, theory & methods
Attention-Based Acoustic Feature Fusion Network for Depression Detection

Xiao Xu,Yang Wang,Xinru Wei,Fei Wang,Xizhe Zhang

2023-08-24

Abstract:Depression, a common mental disorder, significantly influences individuals and imposes considerable societal impacts. The complexity and heterogeneity of the disorder necessitate prompt and effective detection, which nonetheless, poses a difficult challenge. This situation highlights an urgent requirement for improved detection methods. Exploiting auditory data through advanced machine learning paradigms presents promising research directions. Yet, existing techniques mainly rely on single-dimensional feature models, potentially neglecting the abundance of information hidden in various speech characteristics. To rectify this, we present the novel Attention-Based Acoustic Feature Fusion Network (ABAFnet) for depression detection. ABAFnet combines four different acoustic features into a comprehensive deep learning model, thereby effectively integrating and blending multi-tiered features. We present a novel weight adjustment module for late fusion that boosts performance by efficaciously synthesizing these features. The effectiveness of our approach is confirmed via extensive validation on two clinical speech databases, CNRAC and CS-NRAC, thereby outperforming previous methods in depression detection and subtype classification. Further in-depth analysis confirms the key role of each feature and highlights the importance of MFCCrelated features in speech-based depression detection.

Sound,Artificial Intelligence,Audio and Speech Processing
Multi Fine-Grained Fusion Network for Depression Detection

Li Zhou,Zhenyu Liu,Yutong Li,Yuchi Duan,Huimin Yu,Bin Hu

DOI: https://doi.org/10.1145/3665247

2024-06-29

Abstract:Depression is an illness that involves emotional and mental health. Currently, depression detection through interviews is the most popular way. With the advancement of natural language processing and sentiment analysis, automated interview-based depression detection is strongly supported. However, current multimodal depression detection models fail to adequately capture the fine-grained features of depressive behaviors, making it difficult for the models to accurately characterize the subtle changes in depressive symptoms. To address this problem, we propose a Multi Fine-Grained Fusion Network (MFFNet). The core idea of this model is to extract and fuse the information of different scale feature pairs through a Multi-Scale Fastformer (MSfastformer), and then use the Recurrent Pyramid Model to integrate the features of different resolutions, promoting the interaction of multi-level information. Through the interaction of multi-scale and multi-resolution features, it aims to explore richer feature representations. To validate the effectiveness of our proposed MFFNet model, we conduct experiments on two depression interview datasets. The experimental results show that the MFFNet model performs better in depression detection compared to other benchmark multimodal models.

computer science, information systems, theory & methods, software engineering
Multi-Modal Adaptive Fusion Transformer Network for the Estimation of Depression Level

Hao Sun,Jiaqing Liu,Shurong Chai,Zhaolin Qiu,Lanfen Lin,Xinyin Huang,Yenwei Chen

DOI: https://doi.org/10.3390/s21144764

IF: 3.9

2021-07-12

Sensors

Abstract:Depression is a severe psychological condition that affects millions of people worldwide. As depression has received more attention in recent years, it has become imperative to develop automatic methods for detecting depression. Although numerous machine learning methods have been proposed for estimating the levels of depression via audio, visual, and audiovisual emotion sensing, several challenges still exist. For example, it is difficult to extract long-term temporal context information from long sequences of audio and visual data, and it is also difficult to select and fuse useful multi-modal information or features effectively. In addition, how to include other information or tasks to enhance the estimation accuracy is also one of the challenges. In this study, we propose a multi-modal adaptive fusion transformer network for estimating the levels of depression. Transformer-based models have achieved state-of-the-art performance in language understanding and sequence modeling. Thus, the proposed transformer-based network is utilized to extract long-term temporal context information from uni-modal audio and visual data in our work. This is the first transformer-based approach for depression detection. We also propose an adaptive fusion method for adaptively fusing useful multi-modal features. Furthermore, inspired by current multi-task learning work, we also incorporate an auxiliary task (depression classification) to enhance the main task of depression level regression (estimation). The effectiveness of the proposed method has been validated on a public dataset (AVEC 2019 Detecting Depression with AI Sub-challenge) in terms of the PHQ-8 scores. Experimental results indicate that the proposed method achieves better performance compared with currently state-of-the-art methods. Our proposed method achieves a concordance correlation coefficient (CCC) of 0.733 on AVEC 2019 which is 6.2% higher than the accuracy (CCC = 0.696) of the state-of-the-art method.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
A depression detection model based on multimodal graph neural network

Yujing Xia,Lin Liu,Tao Dong,Juan Chen,Yu Cheng,Lin Tang

DOI: https://doi.org/10.1007/s11042-023-18079-7

IF: 2.577

2024-01-13

Multimedia Tools and Applications

Abstract:Depression is a prevalent mental illness, especially major depression, which has a negative impact on individuals and society. In clinical practice, doctors diagnose depression primarily based on self-reported scores, which can be highly subjective. Therefore, developing a framework for diagnosing and identifying depression is a highly significant study. However, existing studies in this field face the challenges of lack of sample size and multimodal data fusion due to difficulties in obtaining patient data. To address these challenges, we propose a multimodal graph neural network-based model for depression detection. In this model, we solve the few-shot learning problem based on a GNN, which can recursively aggregate and transform neighboring nodes to refine the node representation and is very effective for few-shot learning. For multimodal fusion in depression recognition, a pre-fusion strategy is used to fuse three different modal features (audio, text, and video), and input them into the Bi-LSTM fusion network to learn high-level global features of multimodal information to form a multimodal fusion representation. Finally, we embedded the multimodal fusion module into a GNN to predict depression. This study not only solves the multimodal fusion problem but also can effectively improve the generalization performance of few-shot learning. The method achieved an accuracy of 0.861 on the publicly available depression-based dataset DAIC-WOZ, and the final prediction results far exceeded the baseline level, this shows that our model is highly applicable when dealing with small amounts of multimodal medical data.

computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering
SAD-TIME: a Spatiotemporal-fused network for depression detection with Automated multi-scale Depth-wise and TIME-interval-related common feature extractor

Han-Guang Wang,Hui-Rang Hou,Li-Cheng Jin,Chen-Yang Xu,Zhong-Yi Zhang,Qing-Hao Meng

2024-11-13

Abstract:Background and Objective: Depression is a severe mental disorder, and accurate diagnosis is pivotal to the cure and rehabilitation of people with depression. However, the current questionnaire-based diagnostic methods could bring subjective biases and may be denied by subjects. In search of a more objective means of diagnosis, researchers have begun to experiment with deep learning-based methods for identifying depressive disorders in recent years. Methods: In this study, a novel Spatiotemporal-fused network with Automated multi-scale Depth-wise and TIME-interval-related common feature extractor (SAD-TIME) is proposed. SAD-TIME incorporates an automated nodes' common features extractor (CFE), a spatial sector (SpS), a modified temporal sector (TeS), and a domain adversarial learner (DAL). The CFE includes a multi-scale depth-wise 1D-convolutional neural network and a time-interval embedding generator, where the unique information of each channel is preserved. The SpS fuses the functional connectivity with the distance-based connectivity containing spatial position of EEG electrodes. A multi-head-attention graph convolutional network is also applied in the SpS to fuse the features from different EEG channels. The TeS is based on long short-term memory and graph transformer networks, where the temporal information of different time-windows is fused. Moreover, the DAL is used after the SpS to obtain the domain-invariant feature. Results: Experimental results under tenfold cross-validation show that the proposed SAD-TIME method achieves 92.00% and 94.00% depression classification accuracies on two datasets, respectively, in cross-subject mode. Conclusion: SAD-TIME is a robust depression detection model, where the automatedly-generated features, the SpS and the TeS assist the classification performance with the fusion of the innate spatiotemporal information in the EEG signals.

Machine Learning,Artificial Intelligence
MS$^{2}$-GNN: Exploring GNN-Based Multimodal Fusion Network for Depression Detection

Tao Chen,Richang Hong,Yanrong Guo,Shijie Hao,Bin Hu

DOI: https://doi.org/10.1109/tcyb.2022.3197127

IF: 11.8

2022-01-01

IEEE Transactions on Cybernetics

Abstract:Major depressive disorder (MDD) is one of the most common and severe mental illnesses, posing a huge burden on society and families. Recently, some multimodal methods have been proposed to learn a multimodal embedding for MDD detection and achieved promising performance. However, these methods ignore the heterogeneity/homogeneity among various modalities. Besides, earlier attempts ignore interclass separability and intraclass compactness. Inspired by the above observations, we propose a graph neural network (GNN)-based multimodal fusion strategy named modal-shared modal-specific GNN, which investigates the heterogeneity/homogeneity among various psychophysiological modalities as well as explores the potential relationship between subjects. Specifically, we develop a modal-shared and modal-specific GNN architecture to extract the inter/intramodal characteristics. Furthermore, a reconstruction network is employed to ensure fidelity within the individual modality. Moreover, we impose an attention mechanism on various embeddings to obtain a multimodal compact representation for the subsequent MDD detection task. We conduct extensive experiments on two public depression datasets and the favorable results demonstrate the effectiveness of the proposed algorithm.

automation & control systems,computer science, cybernetics, artificial intelligence
Clonazepam (Rivotril) in Migraine Prophylaxis

P. Stensrud,O. Sjaastad

DOI: https://doi.org/10.1111/j.1526-4610.1979.hed1906333.x

1979-09-01

Abstract:SYNOPSIS
Enhancing depression detection: A multimodal approach with text extension and content fusion

Jinyan Chen,Shuxian Liu,Meijia Xu,Peicheng Wang

DOI: https://doi.org/10.1111/exsy.13616

IF: 3.3

2024-06-05

Expert Systems

Abstract:Background With ubiquitous social media platforms, people express their thoughts and emotions, making social media data valuable for studying and detecting depression symptoms. Objective First, we detect depression by leveraging textual, visual, and auxiliary features from the Weibo social media platform. Second, we aim to comprehend the reasons behind the model's results, particularly in medicine, where trust is crucial. Methods To address challenges such as varying text lengths and abundant social media data, we employ a text extension technique to standardize text length, enhancing model robustness and semantic feature learning accuracy. We utilize tree‐long short‐term memory and bidirectional gate recurrent unit models to capture long‐term and short‐term dependencies in text data, respectively. To extract emotional features from images, the integration of optical character recognition (OCR) technology with an emotion lexicon is employed, addressing the limitations of OCR technology in accuracy when dealing with complex or blurred text. In addition, auxiliary features based on social behaviour are introduced. These modalities' output features are fed into an attention fusion network for effective depression indicators. Results Extensive experiments validate our methodology, showing a precision of 0.987 and recall rate of 0.97 in depression detection tasks. Conclusions By leveraging text, images, and auxiliary features from Weibo, we develop text picture sentiment auxiliary (TPSA), a novel depression detection model. we ascertained that the emotional features extracted from images and text play a pivotal role in depression detection, providing valuable insights for the detection and assessment of the psychological disorder.

computer science, artificial intelligence, theory & methods
CANAMRF: An Attention-Based Model for Multimodal Depression Detection

Yuntao Wei,Yuzhe Zhang,Shuyang Zhang,Hong Zhang

DOI: https://doi.org/10.1007/978-981-99-7022-3_10

2024-01-04

Abstract:Multimodal depression detection is an important research topic that aims to predict human mental states using multimodal data. Previous methods treat different modalities equally and fuse each modality by naïve mathematical operations without measuring the relative importance between them, which cannot obtain well-performed multimodal representations for downstream depression tasks. In order to tackle the aforementioned concern, we present a Cross-modal Attention Network with Adaptive Multi-modal Recurrent Fusion (CANAMRF) for multimodal depression detection. CANAMRF is constructed by a multimodal feature extractor, an Adaptive Multimodal Recurrent Fusion module, and a Hybrid Attention Module. Through experimentation on two benchmark datasets, CANAMRF demonstrates state-of-the-art performance, underscoring the effectiveness of our proposed approach.

Computation and Language,Artificial Intelligence,Computer Vision and Pattern Recognition,Image and Video Processing
Multimodal fusion diagnosis of depression and anxiety based on CNN-LSTM model

Wanqing Xie,Chen Wang,Zhixiong Lin,Xudong Luo,Wenqian Chen,Manzhu Xu,Lizhong Liang,Xiaofeng Liu,Yanzhong Wang,Hui Luo,Mingmei Cheng,Wanqing Xie,Chen Wang,Zhixiong Lin,Xudong Luo,Wenqian Chen,Manzhu Xu,Lizhong Liang,Xiaofeng Liu,Yanzhong Wang,Hui Luo,Mingmei Cheng

DOI: https://doi.org/10.1016/j.compmedimag.2022.102128

IF: 7.422

2022-12-01

Computerized Medical Imaging and Graphics

Abstract:BackgroundIn recent years, more and more people suffer from depression and anxiety. These symptoms are hard to be spotted and can be very dangerous. Currently, the Self-Reported Anxiety Scale (SAS) and Self-Reported Depression Scale (SDS) are commonly used for initial screening for depression and anxiety disorders. However, the information contained in these two scales is limited, while the symptoms of subjects are various and complex, which results in the inconsistency between the questionnaire evaluation results and the clinician's diagnosis results. To fully mine the scale data, we propose a method to extract the features from the facial expression and movements, which are generated from the video recorded simultaneously when subjects fill in the scale. Then we collect the facial expression, movements and scale information to establish a multimodal framework for improving the accuracy and robustness of the diagnosis of depression and anxiety.MethodsWe collect the scale results of the subjects and the videos when filling in the scales. Given the two scales, SAS and SDS, we construct a model with two branches, where each branch processes the multimodal data of SAS and SDS, respectively. In the branch, we first build a convolutional neural network (CNN) to extracts the facial expression features in each frame of images. Secondly, we establish a long short-term memory (LSTM) network to further embedding the facial expression feature and build the connections between various frames, so that the movement feature in the video can be generated. Thirdly, we transform the scale scores into one-hot format, and feed them into the corresponding branch of the network to further mining the information of the multimodal data. Finally, we fuse the embeddings of these two branches to generate inference results of depression and anxiety.Results and conclusionsBased on the score results of SAS and SDS, our multimodal model further mines the video information, and can reach the accuracy of 0.946 in diagnosing depression and anxiety. This study demonstrates the feasibility of using our CNN-LSTM-based multimodal model for initial screening and diagnosis of depression and anxiety disorders with high diagnostic performance.

engineering, biomedical,radiology, nuclear medicine & medical imaging

[Fetal phono-electrocardiography. II. Sensitivity of the fetus to some drugs at various periods of pregnancy in physiological and pathological conditions].

Automatic Depression Prediction Via Cross-Modal Attention-Based Multi-Modal Fusion in Social Networks

An Intra- and Inter-Emotion Transformer-Based Fusion Model with Homogeneous and Diverse Constraints Using Multi-Emotional Audiovisual Features for Depression Detection.

Multimodal Sensing for Depression Risk Detection: Integrating Audio, Video, and Text Data

Multi-Modal and Multi-Task Depression Detection with Sentiment Assistance

Hybrid Network Feature Extraction for Depression Assessment from Speech

DepMSTAT: Multimodal Spatio-Temporal Attentional Transformer for Depression Detection

A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention

Multimodal Spatiotemporal Representation for Automatic Depression Level Detection

Multimodal time-aware attention networks for depression detection

Transformer-based multimodal feature enhancement networks for multimodal depression detection integrating video, audio and remote photoplethysmograph signals

Attention-Based Acoustic Feature Fusion Network for Depression Detection

Multi Fine-Grained Fusion Network for Depression Detection

Multi-Modal Adaptive Fusion Transformer Network for the Estimation of Depression Level

A depression detection model based on multimodal graph neural network

SAD-TIME: a Spatiotemporal-fused network for depression detection with Automated multi-scale Depth-wise and TIME-interval-related common feature extractor

MS$^{2}$-GNN: Exploring GNN-Based Multimodal Fusion Network for Depression Detection

Clonazepam (Rivotril) in Migraine Prophylaxis

Enhancing depression detection: A multimodal approach with text extension and content fusion

CANAMRF: An Attention-Based Model for Multimodal Depression Detection

Multimodal fusion diagnosis of depression and anxiety based on CNN-LSTM model