Abstract:In the current age, human lifestyle has become more knowledge oriented leading to generation of sedentary employment. This has given rise to a number of health and mental disorders. Mental wellness is one of the most neglected but crucial aspects of today's world. Mental health issues can, both directly and indirectly, affect other sections of human physiology and impede an individual's day-to-day activities and performance. However, identifying the stress and finding the stress trend for an individual leading to serious mental ailments is challenging and involves multiple factors. Such identification can be achieved accurately by fusing these multiple modalities (due to various factors) arising from behavioral patterns. Certain techniques are identified in the literature for this purpose; however, very few machine learning-based methods are proposed for such multimodal fusion tasks. In this work, a multimodal AI-based framework is proposed to monitor a person's working behavior and stress levels. We propose a methodology for efficiently detecting stress due to workload by concatenating heterogeneous raw sensor data streams (e.g., face expressions, posture, heart rate, computer interaction). This data can be securely stored and analyzed to understand and discover personalized unique behavioral patterns leading to mental strain and fatigue. The contribution of this work is twofold; proposing a multimodal AI-based strategy for fusion to detect stress and its level and secondly identify a stress pattern over a period of time. We were able to achieve 96.09% accuracy on the test set in stress detection and classification. Further, we reduce the stress scale prediction model loss to 0.036 using these modalities. This work can prove important for the community at large, specifically those working sedentary jobs to monitor and identify stress levels, especially in current times of COVID-19.

Dynamic Alignment and Fusion of Multimodal Physiological Patterns for Stress Recognition

Multimodal Stress Detection Using Facial Landmarks and Biometric Signals

A Multimodal Late Fusion Framework for Physiological Sensor and Audio-Signal-Based Stress Detection: An Experimental Study and Public Dataset

Multimodal Hierarchical CNN Feature Fusion for Stress Detection

Modality Fusion Network and Personalized Attention in Momentary Stress Detection in the Wild

Driver stress detection via multimodal fusion using attention-based CNN-LSTM

Employing Multimodal Machine Learning for Stress Detection

Joint low-rank tensor fusion and cross-modal attention for multimodal physiological signals based emotion recognition

Emotion Recognition From Multimodal Physiological Signals via Discriminative Correlation Fusion With a Temporal Alignment Mechanism

A Self-supervised Framework for Improved Data-Driven Monitoring of Stress via Multi-modal Passive Sensing

Combining Multiple Sensor Features for Stress Detection Using Combinatorial Fusion

Shuffled ECA-Net for stress detection from multimodal wearable sensor data

A Multimodal Non-Intrusive Stress Monitoring from the Pleasure-Arousal Emotional Dimensions

A Transductive Model-based Stress Recognition Method Using Peripheral Physiological Signals

Facial Video-Based Non-Contact Stress Recognition Utilizing Multi-Task Learning With Peak Attention

Hybrid Mutimodal Fusion for Dimensional Emotion Recognition

Multimodal Temporal Attention in Sentiment Analysis

Multi-Modal Acute Stress Recognition Using Off-the-Shelf Wearable Devices

Evaluating the Potential of Wearable Technology in Early Stress Detection: A Multimodal Approach

Emotion Recognition From Multimodal Physiological Signals Using a Regularized Deep Fusion of Kernel Machine

Multi-level Stress Assessment Using Multi-domain Fusion of ECG Signal