Abstract:Decoding of cognitive states aims to identify individuals' brain states and brain fingerprints to predict behavior. Deep learning provides an important platform for analyzing brain signals at different developmental stages to understand brain dynamics. Due to their internal architecture and feature extraction techniques, existing machine-learning and deep-learning approaches are suffering from low classification performance and explainability issues that must be improved. In the current study, we hypothesized that even at the early childhood stage (as early as 3-years), connectivity between brain regions could decode brain states and predict behavioral performance in false-belief tasks. To this end, we proposed an explainable deep learning framework to decode brain states (Theory of Mind and Pain states) and predict individual performance on ToM-related false-belief tasks in a developmental dataset. We proposed an explainable spatiotemporal connectivity-based Graph Convolutional Neural Network (Ex-stGCNN) model for decoding brain states. Here, we consider a developmental dataset, N = 155 (122 children; 3-12 yrs and 33 adults; 18-39 yrs), in which participants watched a short, soundless animated movie, shown to activate Theory-of-Mind (ToM) and pain networs. After scanning, the participants underwent a ToM-related false-belief task, leading to categorization into the pass, fail, and inconsistent groups based on performance. We trained our proposed model using Functional Connectivity (FC) and Inter-Subject Functional Correlations (ISFC) matrices separately. We observed that the stimulus-driven feature set (ISFC) could capture ToM and Pain brain states more accurately with an average accuracy of 94%, whereas it achieved 85% accuracy using FC matrices. We also validated our results using five-fold cross-validation and achieved an average accuracy of 92%. Besides this study, we applied the SHapley Additive exPlanations (SHAP) approach to identify brain fingerprints that contributed the most to predictions. We hypothesized that ToM network brain connectivity could predict individual performance on false-belief tasks. We proposed an Explainable Convolutional Variational Auto-Encoder (Ex-Convolutional VAE) model to predict individual performance on false-belief tasks and trained the model using FC and ISFC matrices separately. ISFC matrices again outperformed the FC matrices in prediction of individual performance. We achieved 93.5% accuracy with an F1-score of 0.94 using ISFC matrices and achieved 90% accuracy with an F1-score of 0.91 using FC matrices.

Interpretable Self-Supervised Facial Micro-Expression Learning to Predict Cognitive State and Neurological Disorders

Stuttering Speech Disfluency Prediction using Explainable Attribution Vectors of Facial Muscle Movements

Analyze Spontaneous Gestures for Emotional Stress State Recognition: A Micro-gesture Dataset and Analysis with Deep Learning

Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression

Deep Learning Based Transfer Learning for Possible Facial Psychological Expression Recognition

Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data

Post-hoc and manifold explanations analysis of facial expression data based on deep learning

Evaluation of Interpretability for Deep Learning algorithms in EEG Emotion Recognition: A case study in Autism

AI-Based Visual Early Warning System

Faces of the Mind: Unveiling Mental Health States Through Facial Expressions in 11,427 Adolescents

Automated analysis of facial emotions in subjects with cognitive impairment

Facial Expression and Peripheral Physiology Fusion to Decode Individualized Affective Experience

Facial Emotion Recognition for Aid of Social Interactions for Autistic Children

Your Face Mirrors Your Deepest Beliefs-Predicting Personality and Morals through Facial Emotion Recognition

Can micro-expressions be used as a biomarker for autism spectrum disorder?

Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition

Explainable deep-learning framework: decoding brain states and prediction of individual performance in false-belief task at early childhood stage

Machine Learning-Based Interpretable Modeling for Subjective Emotional Dynamics Sensing Using Facial EMG

Emotion recognition models for companion robots

Analysis of facial expressions recorded from patients during psychiatric interviews

A Deep Learning Perspective on the Origin of Facial Expressions