Abstract:Decoding of cognitive states aims to identify individuals' brain states and brain fingerprints to predict behavior. Deep learning provides an important platform for analyzing brain signals at different developmental stages to understand brain dynamics. Due to their internal architecture and feature extraction techniques, existing machine-learning and deep-learning approaches are suffering from low classification performance and explainability issues that must be improved. In the current study, we hypothesized that even at the early childhood stage (as early as 3-years), connectivity between brain regions could decode brain states and predict behavioral performance in false-belief tasks. To this end, we proposed an explainable deep learning framework to decode brain states (Theory of Mind and Pain states) and predict individual performance on ToM-related false-belief tasks in a developmental dataset. We proposed an explainable spatiotemporal connectivity-based Graph Convolutional Neural Network (Ex-stGCNN) model for decoding brain states. Here, we consider a developmental dataset, N = 155 (122 children; 3-12 yrs and 33 adults; 18-39 yrs), in which participants watched a short, soundless animated movie, shown to activate Theory-of-Mind (ToM) and pain networs. After scanning, the participants underwent a ToM-related false-belief task, leading to categorization into the pass, fail, and inconsistent groups based on performance. We trained our proposed model using Functional Connectivity (FC) and Inter-Subject Functional Correlations (ISFC) matrices separately. We observed that the stimulus-driven feature set (ISFC) could capture ToM and Pain brain states more accurately with an average accuracy of 94%, whereas it achieved 85% accuracy using FC matrices. We also validated our results using five-fold cross-validation and achieved an average accuracy of 92%. Besides this study, we applied the SHapley Additive exPlanations (SHAP) approach to identify brain fingerprints that contributed the most to predictions. We hypothesized that ToM network brain connectivity could predict individual performance on false-belief tasks. We proposed an Explainable Convolutional Variational Auto-Encoder (Ex-Convolutional VAE) model to predict individual performance on false-belief tasks and trained the model using FC and ISFC matrices separately. ISFC matrices again outperformed the FC matrices in prediction of individual performance. We achieved 93.5% accuracy with an F1-score of 0.94 using ISFC matrices and achieved 90% accuracy with an F1-score of 0.91 using FC matrices.

Explainable Deep Learning Framework: Decoding Brain Task and Prediction of Individual Performance in False-Belief Task at Early Childhood Stage

Explainable deep-learning framework: decoding brain states and prediction of individual performance in false-belief task at early childhood stage

A Hybrid Spatiotemporal Deep Belief Network and Sparse Representation-Based Framework Reveals Multilevel Core Functional Components in Decoding Multitask Fmri Signals.

A Hybrid Spatio-Temporal Deep Belief Network and Sparse Representation-Based Framework Reveals Multi-Level Core Functional Components in Decoding Multi-Task fMRI Signals

Functional annotation of human cognitive states using deep graph convolution

Challenges for cognitive decoding using deep learning methods

Transfer Learning-Based Behavioural Task Decoding from Brain Activity

Deep Learning with Explainability for Characterizing Age-Related Intrinsic Differences in Dynamic Brain Functional Connectivity.

Brain Decoding Using Deep Convolutional Network and Its Application in Cross-Subject Analysis.

Decoding task specific and task general functional architectures of the brain

Explainable artificial intelligence model to predict brain states from fNIRS signals

Decoding Behavior Tasks From Brain Activity Using Deep Transfer Learning

Cognitive Networks and Performance Drive fMRI-Based State Classification Using DNN Models

Predicting Task Activation Maps from Resting-State Functional Connectivity using Deep Learning

Attention module improves both performance and interpretability of four‐dimensional functional magnetic resonance imaging decoding neural network

Sparse Deep Neural Network for Encoding and Decoding the Structural Connectome

<i>Neuro</i>Explainer: Fine-Grained Attention Decoding to Uncover Cortical Development Patterns of Preterm Infants

Brainnetformer: Decoding Brain Cognitive States with Spatial-Temporal Cross Attention

Interpretable Fusion Analytics Framework for fMRI Connectivity: Self-Attention Mechanism and Latent Space Item-Response Model

Deep Learning Modeling for Prediction of Cognitive Task Related Features from Resting-state fMRI Data

Decoding working memory-related information from repeated psychophysiological EEG experiments using convolutional and contrastive neural networks