EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNN

Aniqa Arif,Yihe Wang,Rui Yin,Xiang Zhang,Ahmed Helmy

DOI: https://doi.org/10.3390/s24061889

IF: 3.9

2024-03-15

Sensors

Abstract:Analysis of brain signals is essential to the study of mental states and various neurological conditions. The two most prevalent noninvasive signals for measuring brain activities are electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). EEG, characterized by its higher sampling frequency, captures more temporal features, while fNIRS, with a greater number of channels, provides richer spatial information. Although a few previous studies have explored the use of multimodal deep-learning models to analyze brain activity for both EEG and fNIRS, subject-independent training–testing split analysis remains underexplored. The results of the subject-independent setting directly show the model's ability on unseen subjects, which is crucial for real-world applications. In this paper, we introduce EF-Net, a new CNN-based multimodal deep-learning model. We evaluate EF-Net on an EEG-fNIRS word generation (WG) dataset on the mental state recognition task, primarily focusing on the subject-independent setting. For completeness, we report results in the subject-dependent and subject-semidependent settings as well. We compare our model with five baseline approaches, including three traditional machine learning methods and two deep learning methods. EF-Net demonstrates superior performance in both accuracy and F1 score, surpassing these baselines. Our model achieves F1 scores of 99.36%, 98.31%, and 65.05% in the subject-dependent, subject-semidependent, and subject-independent settings, respectively, surpassing the best baseline F1 scores by 1.83%, 4.34%, and 2.13% These results highlight EF-Net's capability to effectively learn and interpret mental states and brain activity across different and unseen subjects.

engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

The paper aims to address a key challenge in neuroscience: identifying mental states by analyzing brain signals. Specifically, the authors propose a new Convolutional Neural Network (CNN) model—EF-Net, for mental state recognition through multimodal (EEG and fNIRS) data analysis.

The main contributions of the paper include:
1. **Proposing the EF-Net model**: This is a CNN-based deep learning method specifically designed for multimodal cross-subject analysis of EEG and fNIRS data.
2. **Exploring the challenges of cross-subject analysis**: The study investigates how to build a generalizable model that can adapt to unseen subjects and conducts detailed experiments under different train-test split settings, including single-modal or multimodal data.
3. **Empirical validation of effectiveness**: The effectiveness of EF-Net is validated through experiments in various settings (subject-dependent, semi-dependent, and independent), demonstrating its potential for cross-subject generalization using multimodal data.

### Main Findings
- EF-Net outperforms five baseline methods under different settings, including three traditional machine learning methods (SVM, Random Forest, k-Nearest Neighbors) and two deep learning methods (VGG and ResNet).
- In subject-dependent, semi-dependent, and independent settings, EF-Net achieved F1 scores of 99.36%, 98.31%, and 65.05%, respectively, surpassing the best baseline methods by 1.83%, 4.34%, and 2.13%.
- The combination of EEG and fNIRS data yielded the best results, especially in the subject-independent setting, indicating that EF-Net has good generalization capabilities.