Abstract:Objective: A motor imagery-based brain-computer interface (MI-BCI) translates spontaneous movement intention from the brain to outside devices. MI-BCI systems based on a single modality have been widely researched in recent decades. Lately, along with the development of neuroimaging methods, multimodal MI-BCI studies that use multiple neural signals have been proposed, which are promising for enhancing the decoding accuracy of MI-BCI. Multimodal MI data contain rich common and complementary information. Effective feature representations are helpful to promote the performance of classification tasks. Thus, it is very important to explore and extract features with higher separability and robustness from the rich information in multimodal data. Approach: In this study, a five-class motor imagery experiment was designed. Electroencephalography and functional near infrared spectroscopy data were collected simultaneously. A multimodal MI decoding neural network was proposed. In this network, to enhance the feature representation, the heterogeneous data of different modalities in the spatial dimension were aligned through the proposed spatial alignment losses. Also, the multimodal features were aligned and fused in the temporal dimension by an attention-based modality fusion module. Main results and Significance: The collected dataset was analyzed from temporal, spatial and frequency perspectives. The results showed that the multimodal data contain visually separable motor imagery patterns. The experimental results show that the proposed decoding method achieved the highest decoding accuracy among the compared methods on the self-collected dataset and a public dataset. Ablation results show that each part of the proposed method is effective. Compared with single-modality decoding, the proposed method obtained 4.6% higher decoding accuracy on the self-collected dataset. This indicates that the proposed method can improve the performance of multimodal MI decoding. This study provides a new approach for capturing the rich information in multimodal MI data and enhancing multimodal MI-BCI decoding accuracy.

A bimodal deep learning network based on CNN for fine motor imagery

Motor Imagery Classification Based on Deep Convolutional Neural Network and Its Application in Exoskeleton Controlled by EEG

EEG-FMCNN: A fusion multi-branch 1D convolutional neural network for EEG-based motor imagery classification

One-Dimensional Convolutional Multi-branch Fusion Network for EEG-Based Motor Imagery Classification

Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network

Single-trial EEG Classification of Motor Imagery Using Deep Convolutional Neural Networks

MCFHNet: Multi-Channel Fusion Hybrid Network for Efficient EEG-fNIRS Multi-modal Motor Imagery Decoding

A coarse-to-fine adaptive spatial–temporal graph convolution network with residuals for motor imagery decoding from the same limb

Motor Imagery EEG Decoding Based on Multi-scale Hybrid Networks and Feature Enhancement.

A Motor Imagery Signals Classification Method via the Difference of EEG Signals Between Left and Right Hemispheric Electrodes

Motor Imagery Decoding Enhancement Based on Hybrid EEG-fNIRS Signals

Multiclass classification of motor imagery tasks based on multi-branch convolutional neural network and temporal convolutional network model

Classification algorithm for motor imagery fusing CNN and attentional mechanisms based on functional near-infrared spectroscopy brain image

Self-attention-based Convolutional Neural Network AND Time-frequency Common Spatial Pattern for enhanced Motor Imagery Classification

A Study of Unilateral Upper Limb Fine Motor Imagery Decoding Using Frequency-Band Attention Network

Motor imagery EEG classification algorithm based on CNN-LSTM feature fusion network

An end-to-end multi-task motor imagery EEG classification neural network based on dynamic fusion of spectral-temporal features

Attention-based convolutional neural network with multi-modal temporal information fusion for motor imagery EEG decoding

Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion

Automatic feature extraction and fusion recognition of motor imagery EEG using multilevel multiscale CNN

Merged CNNs for the classification of EEG motor imagery signals