Abstract:Objective: A motor imagery-based brain-computer interface (MI-BCI) translates spontaneous movement intention from the brain to outside devices. MI-BCI systems based on a single modality have been widely researched in recent decades. Lately, along with the development of neuroimaging methods, multimodal MI-BCI studies that use multiple neural signals have been proposed, which are promising for enhancing the decoding accuracy of MI-BCI. Multimodal MI data contain rich common and complementary information. Effective feature representations are helpful to promote the performance of classification tasks. Thus, it is very important to explore and extract features with higher separability and robustness from the rich information in multimodal data. Approach: In this study, a five-class motor imagery experiment was designed. Electroencephalography and functional near infrared spectroscopy data were collected simultaneously. A multimodal MI decoding neural network was proposed. In this network, to enhance the feature representation, the heterogeneous data of different modalities in the spatial dimension were aligned through the proposed spatial alignment losses. Also, the multimodal features were aligned and fused in the temporal dimension by an attention-based modality fusion module. Main results and Significance: The collected dataset was analyzed from temporal, spatial and frequency perspectives. The results showed that the multimodal data contain visually separable motor imagery patterns. The experimental results show that the proposed decoding method achieved the highest decoding accuracy among the compared methods on the self-collected dataset and a public dataset. Ablation results show that each part of the proposed method is effective. Compared with single-modality decoding, the proposed method obtained 4.6% higher decoding accuracy on the self-collected dataset. This indicates that the proposed method can improve the performance of multimodal MI decoding. This study provides a new approach for capturing the rich information in multimodal MI data and enhancing multimodal MI-BCI decoding accuracy.

Classification of Motor Imagery Signals Using Noise-Assisted Fast Multivariate Empirical Mode Decomposition

EEG rhythm separation and time–frequency analysis of fast multivariate empirical mode decomposition for motor imagery BCI

Motor Imagery Classification Based on Deep Convolutional Neural Network and Its Application in Exoskeleton Controlled by EEG

A Detective Method for Multi-class EEG-based Motor Imagery Classification Based on OCSVM

Motor Imagery Classification Emphasizing Corresponding Frequency Domain Method based on Deep Learning Framework

A Frequency-Temporal-spatial Method for Motor-Related Electroencephalography Pattern Recognition by Comprehensive Feature Optimization

Exploring Feature Selection and Classification Techniques to Improve the Performance of an Electroencephalography-Based Motor Imagery Brain-Computer Interface System

Classification algorithm for motor imagery fusing CNN and attentional mechanisms based on functional near-infrared spectroscopy brain image

A novel method for classification of multi-class motor imagery tasks based on feature fusion

Multi-class Motor Imagery Classification Method Using One-dimensional Convolutional Neural Networks

Parallel Spatial–Temporal Self-Attention CNN-Based Motor Imagery Classification for BCI

Classification of BCI Multiclass Motor Imagery Task Based on Artificial Neural Network

Classification algorithm for EEG-based motor imagery using hybrid neural network with spatio-temporal convolution and multi-head attention mechanism

A classification method for EEG motor imagery signals based on parallel convolutional neural network

A Motor Imagery Signals Classification Method via the Difference of EEG Signals Between Left and Right Hemispheric Electrodes

Motor Imagery Decoding Enhancement Based on Hybrid EEG-fNIRS Signals

3D Convolution neural network with multiscale spatial and temporal cues for motor imagery EEG classification

A motor imagery classification model based on hybrid brain-computer interface and multitask learning of electroencephalographic and electromyographic deep features

Neural Network-Based Feature Extraction for Multi-Class Motor Imagery Classification

Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion

A novel multi-branch hybrid neural network for motor imagery EEG signal classification