Abstract:Objective: A motor imagery-based brain-computer interface (MI-BCI) translates spontaneous movement intention from the brain to outside devices. MI-BCI systems based on a single modality have been widely researched in recent decades. Lately, along with the development of neuroimaging methods, multimodal MI-BCI studies that use multiple neural signals have been proposed, which are promising for enhancing the decoding accuracy of MI-BCI. Multimodal MI data contain rich common and complementary information. Effective feature representations are helpful to promote the performance of classification tasks. Thus, it is very important to explore and extract features with higher separability and robustness from the rich information in multimodal data. Approach: In this study, a five-class motor imagery experiment was designed. Electroencephalography and functional near infrared spectroscopy data were collected simultaneously. A multimodal MI decoding neural network was proposed. In this network, to enhance the feature representation, the heterogeneous data of different modalities in the spatial dimension were aligned through the proposed spatial alignment losses. Also, the multimodal features were aligned and fused in the temporal dimension by an attention-based modality fusion module. Main results and Significance: The collected dataset was analyzed from temporal, spatial and frequency perspectives. The results showed that the multimodal data contain visually separable motor imagery patterns. The experimental results show that the proposed decoding method achieved the highest decoding accuracy among the compared methods on the self-collected dataset and a public dataset. Ablation results show that each part of the proposed method is effective. Compared with single-modality decoding, the proposed method obtained 4.6% higher decoding accuracy on the self-collected dataset. This indicates that the proposed method can improve the performance of multimodal MI decoding. This study provides a new approach for capturing the rich information in multimodal MI data and enhancing multimodal MI-BCI decoding accuracy.

The Chinese classification of mental disorders

Motor Imagery Classification Based on Deep Convolutional Neural Network and Its Application in Exoskeleton Controlled by EEG

A Motor Imagery Signals Classification Method via the Difference of EEG Signals Between Left and Right Hemispheric Electrodes

A Study of Unilateral Upper Limb Fine Motor Imagery Decoding Using Frequency-Band Attention Network

Decoding multiclass motor imagery EEG from the same upper limb by combining Riemannian geometry features and partial least squares regression

A Multifrequency Brain Network-Based Deep Learning Framework for Motor Imagery Decoding

Decoding Multi-class Motor-related Intentions with User-optimized and Robust BCI System Based on Multimodal Dataset

Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network

Multiclass classification of motor imagery tasks based on multi-branch convolutional neural network and temporal convolutional network model

Multi-branch spatial-temporal-spectral convolutional neural networks for multi-task motor imagery EEG classification

Extraction and solubilization of mouse liver proteins enriched for alkaline proteins.

A robust multi-branch multi-attention-mechanism EEGNet for motor imagery BCI decoding

Motor Imagery Decoding Enhancement Based on Hybrid EEG-fNIRS Signals

A Multi-Domain Convolutional Neural Network for EEG-Based Motor Imagery Decoding

Experimental study of reliabilities in tri-gate nanowire transistor ~ What is main reliability issue in 3D transistor?

A coarse-to-fine adaptive spatial–temporal graph convolution network with residuals for motor imagery decoding from the same limb

Decoding the Continuous Motion Imagery Trajectories of Upper Limb Skeleton Points for EEG-Based Brain–Computer Interface

Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion

Classification algorithm for EEG-based motor imagery using hybrid neural network with spatio-temporal convolution and multi-head attention mechanism

MI-MBFT: Superior Motor Imagery Decoding of Raw EEG Data Based on a Multi-Branch and Fusion Transformer Framework

One-Dimensional Convolutional Multi-branch Fusion Network for EEG-Based Motor Imagery Classification