Abstract:Objective: A motor imagery-based brain-computer interface (MI-BCI) translates spontaneous movement intention from the brain to outside devices. MI-BCI systems based on a single modality have been widely researched in recent decades. Lately, along with the development of neuroimaging methods, multimodal MI-BCI studies that use multiple neural signals have been proposed, which are promising for enhancing the decoding accuracy of MI-BCI. Multimodal MI data contain rich common and complementary information. Effective feature representations are helpful to promote the performance of classification tasks. Thus, it is very important to explore and extract features with higher separability and robustness from the rich information in multimodal data. Approach: In this study, a five-class motor imagery experiment was designed. Electroencephalography and functional near infrared spectroscopy data were collected simultaneously. A multimodal MI decoding neural network was proposed. In this network, to enhance the feature representation, the heterogeneous data of different modalities in the spatial dimension were aligned through the proposed spatial alignment losses. Also, the multimodal features were aligned and fused in the temporal dimension by an attention-based modality fusion module. Main results and Significance: The collected dataset was analyzed from temporal, spatial and frequency perspectives. The results showed that the multimodal data contain visually separable motor imagery patterns. The experimental results show that the proposed decoding method achieved the highest decoding accuracy among the compared methods on the self-collected dataset and a public dataset. Ablation results show that each part of the proposed method is effective. Compared with single-modality decoding, the proposed method obtained 4.6% higher decoding accuracy on the self-collected dataset. This indicates that the proposed method can improve the performance of multimodal MI decoding. This study provides a new approach for capturing the rich information in multimodal MI data and enhancing multimodal MI-BCI decoding accuracy.

A single-trial decoding method by integrating accumulated continuous classification for motor imagery based BCI

A Stimulus-Independent Hybrid BCI Based on Motor Imagery and Somatosensory Attentional Orientation

An Approach of One-vs-Rest Filter Bank Common Spatial Pattern and Spiking Neural Networks for Multiple Motor Imagery Decoding

A Multifrequency Brain Network-Based Deep Learning Framework for Motor Imagery Decoding

Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion

Motor Imagery EEG Decoding Based on Multi-scale Hybrid Networks and Feature Enhancement.

A coarse-to-fine adaptive spatial–temporal graph convolution network with residuals for motor imagery decoding from the same limb

Decoding Motor Imagery Tasks Using ESI and Hybrid Feature CNN

Decoding Multi-Class Motor Imagery from Unilateral Limbs Using EEG Signals

Riemannian geometric and ensemble learning for decoding cross-session motor imagery electroencephalography signals

A Model Combining Multi Branch Spectral-Temporal CNN, Efficient Channel Attention, and LightGBM for MI-BCI Classification

Enhancing Motor Imagery Decoding in Brain Computer Interfaces using Riemann Tangent Space Mapping and Cross Frequency Coupling

Application of Continuous Wavelet Transform and Convolutional Neural Network in Decoding Motor Imagery Brain-Computer Interface

A Multi-Domain Convolutional Neural Network for EEG-Based Motor Imagery Decoding

EISATC-Fusion: Inception Self-Attention Temporal Convolutional Network Fusion for Motor Imagery EEG Decoding

Deep Learning Based Inter-subject Continuous Decoding of Motor Imagery for Practical Brain-Computer Interfaces

A Novel Motor Imagery EEG Decoding Method Based on Feature Separation

Motor Imagery EEG Decoding Method Based on a Discriminative Feature Learning Strategy

Multiclass classification of motor imagery tasks based on multi-branch convolutional neural network and temporal convolutional network model

A robust multi-branch multi-attention-mechanism EEGNet for motor imagery BCI decoding

Motor Imagery Classification Emphasizing Corresponding Frequency Domain Method based on Deep Learning Framework