Abstract:Abstract Objective . Motor imagery (MI) electroencephalography (EEG) classification is regarded as a promising technology for brain–computer interface (BCI) systems, which help people to communicate with the outside world using neural activities. However, decoding human intent accurately is a challenging task because of its small signal-to-noise ratio and non-stationary characteristics. Methods that directly extract features from raw EEG signals ignores key frequency domain information. One of the challenges in MI classification tasks is finding a way to supplement the frequency domain information ignored by the raw EEG signal. Approach . In this study, we fuse different models using their complementary characteristics to develop a multiscale space-time-frequency feature-guided multitask learning convolutional neural network (CNN) architecture. The proposed method consists of four modules: the space-time feature-based representation module, time-frequency feature-based representation module, multimodal fused feature-guided generation module, and classification module. The proposed framework is based on multitask learning. The four modules are trained using three tasks simultaneously and jointly optimized. Results . The proposed method is evaluated using three public challenge datasets. Through quantitative analysis, we demonstrate that our proposed method outperforms most state-of-the-art machine learning and deep learning techniques for EEG classification, thereby demonstrating the robustness and effectiveness of our method. Moreover, the proposed method is employed to realize control of robot based on EEG signal, verifying its feasibility in real-time applications. Significance . To the best of our knowledge, a deep CNN architecture that fuses different input cases, which have complementary characteristics, has not been applied to BCI tasks. Because of the interaction of the three tasks in the multitask learning architecture, our method can improve the generalization and accuracy of subject-dependent and subject-independent methods with limited annotated data.

Multimodal Fusion of EEG and Fmri for Epilepsy Detection

A Brain-Computer Interface Based on a Few-Channel EEG-fNIRS Bimodal System

Feature Extraction and Classification Methods for Hybrid fNIRS-EEG Brain-Computer Interfaces

Spatial and Spectral Features Fusion for EEG Classification during Motor Imagery in BCI

A novel multimodal approach for hybrid brain-computer interface

Simultaneous EEG-fNIRS Data Classification Through Selective Channel Representation and Spectrogram Imaging

A Review of EEG-based Localization of Epileptic Seizure Foci: Common Points with Multimodal Fusion of Brain Data

A Graph-Based Feature Extraction Algorithm Towards a Robust Data Fusion Framework for Brain-Computer Interfaces

Local Multimodal Serial Analysis for Fusing EEG-fMRI: A New Method to Study Familial Cortical Myoclonic Tremor and Epilepsy.

Feature Extraction and Classification Algorithm of Brain-computer Interface Based on Human Brain Central Nervous System

A New Generation of Brain-Computer Interfaces Driven by Discovery of Latent EEG-fMRI Linkages Using Tensor Decomposition

Multiscale space-time-frequency feature-guided multitask learning CNN for motor imagery EEG classification

Incomplete multi-modal brain image fusion for epilepsy classification

Analysis of EEG Signals Fused With CNN and Complex Networks

A Hybrid EEG-fNIRS Brain-Computer Interface Based on Dynamic Functional Connectivity and Long Short-Term Memory

Filter bank common spatial pattern and envelope-based features in multimodal EEG-fTCD brain-computer interfaces

Motor Imagery Decoding Enhancement Based on Hybrid EEG-fNIRS Signals

EEG-EMG FAConformer: Frequency Aware Conv-Transformer for the fusion of EEG and EMG

Studying Multi-Frequency Multilayer Brain Network via Deep Learning for EEG-Based Epilepsy Detection

Independent Decision Path Fusion for Bimodal Asynchronous Brain–Computer Interface to Discriminate Multiclass Mental States

EEG Signal Processing Based on Multivariate Empirical Mode Decomposition and Common Spatial Pattern Hybrid Algorithm