Abstract:Objective: Brain-computer interface (BCI) can translate intentions directly into instructions and greatly improve the interaction experience for disabled people or some specific interactive applications. To improve the efficiency of BCI, the objective of this study is to explore the feasibility of an audio-assisted visual BCI speller and a deep learning-based single-trial event related potentials (ERP) decoding strategy. Approach: In this study, a two-stage BCI speller combining the motion-onset visual evoked potential (mVEP) and semantically congruent audio evoked ERP was designed to output the target characters. In the first stage, the different group of characters were presented in the different locations of visual field simultaneously and the stimuli were coded to the mVEP based on a new space division multiple access scheme. And then, the target character can be output based on the audio-assisted mVEP in the second stage. Meanwhile, a spatial-temporal attention-based convolutional neural network (STA-CNN) was proposed to recognize the single-trial ERP components. The CNN can learn 2-dimentional features including the spatial information of different activated channels and time dependence among ERP components. In addition, the STA mechanism can enhance the discriminative event-related features by adaptively learning probability weights. Main results: The performance of the proposed two-stage audio-assisted visual BCI paradigm and STA-CNN model was evaluated using the Electroencephalogram (EEG) recorded from 10 subjects. The average classification accuracy of proposed STA-CNN can reach 59.6 and 77.7% for the first and second stages, which were always significantly higher than those of the comparison methods ( p < 0.05). Significance: The proposed two-stage audio-assisted visual paradigm showed a great potential to be used to BCI speller. Moreover, through the analysis of the attention weights from time sequence and spatial topographies, it was proved that STA-CNN could effectively extract interpretable spatiotemporal EEG features.

Improving Intention Detection in Single-Trial Classification through Fusion of EEG and Eye-tracker Data

An autonomous hybrid brain-computer interface system combined with eye-tracking in virtual environment

A Stimulus-Independent Hybrid BCI Based on Motor Imagery and Somatosensory Attentional Orientation

Multimodal EEG and Eye Tracking Feature Fusion Approaches for Attention Classification in Hybrid BCIs

A Hybrid Method Fusing Frequency Recognition with Attention Detection to Enhance an Asynchronous Brain-Computer Interface

Research on Human-Computer Interaction Intention Recognition Based on EEG and Eye Movement

Dual Attentive Fusion for EEG-based Brain-Computer Interfaces

An EEG-based attention recognition method: fusion of time domain, frequency domain, and non-linear dynamics features

Design a Novel BCI for Neurorehabilitation Using Concurrent LFP and EEG Features: A Case Study

Improved Classification Performance of EEG-fNIRS Multimodal Brain-Computer Interface Based on Multi-Domain Features and Multi-Level Progressive Learning

Ergonomics Analysis Based on Intention Inference

A novel multimodal approach for hybrid brain-computer interface

Cascade and Parallel Convolutional Recurrent Neural Networks on EEG-based Intention Recognition for Brain Computer Interface

MBCFNet: A Multimodal Brain–Computer Fusion Network for Human Intention Recognition

Two Brains, One Target: Design of a Multi-Level Information Fusion Model Based on Dual-Subject RSVP

A Hybrid Asynchronous Brain-Computer Interface Combining SSVEP and EOG Signals for Rapid Target Recognition

Improvement of Error-Related Potential Monitoring in Brain-Computer Interface Based on Optimal Feature Dimensionality Selection

A Novel Hybrid Brain-Computer Interface Combining the Illusion-Induced VEP and SSVEP

Single-Trial EEG Classification Using Spatio-Temporal Weighting and Correlation Analysis for RSVP-Based Collaborative Brain Computer Interface.

Classification of Five Control/Non-control States to Enhance Asynchronous Brain-Computer Interfaces

A novel brain-computer interface based on audio-assisted visual evoked EEG and spatial-temporal attention CNN