Abstract:Objective: Brain-computer interface (BCI) can translate intentions directly into instructions and greatly improve the interaction experience for disabled people or some specific interactive applications. To improve the efficiency of BCI, the objective of this study is to explore the feasibility of an audio-assisted visual BCI speller and a deep learning-based single-trial event related potentials (ERP) decoding strategy. Approach: In this study, a two-stage BCI speller combining the motion-onset visual evoked potential (mVEP) and semantically congruent audio evoked ERP was designed to output the target characters. In the first stage, the different group of characters were presented in the different locations of visual field simultaneously and the stimuli were coded to the mVEP based on a new space division multiple access scheme. And then, the target character can be output based on the audio-assisted mVEP in the second stage. Meanwhile, a spatial-temporal attention-based convolutional neural network (STA-CNN) was proposed to recognize the single-trial ERP components. The CNN can learn 2-dimentional features including the spatial information of different activated channels and time dependence among ERP components. In addition, the STA mechanism can enhance the discriminative event-related features by adaptively learning probability weights. Main results: The performance of the proposed two-stage audio-assisted visual BCI paradigm and STA-CNN model was evaluated using the Electroencephalogram (EEG) recorded from 10 subjects. The average classification accuracy of proposed STA-CNN can reach 59.6 and 77.7% for the first and second stages, which were always significantly higher than those of the comparison methods ( p < 0.05). Significance: The proposed two-stage audio-assisted visual paradigm showed a great potential to be used to BCI speller. Moreover, through the analysis of the attention weights from time sequence and spatial topographies, it was proved that STA-CNN could effectively extract interpretable spatiotemporal EEG features.

Two Frequencies Sequential Coding for the ASSR-based Brain-Computer Interface Application

A Novel Multiple Time-Frequency Sequential Coding Strategy for Hybrid Brain-Computer Interface

Multiple Frequencies Sequential Coding for SSVEP-based Brain-Computer Interface.

Author Response for "single Stimulus Location for Two Inputs: A Combined Brain‐computer Interface Based on Steady State Visual Evoked Potential (SSVEP)"

Auditory Steady-State Response Stimuli based BCI Application - The Optimization of the Stimuli Types and Lengths

Right-and-left Field Stimulation with Two Frequencies for a SSVEP-based Brain-Computer Interface

The Realization of Bci Multi-Dimension Control Based on Time-Frequency Coding

Research on High-Frequency Combination Coding-Based SSVEP-BCIs and Its Signal Processing Algorithms

A multi-channel SSVEP-based brain-computer interface using a canonical correlation analysis in the frequency domain

A sub-region combination scheme for spatial coding in a high-frequency SSVEP-based BCI

High-Frequency Ssvep Stimulation Paradigm Based On Dual Frequency Modulation

Correlated Component Analysis for Enhancing the Performance of SSVEP-Based Brain-Computer Interface.

A Frequency Recognition Method Based on Multitaper Spectral Analysis and SNR Estimation for SSVEP-based Brain-Computer Interface.

Right-and-left Visual Field Stimulation: A Frequency and Space Mixed Coding Method for SSVEP Based Brain-Computer Interface

An online multi-channel SSVEP-based brain–computer interface using a canonical correlation analysis method

Hybrid frequency and phase coding for a high-speed SSVEP-based BCI speller.

Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain–computer interface

Two-stage Frequency Recognition Method Based on Correlated Component Analysis for SSVEP-based BCI

SSVEP-Based Brain-Computer Interface With a Limited Number of Frequencies Based on Dual-Frequency Biased Coding

A novel brain-computer interface based on audio-assisted visual evoked EEG and spatial-temporal attention CNN

Digital Spelling BCI Based on Visual-Auditory Associate Stimulation