Abstract:Virtual reality environments offer great opportunities to study the performance of brain-computer interfaces (BCIs) in real-world contexts. As real-world stimuli are typically multimodal, their neuronal integration elicits complex response patterns. To investigate the effect of additional auditory cues on the processing of visual information, we used virtual reality to mimic safety-related events in an industrial environment while we concomitantly recorded electroencephalography (EEG) signals. We simulated a box traveling on a conveyor belt system where two types of stimuli – an exploding and a burning box – interrupt regular operation. The recordings from 16 subjects were divided into two subsets, a visual-only and an audio-visual experiment. In the visual-only experiment, the response patterns for both stimuli elicited a similar pattern – a visual evoked potential (VEP) followed by an event-related potential (ERP) over the occipital-parietal lobe. Moreover, we found the perceived severity of the event to be reflected in the signal amplitude. Interestingly, the additional auditory cues had a twofold effect on the previous findings: The P1 component was significantly suppressed in the case of the exploding box stimulus, whereas the N2c showed an enhancement for the burning box stimulus. This result highlights the impact of multisensory integration on the performance of realistic BCI applications. Indeed, we observed alterations in the offline classification accuracy for a detection task based on a mixed feature extraction (variance, power spectral density, and discrete wavelet transform) and a support vector machine classifier. In the case of the explosion, the accuracy slightly decreased by –1.64% p. in an audio-visual experiment compared to the visual-only. Contrarily, the classification accuracy for the burning box increased by 5.58% p. when additional auditory cues were present. Hence, we conclude, that especially in challenging detection tasks, it is favorable to consider the potential of multisensory integration when BCIs are supposed to operate under (multimodal) real-world conditions.

The Effect of Semantic Congruence for Visual-Auditory Bimodal Stimuli

Combining Motor Imagery with Selective Sensation Toward a Hybrid-Modality BCI

Selective Sensation Based Brain-Computer Interface Via Mechanical Vibrotactile Stimulation.

Performance of Brain–Computer Interfacing Based on Tactile Selective Sensation and Motor Imagery

Effect of Competing Stimuli for Steady-State Visually Evoked Potential and Steady-State Motion Visually Evoked Potential

Author Response for "single Stimulus Location for Two Inputs: A Combined Brain‐computer Interface Based on Steady State Visual Evoked Potential (SSVEP)"

Psychophysical Responses Comparison in Spatial Visual, Audiovisual, and Auditory BCI-Spelling Paradigms

Reproducibility And Discriminability Of Brain Patterns Of Semantic Categories Enhanced By Congruent Audiovisual Stimuli

The dissociation of semantically congruent and incongruent cross-modal effects on the visual attentional blink

Does Inter-Stimulus Distance Influence the Decoding Performance of SSVEP and SSMVEP BCI?

Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting

Semantically Congruent Visual Stimuli Can Improve Auditory Memory

A new auditory multi-class brain-computer interface paradigm: spatial hearing as an informative cue

The Effect of SOA on An Asynchronous ERP and VEP-Based BCI

Auditory Dominance in Processing Chinese Semantic Abnormalities in Response to Competing Audio-visual Stimuli.

Performance Evaluation of Visual Noise Imposed Stochastic Resonance Effect on Brain-Computer Interface Application: A Comparison Between Motion-Reversing Simple Ring and Complex Checkerboard Patterns

A Novel Stimulation Method for Multi-Class SSVEP-BCI Using Intermodulation Frequencies

Happy emotion cognition of bimodal audiovisual stimuli optimizes the performance of the P300 speller

An Auditory Brain-Computer Interface Using Virtual Sound Field

Investigation of Visual Stimulus With Various Colors and the Layout for the Oddball Paradigm in Evoked Related Potential-Based Brain–Computer Interface

Design And Implementation Of An Asynchronous Bci System With Alpha Rhythm And Ssvep