Abstract:Objective Although many studies have attempted to improve the performance of the visual-based P300-speller system, its performance is still not satisfactory. The current system has limitations for patients with neurodegenerative diseases, in which muscular control of the eyes may be impaired or deteriorate over time. Some studies have shown that the audiovisual stimuli with spatial and semantic congruence elicited larger event-related potential (ERP) amplitudes than do unimodal visual stimuli. Therefore, this study proposed a novel multisensory P300-speller based on audiovisual spatial and semantic congruence. Methods We designed a novel audiovisual P300-speller paradigm (AV spelling paradigm) in which the pronunciation and visual presentation of characters were matched in spatial position and semantics. We analyzed the ERP waveforms elicited in the AV spelling paradigm and visual-based spelling paradigm (V spelling paradigm) and compared the classification accuracies between these two paradigms. Results ERP analysis revealed significant differences in ERP amplitudes between the two paradigms in the following areas (AV > V): the frontal area at 60–140 ms, frontal–central–parietal area at 360–460 ms, frontal area at 700–800 ms, right temporal area at 380–480 and 700–780 ms, and left temporal area at 500–780 ms. Offline classification results showed that the accuracies were significantly higher in the AV spelling paradigm than in the V spelling paradigm after superposing 1, 2, 5, 6, 9, and 10 times (P < 0.05), and there were trends toward improvement in the accuracies at superposing 3, 4, 7, and 8 times (P = 0.06). Similar results were found for information transfer rate between V and AV spelling paradigms at 1, 2, 5, 6, and 10 superposition times (P < 0.05). Significance The proposed audiovisual P300-speller paradigm significantly improved the classification accuracies compared with the visual-based P300-speller paradigm. Our novel paradigm combines spatial and semantic features of two sensory modalities, and the present findings provide valuable insights into the development of multimodal ERP-based BCI paradigms.

Auditory Dominance in Processing Chinese Semantic Abnormalities in Response to Competing Audio-visual Stimuli.

The Effects of Semantic Congruency: a Research of Audiovisual P300-speller.

Brain Hemispheres Swap Dominance for Processing Semantically Meaningful Pitch

Opposite Patterns of Hemisphere Dominance for Early Auditory Processing of Lexical Tones and Consonants.

Cortical competition during language discrimination.

Sensory Dominance Depends on Locus of Attentional Selection: Cross-Modal Distraction at Different Levels of Information Processing

Prosody Dominates over Semantics in Emotion Word Processing: Evidence from Cross-Channel and Cross-Modal Stroop Effects

Vision Dominates At The Preresponse Level And Audition Dominates At The Response Level In Cross-Modal Interaction: Behavioral And Neural Evidence

Multisensory Competition Is Modulated by Sensory Pathway Interactions with Fronto-Sensorimotor and Default-Mode Network Regions

The different brain areas occupied for integrating information of hierarchical linguistic units: a study based on EEG and TMS

Audiovisual Congruency and Incongruency Effects on Auditory Intensity Discrimination.

Distinct mechanisms underlying cross-modal semantic conflict and response conflict processing

Audiovisual Emotional Congruency Modulates the Stimulus-Driven Cross-Modal Spread of Attention.

Hemispheric Specialization in Auditory Processing of Chinese Lexical Tones: a Study Using Whole-Head Recordings of EEG

Interaction between Phonological and Semantic Processes in Visual Word Recognition using Electrophysiology

Auditory change detection and visual selective attention: association between MMN and N2pc

Testing Native Language Neural Commitment at the Brainstem Level: A Cross-Linguistic Investigation of the Association Between Frequency-Following Response and Speech Perception.

The Effect of Semantic Congruence for Visual-Auditory Bimodal Stimuli

A Novel Audiovisual P300-Speller Paradigm Based on Cross-Modal Spatial and Semantic Congruence

Combining Behavioral and ERP Methodologies to Investigate the Differences Between McGurk Effects Demonstrated by Cantonese and Mandarin Speakers.

Brain activation during phonological and semantic processing of Chinese characters in deaf signers.