Abstract:Objetive. Decoding speech from brain activity can enable communication for individuals with speech disorders. Deep neural networks have shown great potential for speech decoding applications. However, the limited availability of large datasets containing neural recordings from speech-impaired subjects poses a challenge. Leveraging data from healthy participants can mitigate this limitation and expedite the development of speech neuroprostheses while minimizing the need for patient-specific training data. Approach. In this study, we collected a substantial dataset consisting of recordings from 56 healthy participants using 64 EEG channels. Multiple neural networks were trained to classify perceived sentences in the Spanish language using subject-independent, mixed-subjects, and fine-tuning approaches. The dataset has been made publicly available to foster further research in this area. Main results. Our results demonstrate a remarkable level of accuracy in distinguishing sentence identity across 30 classes, showcasing the feasibility of training Deep Neural Networks (DNNs) to decode sentence identity from perceived speech using EEG. Notably, the subject-independent approach rendered accuracy comparable to the mixed-subjects approach, although with higher variability among subjects. Additionally, our fine-tuning approach yielded even higher accuracy, indicating an improved capability to adapt to individual subject characteristics, which enhances performance. This suggests that DNNs have effectively learned to decode universal features of brain activity across individuals while also being adaptable to specific participant data. Furthermore, our analyses indicate that EEGNet and DeepConvNet exhibit comparable performance, outperforming ShallowConvNet for sentence identity decoding. Finally, our Grad-CAM visualization analysis identifies key areas influencing the network's predictions, offering valuable insights into the neural processes underlying language perception and comprehension. Significance. These findings advance our understanding of EEG-based speech perception decoding and hold promise for the development of speech neuroprostheses, particularly in scenarios where subjects cannot provide their own training data.

Perception Point: Identifying Critical Learning Periods in Speech for Bilingual Networks

Neural Substrate Underlying the Learning of a Passage with Unfamiliar Vocabulary and Syntax.

Developmental Predictive Coding Model for Early Infancy Mono and Bilingual Vocal Continual Learning

Evaluating computational models of infant phonetic learning across languages

Unraveling Predictive Mechanism in Speech Perception and Production: Insights from EEG Analyses of Brain Network Dynamics

Modeling early phonetic acquisition from child-centered audio data

Revisiting perceptual sensitivity to non-native speech in a diverse sample of bilinguals

Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigation

A model of infant speech perception and learning

Sketching the Landscape of Speech Perception Research (2000–2020): A Bibliometric Study

Neural Speech Decoding During Audition, Imagination and Production

Assessing language acquisition from parent-child interaction: An event-related potential study on perception of audio-visual cues in infancy

When Jack isn't Jacques: Simultaneous opposite language-specific speech perceptual learning in French-English bilinguals

A computational model of early language acquisition from audiovisual experiences of young infants

Emergence of the neural network underlying phonological processing from the prereading to the emergent reading stage: A longitudinal study

Identification of perceived sentences using deep neural networks in EEG

Task-Specific Rapid Auditory Perceptual Learning in Adult Cochlear Implant Recipients: What Could It Mean for Speech Recognition

Foreign-language experience in infancy: Effects of short-term exposure and social interaction on phonetic learning

Neural Dynamics of the Processing of Speech Features: Evidence for a Progression of Features from Acoustic to Sentential Processing

The formation of perceptual space in early phonetic acquisition: a cross-linguistic modeling approach

Perceptual Narrowing in Speech and Face Recognition: Evidence for Intra-individual Cross-Domain Relations