Abstract:Damage or degeneration of motor pathways necessary for speech and other movements, as in brainstem strokes or amyotrophic lateral sclerosis (ALS), can interfere with efficient communication without affecting brain structures responsible for language or cognition. In the worst-case scenario, this can result in the locked in syndrome (LIS), a condition in which individuals cannot initiate communication and can only express themselves by answering yes/no questions with eye blinks or other rudimentary movements. Existing augmentative and alternative communication (AAC) devices that rely on eye tracking can improve the quality of life for people with this condition, but brain-computer interfaces (BCIs) are also increasingly being investigated as AAC devices, particularly when eye tracking is too slow or unreliable. Moreover, with recent and ongoing advances in machine learning and neural recording technologies, BCIs may offer the only means to go beyond cursor control and text generation on a computer, to allow real-time synthesis of speech, which would arguably offer the most efficient and expressive channel for communication. The potential for BCI speech synthesis has only recently been realized because of seminal studies of the neuroanatomical and neurophysiological underpinnings of speech production using intracranial electrocorticographic (ECoG) recordings in patients undergoing epilepsy surgery. These studies have shown that cortical areas responsible for vocalization and articulation are distributed over a large area of ventral sensorimotor cortex, and that it is possible to decode speech and reconstruct its acoustics from ECoG if these areas are recorded with sufficiently dense and comprehensive electrode arrays. In this article, we review these advances, including the latest neural decoding strategies that range from deep learning models to the direct concatenation of speech units. We also discuss state-of-the-art vocoders that are integral in constructing natural-sounding audio waveforms for speech BCIs. Finally, this review outlines some of the challenges ahead in directly synthesizing speech for patients with LIS.

AI & robotics briefing: AI decodes languages in first 'bilingual' brain-reading device

First 'bilingual' brain-reading device decodes Spanish and English words

AI & robotics briefing: Lack of transparency surrounds Neuralink's 'brain-reading' chip

Artificial Intelligence Based Multimodal Language Decoding from Brain Activity: A Review

Decoding Linguistic Representations of Human Brain

A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages

AI & robotics briefing: How psychology and neuroscience crack open the AI 'black box'

Brain–machine-interface device translates internal speech into text

Brain-reading device is best yet at decoding 'internal speech'

Mind-reading devices are revealing the brain's secrets

A high-performance speech neuroprosthesis

A high-performance neuroprosthesis for speech decoding and avatar control

Translating neural signals to text using a Brain-Machine Interface

Speech synthesis from neural decoding of spoken sentences

A neural decoding algorithm that generates language from visual activity evoked by natural images

Brain-Computer Interface: Applications to Speech Decoding and Synthesis to Augment Communication

A neural speech decoding framework leveraging deep learning and speech synthesis

Real-Time Deep Neurolinguistic Learning Enhances Noninvasive Neural Language Decoding for Brain–Machine Interaction

NeuSpeech: Decode Neural signal as Speech

Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models

Speech decoding from stereo-electroencephalography (sEEG) signals using advanced deep learning methods