Abstract:Damage or degeneration of motor pathways necessary for speech and other movements, as in brainstem strokes or amyotrophic lateral sclerosis (ALS), can interfere with efficient communication without affecting brain structures responsible for language or cognition. In the worst-case scenario, this can result in the locked in syndrome (LIS), a condition in which individuals cannot initiate communication and can only express themselves by answering yes/no questions with eye blinks or other rudimentary movements. Existing augmentative and alternative communication (AAC) devices that rely on eye tracking can improve the quality of life for people with this condition, but brain-computer interfaces (BCIs) are also increasingly being investigated as AAC devices, particularly when eye tracking is too slow or unreliable. Moreover, with recent and ongoing advances in machine learning and neural recording technologies, BCIs may offer the only means to go beyond cursor control and text generation on a computer, to allow real-time synthesis of speech, which would arguably offer the most efficient and expressive channel for communication. The potential for BCI speech synthesis has only recently been realized because of seminal studies of the neuroanatomical and neurophysiological underpinnings of speech production using intracranial electrocorticographic (ECoG) recordings in patients undergoing epilepsy surgery. These studies have shown that cortical areas responsible for vocalization and articulation are distributed over a large area of ventral sensorimotor cortex, and that it is possible to decode speech and reconstruct its acoustics from ECoG if these areas are recorded with sufficiently dense and comprehensive electrode arrays. In this article, we review these advances, including the latest neural decoding strategies that range from deep learning models to the direct concatenation of speech units. We also discuss state-of-the-art vocoders that are integral in constructing natural-sounding audio waveforms for speech BCIs. Finally, this review outlines some of the challenges ahead in directly synthesizing speech for patients with LIS.

The speech neuroprosthesis

A high-performance neuroprosthesis for speech decoding and avatar control

A high-performance speech neuroprosthesis

An accurate and rapidly calibrating speech neuroprosthesis

A Speech Neuroprosthesis in the Frontal Lobe and Hippocampus: Decoding High-Frequency Activity into Phonemes

A wireless brain-machine interface for real-time speech synthesis

Speech synthesis from neural decoding of spoken sentences

Real-time Synthesis of Imagined Speech Processes from Minimally Invasive Recordings of Neural Activity

Generalizable spelling using a speech neuroprosthesis in an individual with severe limb and vocal paralysis

An instantaneous voice synthesis neuroprosthesis

Brain-Computer Interface: Applications to Speech Decoding and Synthesis to Augment Communication

Representation of Verbal Thought in Motor Cortex and Implications for Speech Neuroprostheses

The Potential for a Speech Brain–Computer Interface Using Chronic Electrocorticography

Implanted cortical neuroprosthetics for speech and movement restoration

Speech decoding from a small set of spatially segregated minimally invasive intracranial EEG electrodes with a compact and interpretable neural network

Synthesizing Speech from Intracranial Depth Electrodes using an Encoder-Decoder Framework

Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models

Progress in speech decoding from the electrocorticogram

A bilingual speech neuroprosthesis driven by cortical articulatory representations shared between languages

High-resolution neural recordings improve the accuracy of speech decoding

A neural speech decoding framework leveraging deep learning and speech synthesis