Abstract:Introduction The goal of communication is delivery of the content of a message. Vocalizations carry a symbolic representation of a message from the mind of the speaker to a listener (human or machine) who must decode the representation. Machine-based speech interpreters have libraries of sound templates and word and grammar logic systems that assist in decoding a message. Human listeners have some additional information to assist interpretation such as the context of a discourse: the current topic, the time, etc. Though human-to-human discourse is held as the model for delivery of a message from a human to a machine, even speech between humans is sometimes fraught with difficulty in passing meaning. Can we improve sending the meaning of a message from a speaker? In the mind of the speaker is located the content of a message. Can we access information related to that content and help deliver the message? The production of an utterance by the brain of a speaker involves the setting of a message, selection of vocabulary, determination of syntax, fixing of semantics and creation and execution of a motor program that activates the voice. We suggest that it will be possible to assist in content delivery of a message by accessing neural signals related to the production of an utterance. Since 1980, there have been research studies on electrical signals related to brain events related to speech. Current work on these evoked potentials (EPs) has required averaging. However, new techniques are increasing our ability to resolve signal components that are leading us to the "single shot": pattern detection in a single trial. These techniques include: increasing the number of electrodes and new computational methods such as trained neural networks. Importantly, the technique of Independent Components Analysis (ICA) has lead to the ability to discriminate the many signals that summate to create the potentials recorded at the surface of the scalp. ICA initially requires substantial training of a neural network system based on the signals of a large number of electrical sources. However it can then be used to discriminate features of the data set and match a template of those features to subsequent trials. We propose to use the utterance of a word as a timing signal to match ICA templates to the "pre-potential", the EEG signal preceding the utterance. A statistical match could be made with the pre-potential to candidate meanings for the utterance created by the SR system. Preliminary Results We have collected pilot EEG data from a subject who was triggered to utter the sound (tu(I)) at the presentation a visual cue of the randomly presented words "to" or "too". The subject was instructed to speak the word as soon as they saw it presented. The goal was to review the EEG related to the individual trial utterances and determine if there was a robust set of features related to the utterances that could discriminate the different words. EEG data were collected from 28 scalp electrodes and I EOG electrode. Averaged 2-sec EPs related to the visual stimulus presentations under the two different conditions were examined. EPs of two conditions differed largely at site CPz. However, the time courses of single-trials did not always resemble those of the averaged EPs. The trial-to-trial variability made it difficult to distinguish the brain responses under two conditions without further processing. A single-layer perceptron neural network was trained through supervised methods and found a linear boundary between two clusters ('to' and 'too'). The boundary based on the scalp EEG misclassified 19 of a total of 82 trails. ICA was applied to the concatenated 2-sec single-trial epochs from the subject and derived 29 temporally independent components. The averaged activations and scalp topographies of components 6, 8, and 16 suggested that they together account for the large differences in EPs at cite CPz. The trial-to-trial variability was reduced compared to the raw EPs since the contributions of other components were removed from (not projected to) the electrodes. Similarly, the activations of ICA components accounted for the differences in EPs at site T8 were projected and summed at the scalp electrodes. For each trial, the summed projections between 200 and 1400 ms following stimulus presentations at cites CPz and T8 were averaged. The clusters were tighter compared to those from site CPz alone. Again, a single-layer perceptron neural network was trained through supervised methods and found a linear boundary between two clusters ('to' and 'too'). The boundary based on the scalp EEG misclassified I I of a total of 82 traits (13%). Conclusions In this example, ICA separated the distinct brain processes into different independent components and extracted the signals of interest. Mathematically reducing the interference from the other brain or extra-brain activity generators improved the chance to detect small differences in EEG across conditions. It appears feasible that EPs can be used to assist speech recognition.

Responses to Visual Speech in Human Posterior Superior Temporal Gyrus Examined with iEEG Deconvolution

A Double Dissociation between Anterior and Posterior Superior Temporal Gyrus for Processing Audiovisual Speech Demonstrated by Electrocorticography

A Method to Study Global Spatial Patterns Related to Sensory Perception in Scalp EEG.

Inverted encoding of neural responses to audiovisual stimuli reveals super-additive multisensory enhancement

Can you hear me yet? An intracranial investigation of speech and non-speech audiovisual interactions in human cortex

Visual speech differentially modulates beta, theta, and high gamma bands in auditory cortex

Converging Evidence From Electrocorticography and BOLD fMRI for a Sharp Functional Boundary in Superior Temporal Gyrus Related to Multisensory Speech Processing

Intracranial electrophysiology of spectrally degraded speech in the human cortex

Intracranial Mapping of Response Latencies and Task Effects for Spoken Syllable Processing in the Human Brain

Representational interactions during audiovisual speech entrainment: Redundancy in left posterior superior temporal gyrus and synergy in left motor cortex

The timecourse of multisensory speech processing in unilaterally stimulated cochlear implant users revealed by ERPs

Stereotactic electroencephalography in humans reveals multisensory signal in early visual and auditory cortices

Neural Mechanisms Underlying Cross-Modal Phonetic Encoding

Congruent audiovisual speech enhances auditory attention decoding with EEG.

Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex

Congruent Audiovisual Speech Enhances Cortical Envelope Tracking During Auditory Selective Attention.

Entrained audiovisual speech integration implemented by two independent computational mechanisms: Redundancy in left posterior superior temporal gyrus and Synergy in left motor cortex

Electrophysiological differences and similarities in audiovisual speech processing in CI users with unilateral and bilateral hearing loss

Visual speech enhances auditory onset timing and envelope tracking through distinct mechanisms

Augmented higher cognition: Enhancing speech recognition through neural activity measures

Retinotopic effects during spatial audio-visual integration