Information-making processes in the speaker's brain drive human conversations forward
Ariel Goldstein,Haocheng Wang,Tom Sheffer,Mariano Schain,Zaid Zada,Leonard Niekerken,Bobbi Aubrey,Samuel A Nastase,Harshvardhan Gazula,Colton Casto,Werner K Doyle,Daniel Friedman,Sasha Devore,Patricia Dugan,Avinatan Hassidim,Michael Brenner,Yossi Matias,Orrin Devinsky,Adeen Flinker,Uri Hasson
DOI: https://doi.org/10.1101/2024.08.27.609946
2024-08-28
Abstract:A conversation following an overly predictable pattern is likely boring and uninformative; conversely, if it lacks structure, it is likely nonsensical. The delicate balance between predictability and surprise has been well studied using information theory during speech perception, focusing on how listeners predict upcoming words based on context and respond to unexpected information. However, less is known about how speakers' brains generate structured yet surprisingly informative speech. This study uses continuous electrocorticography (ECoG) recordings during free, 24/7 conversations to investigate the neural basis of speech production and comprehension. We employed large language models (Llama-2 and GPT-2) to calculate word probabilities based on context and categorized words into probable (top 30%) and improbable (bottom 30%) groups. We then extracted word embeddings from the LLMs and used encoding models to estimate the neural activity while producing or listening to probable and improbable words. Our findings indicate that before word-onset, the human brain functions in opposing, perhaps complementary, ways while listening and speaking. Results show that listeners exhibit increased neural encoding for predictable words before word onset, while speakers show increased encoding for surprising, improbable words. Speakers also show a lower speech production rate before articulating unexpected words, suggesting additional cognitive processes are involved in producing novel information. This indicates that human speech production includes information-making processes for generating informative words that are absent in language models, which primarily rely on statistical probabilities to generate contextually appropriate speech.
Neuroscience