Abstract:Speech comprehension involves detecting words and interpreting their meaning according to the preceding semantic context. This process is thought to be underpinned by a predictive neural system that uses that context to anticipate upcoming words. Recent work demonstrated that such a predictive process can be probed from neural signals recorded during ecologically-valid speech listening tasks by using linear lagged models, such as the temporal response function. This is typically done by extracting stimulus features, such as the estimated word-level surprise, and relate such features to the neural signal. While modern large language models (LLM) have led to a substantial leap forward on how word-level features and predictions are modelled, there has been little progress made towards the metrics used for evaluating how well a model is relating stimulus features and neural signals. In fact, previous studies relied on evaluation metrics that were designed for studying continuous univariate sound features, such as the sound envelope, without considering the different requirements of word-level features, which are discrete and sparse in nature. As a result, studies probing lexical prediction mechanisms in ecologically-valid experiments typically exhibit small effect-sizes, severely limiting the type of observations that can be drawn and leaving considerable uncertainty on how exactly our brains build lexical predictions. First, the present study discusses and quantifies these limitations on both simulated and actual electroencephalography signals capturing responses to a speech comprehension task. Second, we tackle the issue by introducing two assessment metrics for the neural encoding of lexical surprise that substantially improve the state-of-the-art. The new metrics were tested on both the simulated and actual electroencephalography datasets, demonstrating effect-sizes over 140% larger than those for the vanilla temporal response function evaluation.

Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?

Testing the Predictions of Surprisal Theory in 11 Languages

Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differences

Unused information in token probability distribution of generative LLM: improving LLM reading comprehension through calculation of expected values

Probabilistic Predictions of People Perusing: Evaluating Metrics of Language Model Performance for Psycholinguistic Modeling

Evaluation of an Algorithmic‐Level Left‐Corner Parsing Account of Surprisal Effects

On the Proper Treatment of Tokenization in Psycholinguistics

Temperature-scaling surprisal estimates improve fit to human reading times -- but does it do so for the "right reasons"?

Information properties of morphologically complex words modulate brain activity during word reading

Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokens

The Effect of Surprisal on Reading Times in Information Seeking and Repeated Reading

Prediction at the intersection of sentence context and word form: Evidence from eye-movements and self-paced reading

Evaluating the timecourses of morpho-orthographic, lexical, and grammatical processing following rapid parallel visual presentation: an EEG investigation in English.

Investigating the Utility of Surprisal from Large Language Models for Speech Synthesis Prosody

What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length

Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities

Robust assessment of the cortical encoding of word-level expectations using the temporal response function

Tokenization Falling Short: On Subword Robustness in Large Language Models

The Effectiveness of Morphology-aware Segmentation in Low-Resource Neural Machine Translation

A study of universal morphological analysis using morpheme-based, holistic, and neural approaches under various data size conditions

Language Models Outperform Cloze Predictability in a Cognitive Model of Reading