Abstract:Human reading behavior is tuned to the statistics of natural language: the time it takes human subjects to read a word can be predicted from estimates of the word's probability in context. However, it remains an open question what computational architecture best characterizes the expectations deployed in real time by humans that determine the behavioral signatures of reading. Here we test over two dozen models, independently manipulating computational architecture and training dataset size, on how well their next-word expectations predict human reading time behavior on naturalistic text corpora. We find that across model architectures and training dataset sizes the relationship between word log-probability and reading time is (near-)linear. We next evaluate how features of these models determine their psychometric predictive power, or ability to predict human reading behavior. In general, the better a model's next-word expectations, the better its psychometric predictive power. However, we find nontrivial differences across model architectures. For any given perplexity, deep Transformer models and n-gram models generally show superior psychometric predictive power over LSTM or structurally supervised neural models, especially for eye movement data. Finally, we compare models' psychometric predictive power to the depth of their syntactic knowledge, as measured by a battery of syntactic generalization tests developed using methods from controlled psycholinguistic experiments. Once perplexity is controlled for, we find no significant relationship between syntactic knowledge and predictive power. These results suggest that different approaches may be required to best model human real-time language comprehension behavior in naturalistic reading versus behavior for controlled linguistic materials designed for targeted probing of syntactic knowledge.

Humans and language models diverge when predicting repeating text

On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior

Language models are better than humans at next-token prediction

Predict the Next Word: Humans exhibit uncertainty in this task and language models _____

Divergences between Language Models and Human Brains

Language Models Outperform Cloze Predictability in a Cognitive Model of Reading

Language models and brains align due to more than next-word prediction and word-level information

A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans

Language models emulate certain cognitive profiles: An investigation of how predictability measures interact with individual differences

Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned

Embers of autoregression show how large language models are shaped by the problem they are trained to solve

Do Large Language Models Perform the Way People Expect? Measuring the Human Generalization Function

Psychometric Predictive Power of Large Language Models

Testing the limits of natural language models for predicting human language judgments

Multilingual Language Models Predict Human Reading Behavior

Towards a Psychology of Machines: Large Language Models Predict Human Memory

Can language models handle recursively nested grammatical structures? A case study on comparing models and humans

Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling

A Sentence is Worth a Thousand Pictures: Can Large Language Models Understand Hum4n L4ngu4ge and the W0rld behind W0rds?