Abstract:Human reading behavior is tuned to the statistics of natural language: the time it takes human subjects to read a word can be predicted from estimates of the word's probability in context. However, it remains an open question what computational architecture best characterizes the expectations deployed in real time by humans that determine the behavioral signatures of reading. Here we test over two dozen models, independently manipulating computational architecture and training dataset size, on how well their next-word expectations predict human reading time behavior on naturalistic text corpora. We find that across model architectures and training dataset sizes the relationship between word log-probability and reading time is (near-)linear. We next evaluate how features of these models determine their psychometric predictive power, or ability to predict human reading behavior. In general, the better a model's next-word expectations, the better its psychometric predictive power. However, we find nontrivial differences across model architectures. For any given perplexity, deep Transformer models and n-gram models generally show superior psychometric predictive power over LSTM or structurally supervised neural models, especially for eye movement data. Finally, we compare models' psychometric predictive power to the depth of their syntactic knowledge, as measured by a battery of syntactic generalization tests developed using methods from controlled psycholinguistic experiments. Once perplexity is controlled for, we find no significant relationship between syntactic knowledge and predictive power. These results suggest that different approaches may be required to best model human real-time language comprehension behavior in naturalistic reading versus behavior for controlled linguistic materials designed for targeted probing of syntactic knowledge.

Temporal Kernel Neural Network Language Model

A neural probabilistic language model

Neural Network Language Modeling With Letter-Based Features And Importance Sampling

Temporal Segment Convolutional Kernel Networks for Sequence Modeling of Videos

Deep Neural Networks Language Model Based on CNN and LSTM Hybrid Architecture

Long-Short Range Context Neural Networks for Language Modeling

A Study on Neural Network Language Modeling

A Latent Variable Recurrent Neural Network for Discourse Relation Language Models

Improved TDNNs Using Deep Kernels and Frequency Dependent Grid-RNNs.

NEWLSTM: an Optimized Long Short-Term Memory Language Model for Sequence Prediction.

Biologically Plausible Sparse Temporal Word Representations

TKAN: Temporal Kolmogorov-Arnold Networks

RECURRENT NEURAL NETWORK BASED LANGUAGE MODELING WITH CONTROLLABLE EXTERNAL MEMORY

Spontaneous Temporal Grouping Neural Network for Long-Term Memory Modeling

Mind the Gap: Assessing Temporal Generalization in Neural Language Models

On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behavior

Multi-Timescale Long Short-Term Memory Neural Network for Modelling Sentences and Documents

Large Margin Neural Language Model

Nonrecurrent Neural Structure for Long-Term Dependence.

Stochastic Learning of Nonstationary Kernels for Natural Language Modeling

Neural Network-Based Modeling of Phonetic Durations