Integrating large language models and active inference to understand eye movements in reading and dyslexia

Francesco Donnarumma,Mirco Frosolone,Giovanni Pezzulo

2024-02-24

Abstract:We present a novel computational model employing hierarchical active inference to simulate reading and eye movements. The model characterizes linguistic processing as inference over a hierarchical generative model, facilitating predictions and inferences at various levels of granularity, from syllables to sentences. Our approach combines the strengths of large language models for realistic textual predictions and active inference for guiding eye movements to informative textual information, enabling the testing of predictions. The model exhibits proficiency in reading both known and unknown words and sentences, adhering to the distinction between lexical and nonlexical routes in dual-route theories of reading. Notably, our model permits the exploration of maladaptive inference effects on eye movements during reading, such as in dyslexia. To simulate this condition, we attenuate the contribution of priors during the reading process, leading to incorrect inferences and a more fragmented reading style, characterized by a greater number of shorter saccades. This alignment with empirical findings regarding eye movements in dyslexic individuals highlights the model's potential to aid in understanding the cognitive processes underlying reading and eye movements, as well as how reading deficits associated with dyslexia may emerge from maladaptive predictive processing. In summary, our model represents a significant advancement in comprehending the intricate cognitive processes involved in reading and eye movements, with potential implications for understanding and addressing dyslexia through the simulation of maladaptive inference. It may offer valuable insights into this condition and contribute to the development of more effective interventions for treatment.

Neurons and Cognition,Computation and Language

What problem does this paper attempt to address?

The problem this paper attempts to address is the construction of a novel computational model to simulate eye movement behavior during reading by combining large language models (LLM) and active inference. Specifically, the model aims to:

1. **Understand eye movement behavior during reading**: By viewing language processing as an inference process based on a hierarchical generative model, the model can make predictions and inferences at different levels of granularity (from syllables to sentences).
2. **Explore abnormal inference effects in reading disorders (such as dyslexia)**: By weakening the contribution of prior information during the reading process, the model can simulate eye movement patterns similar to those of dyslexic patients, characterized by more frequent short saccades.
3. **Validate the dual-route reading theory**: The model can distinguish between the lexical route and the nonlexical route, thereby better understanding the cognitive mechanisms involved in the reading process.
4. **Provide insights into the potential mechanisms of reading disorders**: By simulating abnormal predictive processing in reading disorders, the model helps to understand the cognitive basis of reading disorders and may offer valuable insights for therapeutic interventions.

In summary, the goal of this paper is to combine large language models and active inference to establish a computational model capable of simulating both normal and abnormal reading processes, thereby gaining a deeper understanding of the cognitive processes involved in reading and eye movement behavior, particularly the mechanisms related to reading disorders.