Decoding Reading Goals from Eye Movements

Omer Shubi,Cfir Avraham Hadar,Yevgeni Berzak

2024-10-28

Abstract:Readers can have different goals with respect to the text they are reading. Can these goals be decoded from the pattern of their eye movements over the text? In this work, we examine for the first time whether it is possible to decode two types of reading goals that are common in daily life: information seeking and ordinary reading. Using large scale eye-tracking data, we apply to this task a wide range of state-of-the-art models for eye movements and text that cover different architectural and data representation strategies, and further introduce a new model ensemble. We systematically evaluate these models at three levels of generalization: new textual item, new participant, and the combination of both. We find that eye movements contain highly valuable signals for this task. We further perform an error analysis which builds on prior empirical findings on differences between ordinary reading and information seeking and leverages rich textual annotations. This analysis reveals key properties of textual items and participant eye movements that contribute to the difficulty of the task.

Computation and Language

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper attempts to solve the problem of decoding readers' reading goals from their eye movement patterns. Specifically, the researchers focus on two common everyday reading goals: information seeking and ordinary reading. Information seeking refers to the behavior of readers looking for specific information in the text, while ordinary reading usually refers to the general understanding of the text by the readers.

### Research Background and Motivation

1. **Importance of Reading Behavior**: Reading is a universal skill and indispensable in modern society. During reading, the reader's eyes move in a jumping manner, forming a series of fixations and saccades.
2. **Research Value of Eye Movement Data**: Eye movement data is considered to contain rich information that can reflect how readers interact with the text. Automatically decoding this information is a current research hotspot.
3. **Limitations of Existing Research**: Although research in the fields of cognitive science and natural language processing (NLP) has widely used eye movement data, most studies have focused mainly on ordinary reading, with other types of reading goals (such as information seeking) receiving less attention.

### Research Objectives

- **Task Definition**: Predicting whether the reading goal is information seeking or ordinary reading from the eye movement patterns of a single participant reading a single paragraph.
- **Model Evaluation**: Applying various state-of-the-art eye movement and text processing models to systematically evaluate their performance at different levels of generalization, including new text items, new participants, and a combination of both.
- **Error Analysis**: Revealing key factors affecting task difficulty through detailed text annotation and statistical modeling.

### Main Contributions

1. **New Task**: Introducing a new decoding task, predicting the reading goal from the eye movement patterns of a single participant.
2. **Modeling and Evaluation**: Adapting and applying 10 different state-of-the-art prediction models and introducing an ensemble model, demonstrating the performance of these models at different levels of generalization.
3. **Error Analysis**: Conducting systematic error analysis through statistical modeling and detailed text annotation, revealing key axes of variation that affect task difficulty.

### Experimental Setup

- **Dataset**: Using the OneStop dataset, which contains extensive eye movement data under both ordinary reading and information seeking reading goals.
- **Model Training and Evaluation**: Using 10-fold cross-validation to evaluate the models' generalization ability under new text items, new participants, and a combination of both.
- **Baseline Models**: Introducing two simple baseline models, namely a majority class classifier and a reading time-based classifier.

### Results

- **Model Performance**: The RoBERTa-Eye-F model performed the best under all evaluation conditions, especially in terms of generalization ability under new text items and new participants.
- **Ensemble Model**: The introduced simple logistic regression ensemble model further improved overall performance, particularly under the new text item condition.
- **Error Analysis**: Detailed analysis revealed key factors affecting task difficulty, including reading speed, paragraph position, and answer correctness.

### Conclusion

The paper successfully decodes readers' reading goals from eye movement data by introducing a new decoding task and various advanced models. The research results not only demonstrate the potential of eye movement data in decoding reading goals but also provide important references for future research.