Towards a Better Understanding Human Reading Comprehension with Brain Signals

Ziyi Ye,Xiaohui Xie,Yiqun Liu,Zhihong Wang,Xuesong Chen,Min Zhang,Shaoping Ma

DOI: https://doi.org/10.1145/3485447.3511966

2022-08-17

Abstract:Reading comprehension is a complex cognitive process involving many human brain activities. Plenty of works have studied the patterns and attention allocations of reading comprehension in information retrieval related scenarios. However, little is known about what happens in human brain during reading comprehension and how these cognitive activities can affect information retrieval process. Additionally, with the advances in brain imaging techniques such as electroencephalogram (EEG), it is possible to collect brain signals in almost real time and explore whether it can be utilized as feedback to facilitate information acquisition performance. In this paper, we carefully design a lab-based user study to investigate brain activities during reading comprehension. Our findings show that neural responses vary with different types of reading contents, i.e., contents that can satisfy users' information needs and contents that cannot. We suggest that various cognitive activities, e.g., cognitive loading, semantic-thematic understanding, and inferential processing, underpin these neural responses at the micro-time scale during reading comprehension. From these findings, we illustrate several insights for information retrieval tasks, such as ranking models construction and interface design. Besides, we suggest the possibility of detecting reading comprehension status for a proactive real-world system. To this end, we propose a Unified framework for EEG-based Reading Comprehension Modeling (UERCM). To verify its effectiveness, we conduct extensive experiments based on EEG features for two reading comprehension tasks: answer sentence classification and answer extraction. Results show that it is feasible to improve the performance of two tasks with brain signals.

Information Retrieval,Artificial Intelligence,Information Theory

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to understand the specific situation of human brain activities during the process of reading comprehension, and how these cognitive activities affect the information retrieval process. Specifically, the research aims to:

1. **Detect the differences in brain activities triggered by different types of reading content**: Researchers hope to understand whether there are detectable differences in brain activities when users encounter content that can meet their information needs and content that cannot meet their information needs during the reading process.
2. **Reveal the cognitive basis behind these differences and their implications for information retrieval**: By analyzing these differences, researchers hope to reveal how different cognitive activities (such as cognitive load, semantic - topic understanding, reasoning processing) affect reading comprehension on a micro - time scale, and gain insights into information retrieval tasks (such as ranking model construction and interface design).
3. **Explore the possibility of using EEG signals to detect reading states**: Researchers proposed a unified EEG - based reading comprehension modeling framework (UERCM) and verified its effectiveness in answering sentence classification and answer extraction tasks.

To answer these questions, researchers designed a laboratory - based user study, using EEG devices to record the brain activities of participants during the reading comprehension process, and conducted a detailed analysis through methods such as event - related potential (ERP) analysis. The experimental results show that EEG signals can be used as valuable feedback to enhance human - computer interaction, especially during the reading comprehension process.

### Summary of research questions

- **RQ1**: Are there detectable differences in the brain's response to key information and ordinary information during the reading comprehension process?
- **RQ2**: If there are differences, what are the cognitive bases of these differences? What are their implications for information retrieval?
- **RQ3**: Can these differences be used to classify answering sentences and locate potential answer words?

### Main findings

1. **N100 - P200 components**: Related to cognitive load, showing a significantly higher amplitude in answering words, indicating that users need fewer cognitive resources when locating answers.
2. **N400 component**: Related to semantic expectation, the N400 negative wave of answering words is smaller, indicating that it has higher expectation in the current semantic context.
3. **P600 component**: Related to semantic - topic anomalies and reasoning processing, the P600 positive wave of answering words is the largest, indicating that reasoning processing is initiated in the brain.

These findings not only reveal the neural basis in the reading comprehension process, but also provide new ideas and methods for the improvement of information retrieval systems.