Learning to Infer Human Attention in Daily Activities.

Zhixiong Nan,Tianmin Shu,Ran Gong,Shu Wang,Ping Wei,Song-Chun Zhu,Nanning Zheng

DOI: https://doi.org/10.1016/j.patcog.2020.107314

IF: 8

2020-01-01

Pattern Recognition

Abstract:•Different encoder-decoder architectures significantly affect the performance.•The performance improves after considering the task encoding loss.•The pretrain of the neural network contributes to better performance.

What problem does this paper attempt to address?

Deep Neural Networks Evolve Human-like Attention Distribution during Reading Comprehension.

Jiajie Zou,Nai Ding

2021-01-01

Abstract:Attention is a key mechanism for information selection in both biological brains and many state-of-the-art deep neural networks (DNNs). Here, we investigate whether humans and DNNs allocate attention in comparable ways when reading a text passage to subsequently answer a specific question. We analyze 3 transformer-based DNNs that reach human-level performance when trained to perform the reading comprehension task. We find that the DNN attention distribution quantitatively resembles human attention distribution measured by fixation times. Human readers fixate longer on words that are more relevant to the question-answering task, demonstrating that attention is modulated by top-down reading goals, on top of lower-level visual and text features of the stimulus. Further analyses reveal that the attention weights in DNNs are also influenced by both top-down reading goals and lower-level stimulus features, with the shallow layers more strongly influenced by lower-level text features and the deep layers attending more to task-relevant words. Additionally, deep layers' attention to task-relevant words gradually emerges when pre-trained DNN models are fine-tuned to perform the reading comprehension task, which coincides with the improvement in task performance. These results demonstrate that DNNs can evolve human-like attention distribution through task optimization, which suggests that human attention during goal-directed reading comprehension is a consequence of task optimization.
Deep Neural Networks Evolve Human-like Attention Distribution during Goal-directed Reading Comprehension

Jiajie Zou,Nai Ding

DOI: https://doi.org/10.21203/rs.3.rs-813994/v1

2021-01-01

Abstract:Abstract Attention is a key mechanism for information selection in both biological brains and many state-of-the-art deep neural networks (DNNs). Here, we investigate whether humans and DNNs allocate attention in comparable ways when seeking information in a text passage to answer a question. We analyze 3 transformer-based DNNs that reach human-level performance when trained to perform the reading comprehension task. We find that the DNN attention distribution quantitatively resembles human attention distribution measured by eye tracking: Human readers fixate longer on words that are more relevant to the question-answering task, demonstrating that attention is modulated by the top-down reading goal, on top of lower-level visual layout and textual features. Further analyses reveal that the attention weights in DNNs are also influenced by both the top-down reading goal and lower-level textual features, with the shallow layers more strongly influenced by lower-level textual features and the deep layers attending more to task-relevant words. Additionally, deep layers’ attention to task-relevant words gradually emerges when pre-trained DNN models are fine-tuned to perform the reading comprehension task, which coincides with the improvement in task performance. These results demonstrate that DNNs can naturally evolve human-like attention distribution through task optimization. The results suggest that human attention during goal-directed reading comprehension is a consequence of task optimization and the attention weights in DNN are of biological significance.
Task Optimization Leads to Human-like Top-down and Bottom-up Attention during Reading Comprehension

Jiajie Zou,Nai Ding

DOI: https://doi.org/10.21203/rs.3.rs-1034025/v1

2021-01-01

Abstract:Attention is a complex system involving multiple interactive components that jointly regulate information processing in the brain. It has been hypothesized that the computational goal of attention is to optimally integrate information under task demands, and evidence has been provided in relatively simple learning and decision making tasks. It remains unclear, however, whether this hypothesis can explain attention distribution in more complex real-world tasks that engage multiple attention systems. Here, taking advantage of the development of attention mechanisms in deep neural network (DNN) models, we investigate whether human attention during real-world reading comprehension tasks can be explained as a consequence of task optimization. In a goal-directed reading task, participants read a passage to answer a question. Eye tracking results show that the attention on each word, quantified by the fixation time, is modulated by both the top-down reading goal and lower-level visual layout and textual features. When trained to perform the same goal-directed reading task, DNN models yield human-level performance and naturally evolve human-like attention distribution, with deep layers tuned to the reading goal and shallow layers tuned to textual features. Further experiments suggest that different training tasks separately contribute to goal-directed and text-based attention. In summary, the results strongly suggest that human attention can be interpreted as a consequence of task optimization during real-world reading tasks.
Understanding More about Human and Machine Attention in Deep Neural Networks

Qiuxia Lai,Salman Khan,Yongwei Nie,Jianbing Shen,Hanqiu Sun,Ling Shao

DOI: https://doi.org/10.48550/arXiv.1906.08764

2020-07-06

Abstract:Human visual system can selectively attend to parts of a scene for quick perception, a biological mechanism known as Human attention. Inspired by this, recent deep learning models encode attention mechanisms to focus on the most task-relevant parts of the input signal for further processing, which is called Machine/Neural/Artificial attention. Understanding the relation between human and machine attention is important for interpreting and designing neural networks. Many works claim that the attention mechanism offers an extra dimension of interpretability by explaining where the neural networks look. However, recent studies demonstrate that artificial attention maps do not always coincide with common intuition. In view of these conflicting evidence, here we make a systematic study on using artificial attention and human attention in neural network design. With three example computer vision tasks, diverse representative backbones, and famous architectures, corresponding real human gaze data, and systematically conducted large-scale quantitative studies, we quantify the consistency between artificial attention and human visual attention and offer novel insights into existing artificial attention mechanisms by giving preliminary answers to several key questions related to human and artificial attention mechanisms. Overall results demonstrate that human attention can benchmark the meaningful `ground-truth' in attention-driven tasks, where the more the artificial attention is close to human attention, the better the performance; for higher-level vision tasks, it is case-by-case. It would be advisable for attention-driven tasks to explicitly force a better alignment between artificial and human attention to boost the performance; such alignment would also improve the network explainability for higher-level computer vision tasks.

Computer Vision and Pattern Recognition
Human Vs Machine Attention in Neural Networks: A Comparative Study.

Qiuxia Lai,Wenguan Wang,Salman H. Khan,Jianbing Shen,Hanqiu Sun,Ling Shao

2019-01-01

Abstract:Human visual system can selectively attend to parts of a scene for quick perception, a biological mechanism known as Human attention. Inspired by this, recent deep learning models encode attention mechanisms to focus on the most task-relevant parts of the input signal for further processing, which is called Machine/Neural/Artificial attention. Understanding the relation between human and machine attention is important for interpreting and designing neural networks. Many works claim that the attention mechanism offers an extra dimension of interpretability by explaining where the neural networks look. However, recent studies demonstrate that artificial attention maps do not always coincide with common intuition. In view of these conflicting evidence, here we make a systematic study on using artificial attention and human attention in neural network design. With three example computer vision tasks, diverse representative backbones, and famous architectures, corresponding real human gaze data, and systematically conducted large-scale quantitative studies, we quantify the consistency between artificial attention and human visual attention and offer novel insights into existing artificial attention mechanisms by giving preliminary answers to several key questions related to human and artificial attention mechanisms. Overall results demonstrate that human attention can benchmark the meaningful `ground-truth' in attention-driven tasks, where the more the artificial attention is close to human attention, the better the performance; for higher-level vision tasks, it is case-by-case. It would be advisable for attention-driven tasks to explicitly force a better alignment between artificial and human attention to boost the performance; such alignment would also improve the network explainability for higher-level computer vision tasks.
Understanding attention-based encoder-decoder networks: a case study with chess scoresheet recognition

Sergio Y. Hayashi,Nina S. T. Hirata

DOI: https://doi.org/10.1109/ICPR56361.2022.9956133

2024-04-24

Abstract:Deep neural networks are largely used for complex prediction tasks. There is plenty of empirical evidence of their successful end-to-end training for a diversity of tasks. Success is often measured based solely on the final performance of the trained network, and explanations on when, why and how they work are less emphasized. In this paper we study encoder-decoder recurrent neural networks with attention mechanisms for the task of reading handwritten chess scoresheets. Rather than prediction performance, our concern is to better understand how learning occurs in these type of networks. We characterize the task in terms of three subtasks, namely input-output alignment, sequential pattern recognition, and handwriting recognition, and experimentally investigate which factors affect their learning. We identify competition, collaboration and dependence relations between the subtasks, and argue that such knowledge might help one to better balance factors to properly train a network.

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning
An Empirical Study of Spatial Attention Mechanisms in Deep Networks

Xizhou Zhu,Dazhi Chen,Zheng Zhang,Stephen Lin,Jifeng Dai,Dazhi Cheng

DOI: https://doi.org/10.1109/iccv.2019.00679

2019-10-01

Abstract:Attention mechanisms have become a popular component in deep neural networks, yet there has been little examination of how different influencing factors and methods for computing attention from these factors affect performance. Toward a better general understanding of attention mechanisms, we present an empirical study that ablates various spatial attention elements within a generalized attention formulation, encompassing the dominant Transformer attention as well as the prevalent deformable convolution and dynamic convolution modules. Conducted on a variety of applications, the study yields significant findings about spatial attention in deep networks, some of which run counter to conventional understanding. For example, we find that the comparison of query and key content in Transformer attention is negligible for self-attention, but vital for encoder-decoder attention. On the other hand, a proper combination of deformable convolution with key content saliency achieves the best accuracy-efficiency tradeoff in self-attention. Our results suggest that there exists much room for improvement in the design of attention mechanisms.
Learning When to Attend for Neural Machine Translation

Junhui Li,Muhua Zhu

DOI: https://doi.org/10.48550/arXiv.1705.11160

2017-06-01

Abstract:In the past few years, attention mechanisms have become an indispensable component of end-to-end neural machine translation models. However, previous attention models always refer to some source words when predicting a target word, which contradicts with the fact that some target words have no corresponding source words. Motivated by this observation, we propose a novel attention model that has the capability of determining when a decoder should attend to source words and when it should not. Experimental results on NIST Chinese-English translation tasks show that the new model achieves an improvement of 0.8 BLEU score over a state-of-the-art baseline.

Computation and Language
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Sergey Zagoruyko,Nikos Komodakis

DOI: https://doi.org/10.48550/arXiv.1612.03928

2016-12-12

Computer Vision and Pattern Recognition

Abstract:Attention plays a critical role in human visual experience. Furthermore, it has recently been demonstrated that attention can also play an important role in the context of applying artificial neural networks to a variety of tasks from fields such as computer vision and NLP. In this work we show that, by properly defining attention for convolutional neural networks, we can actually use this type of information in order to significantly improve the performance of a student CNN network by forcing it to mimic the attention maps of a powerful teacher network. To that end, we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures. Code and models for our experiments are available at https://github.com/szagoruyko/attention-transfer
Attention module improves both performance and interpretability of 4D fMRI decoding neural network

Zhoufan Jiang,Yanming Wang,ChenWei Shi,Yueyang Wu,Rongjie Hu,Shishuo Chen,Sheng Hu,Xiaoxiao Wang,Bensheng Qiu

2021-01-01

Abstract: Decoding brain cognitive states from neuroimaging signals is an important topic in neuroscience. In recent years, deep neural networks (DNNs) have been recruited for multiple brain state decoding and achieved good performance. However, the open question of how to interpret the DNN black box remains unanswered. Capitalizing on advances in machine learning, we integrated attention modules into brain decoders to facilitate an in-depth interpretation of DNN channels. A 4D convolution operation was also included to extract temporo-spatial interaction within the fMRI signal. The experiments showed that the proposed model obtains a very high accuracy (97.4%) and outperforms previous researches on the 7 different task benchmarks from the Human Connectome Project (HCP) dataset. The visualization analysis further illustrated the hierarchical emergence of task-specific masks with depth. Finally, the model was retrained to regress individual traits within the HCP and to classify viewing images from the BOLD5000 dataset, respectively. Transfer learning also achieves good performance. A further visualization analysis shows that, after transfer learning, low-level attention masks remained similar to the source domain, whereas high-level attention masks changed adaptively. In conclusion, the proposed 4D model with attention module performed well and facilitated interpretation of DNNs, which is helpful for subsequent research.
Predicting Task-Driven Attention Via Integrating Bottom-Up Stimulus and Top-Down Guidance

Zhixiong Nan,Jingjing Jiang,Xiaofeng Gao,Sanping Zhou,Weiliang Zuo,Ping Wei,Nanning Zheng

DOI: https://doi.org/10.1109/tip.2021.3113799

IF: 10.6

2021-01-01

IEEE Transactions on Image Processing

Abstract:Task-free attention has gained intensive interest in the computer vision community while relatively few works focus on task-driven attention (TDAttention). Thus this paper handles the problem of TDAttention prediction in daily scenarios where a human is doing a task. Motivated by the cognition mechanism that human attention allocation is jointly controlled by the top-down guidance and bottom-up stimulus, this paper proposes a cognitively-explanatory deep neural network model to predict TDAttention. Given an image sequence, bottom-up features, such as human pose and motion, are firstly extracted. At the same time, the coarse-grained task information and fine-grained task information are embedded as a top-down feature. The bottom-up features are then fused with the top-down feature to guide the model to predict TDAttention. Two public datasets are re-annotated to make them qualified for TDAttention prediction, and our model is widely compared with other models on the two datasets. In addition, some ablation studies are conducted to evaluate the individual modules in our model. Experiment results demonstrate the effectiveness of our model.
Attention module improves both performance and interpretability of four‐dimensional functional magnetic resonance imaging decoding neural network

Zhoufan Jiang,Yanming Wang,ChenWei Shi,Yueyang Wu,Rongjie Hu,Shishuo Chen,Sheng Hu,Xiaoxiao Wang,Bensheng Qiu

DOI: https://doi.org/10.1002/hbm.25813

IF: 4.8

2022-02-25

Human Brain Mapping

Abstract:Decoding brain cognitive states from neuroimaging signals is an important topic in neuroscience. In recent years, deep neural networks (DNNs) have been recruited for multiple brain state decoding and achieved good performance. However, the open question of how to interpret the DNN black box remains unanswered. Capitalizing on advances in machine learning, we integrated attention modules into brain decoders to facilitate an in-depth interpretation of DNN channels. A four-dimensional (4D) convolution operation was also included to extract temporo-spatial interaction within the fMRI signal. The experiments showed that the proposed model obtains a very high accuracy (97.4%) and outperforms previous researches on the seven different task benchmarks from the Human Connectome Project (HCP) dataset. The visualization analysis further illustrated the hierarchical emergence of task-specific masks with depth. Finally, the model was retrained to regress individual traits within the HCP and to classify viewing images from the BOLD5000 dataset, respectively. Transfer learning also achieves good performance. Further visualization analysis shows that, after transfer learning, low-level attention masks remained similar to the source domain, whereas high-level attention masks changed adaptively. In conclusion, the proposed 4D model with attention module performed well and facilitated interpretation of DNNs, which is helpful for subsequent research.

radiology, nuclear medicine & medical imaging,neurosciences,neuroimaging
Inferring Human Attention by Learning Latent Intentions.

Ping Wei,Dan Xie,Nanning Zheng,Song-Chun Zhu

DOI: https://doi.org/10.24963/ijcai.2017/180

2017-01-01

Abstract:This paper addresses the problem of inferring 3D human attention in RGB-D videos at scene scale. 3D human attention describes where a human is looking in 3D scenes. We propose a probabilistic method to jointly model attention, intentions, and their interactions. Latent intentions guide human attention which conversely reveals the intention features. This mutual interaction makes attention inference a joint optimization with latent intentions. An EM-based approach is adopted to learn the latent intentions and model parameters. Given an RGB-D video with 3D human skeletons, a joint-state dynamic programming algorithm is utilized to jointly infer the latent intentions, the 3D attention directions, and the attention voxels in scene point clouds. Experiments on a new 3D human attention dataset prove the strength of our method.
Attending Via Both Fine-tuning and Compressing.

Jie Zhou,Yuanbin Wu,Qin Chen,Xuanjing Huang,Liang He

DOI: https://doi.org/10.18653/v1/2021.findings-acl.189

2021-01-01

Abstract:Though being a primary trend for enhancing interpretability of neural networks, attention mechanism's reliability and validity are still under debate. In this paper, we try to purify attention scores to obtain a more faithful explanation of downstream models. Specifically, we propose a framework consisting of a learner and a compressor, which performs fine-tuning and compressing iteratively to enhance the performance and interpretability of the attention mechanism. The learner focuses on learning better text representations to achieve good decisions by fine-tuning, while the compressor aims to perform compressions over the representations to retain the most useful clues for explanations with a Variational information bottleneck ATtention (VAT) mechanism. Extensive experiments on eight benchmark datasets show the great advantages of our proposed approach in terms of both performance and interpretability.
Enhancing spiking neural networks with hybrid top-down attention

Faqiang Liu,Rong Zhao

DOI: https://doi.org/10.3389/fnins.2022.949142

IF: 4.3

2022-08-23

Frontiers in Neuroscience

Abstract:As the representatives of brain-inspired models at the neuronal level, spiking neural networks (SNNs) have shown great promise in processing spatiotemporal information with intrinsic temporal dynamics. SNNs are expected to further improve their robustness and computing efficiency by introducing top-down attention at the architectural level, which is crucial for the human brain to support advanced intelligence. However, this attempt encounters difficulties in optimizing the attention in SNNs largely due to the lack of annotations. Here, we develop a hybrid network model with a top-down attention mechanism (HTDA) by incorporating an artificial neural network (ANN) to generate attention maps based on the features extracted by a feedforward SNN. The attention map is then used to modulate the encoding layer of the SNN so that it focuses on the most informative sensory input. To facilitate direct learning of attention maps and avoid labor-intensive annotations, we propose a general principle and a corresponding weakly-supervised objective, which promotes the HTDA model to utilize an integral and small subset of the input to give accurate predictions. On this basis, the ANN and the SNN can be jointly optimized by surrogate gradient descent in an end-to-end manner. We comprehensively evaluated the HTDA model on object recognition tasks, which demonstrates strong robustness to adversarial noise, high computing efficiency, and good interpretability. On the widely-adopted CIFAR-10, CIFAR-100, and MNIST benchmarks, the HTDA model reduces firing rates by up to 50% and improves adversarial robustness by up to 10% with comparable or better accuracy compared with the state-of-the-art SNNs. The HTDA model is also verified on dynamic neuromorphic datasets and achieves consistent improvements. This study provides a new way to boost the performance of SNNs by employing a hybrid top-down attention mechanism.

neurosciences
Inferring Shared Attention In Social Scene Videos

Lifeng Fan,Yixin Chen,Ping Wei,Wenguan Wang,Song-Chun Zhu

DOI: https://doi.org/10.1109/CVPR.2018.00676

2018-01-01

Abstract:This paper addresses a new problem of inferring shared attention in third-person social scene videos. Shared attention is a phenomenon that two or more individuals simultaneously look at a common target in social scenes. Perceiving and identifying shared attention in videos plays crucial roles in social activities and social scene understanding. We propose a spatial-temporal neural network to detect shared attention intervals in videos and predict shared attention locations in frames. In each video frame, human gaze directions and potential target boxes are two key features for spatially detecting shared attention in the social scene. In temporal domain, a convolutional Long ShortTerm Memory network utilizes the temporal continuity and transition constraints to optimize the predicted shared attention heatmap. We collect a new dataset VideoCoAtt1 from public TV show videos, containing 380 complex video sequences with more than 492,000 frames that include diverse social scenes for shared attention study. Experiments on this dataset show that our model can effectively infer shared attention in videos. We also empirically verify the effectiveness of different components in our model.
EyeTrans: Merging Human and Machine Attention for Neural Code Summarization

Yifan Zhang,Jiliang Li,Zachary Karas,Aakash Bansal,Toby Jia-Jun Li,Collin McMillan,Kevin Leach,Yu Huang

DOI: https://doi.org/10.1145/3643732

2024-02-22

Abstract:Neural code summarization leverages deep learning models to automatically generate brief natural language summaries of code snippets. The development of Transformer models has led to extensive use of attention during model design. While existing work has primarily and almost exclusively focused on static properties of source code and related structural representations like the Abstract Syntax Tree (AST), few studies have considered human attention, that is, where programmers focus while examining and comprehending code. In this paper, we develop a method for incorporating human attention into machine attention to enhance neural code summarization. To facilitate this incorporation and vindicate this hypothesis, we introduce EyeTrans, which consists of three steps: (1) we conduct an extensive eye-tracking human study to collect and pre-analyze data for model training, (2) we devise a data-centric approach to integrate human attention with machine attention in the Transformer architecture, and (3) we conduct comprehensive experiments on two code summarization tasks to demonstrate the effectiveness of incorporating human attention into Transformers. Integrating human attention leads to an improvement of up to 29.91% in Functional Summarization and up to 6.39% in General Code Summarization performance, demonstrating the substantial benefits of this combination. We further explore performance in terms of robustness and efficiency by creating challenging summarization scenarios in which EyeTrans exhibits interesting properties. We also visualize the attention map to depict the simplifying effect of machine attention in the Transformer by incorporating human attention. This work has the potential to propel AI research in software engineering by introducing more human-centered approaches and data.

Artificial Intelligence,Human-Computer Interaction,Software Engineering
An Intersubject Brain-Computer Interface Based on Domain-Adversarial Training of Convolutional Neural Network for Online Attention Decoding

Dichen,Haiyun Huang,Zijing Guan,Jiahui Pan,Yuanqing Li

DOI: https://doi.org/10.1109/TBME.2024.3404131

2024-05-23

Abstract:Attention decoding plays a vital role in daily life, where electroencephalography (EEG) has been widely involved. However, training a universally effective model for everyone is impractical due to substantial interindividual variability in EEG signals. To tackle the above challenge, we propose an end-to-end brain-computer interface (BCI) framework, including temporal and spatial one-dimensional (1D) convolutional neural network and domain-adversarial training strategy, namely DA-TSnet. Specifically, DA-TSnet extracts temporal and spatial features of EEG, while it is jointly supervised by task loss and domain loss. During training, DA-TSnet aims to maximize the domain loss while simultaneously minimizing the task loss. We conduct an offline analysis, simulate online experiments on a self-collected dataset of 85 subjects, and real online experiments on 22 subjects. Main results: DA-TSnet achieves a leave-one-subject-out (LOSO) cross-validation (CV) classification accuracy of 89.40% ± 9.96%, outperforming several state-of-the-art attention EEG decoding methods. In simulated online experiments, DA-TSnet achieves an outstanding accuracy of 88.07% ± 11.22%. In real online experiments, it achieves an average accuracy surpassing 86%. Significance: An end-to-end network framework does not rely on elaborate preprocessing and feature extraction steps, which saves time and human workload. Moreover, our framework utilizes domain-adversarial training neural network (DANN) to tackle the challenge posed by the high interindividual variability in EEG signals, which has significant reference value for handling other EEG signal decoding issues. Last, the performance of the DA-TSnet framework in offline and online experiments underscores its potential to facilitate more reliable applications.
Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension

Ekta Sood,Simon Tannert,Diego Frassinelli,Andreas Bulling,Ngoc Thang Vu

DOI: https://doi.org/10.48550/arXiv.2010.06396

2020-10-13

Computation and Language

Abstract:While neural networks with attention mechanisms have achieved superior performance on many natural language processing tasks, it remains unclear to which extent learned attention resembles human visual attention. In this paper, we propose a new method that leverages eye-tracking data to investigate the relationship between human visual attention and neural attention in machine reading comprehension. To this end, we introduce a novel 23 participant eye tracking dataset - MQA-RC, in which participants read movie plots and answered pre-defined questions. We compare state of the art networks based on long short-term memory (LSTM), convolutional neural models (CNN) and XLNet Transformer architectures. We find that higher similarity to human attention and performance significantly correlates to the LSTM and CNN models. However, we show this relationship does not hold true for the XLNet models -- despite the fact that the XLNet performs best on this challenging task. Our results suggest that different architectures seem to learn rather different neural attention strategies and similarity of neural to human attention does not guarantee best performance.
Human brain activity for machine attention

Lukas Muttenthaler,Nora Hollenstein,Maria Barrett

DOI: https://doi.org/10.48550/arXiv.2006.05113

2020-06-09

Computation and Language

Abstract:Cognitively inspired NLP leverages human-derived data to teach machines about language processing mechanisms. Recently, neural networks have been augmented with behavioral data to solve a range of NLP tasks spanning syntax and semantics. We are the first to exploit neuroscientific data, namely electroencephalography (EEG), to inform a neural attention model about language processing of the human brain. The challenge in working with EEG data is that features are exceptionally rich and need extensive pre-processing to isolate signals specific to text processing. We devise a method for finding such EEG features to supervise machine attention through combining theoretically motivated cropping with random forest tree splits. After this dimensionality reduction, the pre-processed EEG features are capable of distinguishing two reading tasks retrieved from a publicly available EEG corpus. We apply these features to regularise attention on relation classification and show that EEG is more informative than strong baselines. This improvement depends on both the cognitive load of the task and the EEG frequency domain. Hence, informing neural attention models with EEG signals is beneficial but requires further investigation to understand which dimensions are the most useful across NLP tasks.

Learning to Infer Human Attention in Daily Activities.

Deep Neural Networks Evolve Human-like Attention Distribution during Reading Comprehension.

Deep Neural Networks Evolve Human-like Attention Distribution during Goal-directed Reading Comprehension

Task Optimization Leads to Human-like Top-down and Bottom-up Attention during Reading Comprehension

Understanding More about Human and Machine Attention in Deep Neural Networks

Human Vs Machine Attention in Neural Networks: A Comparative Study.

Understanding attention-based encoder-decoder networks: a case study with chess scoresheet recognition

An Empirical Study of Spatial Attention Mechanisms in Deep Networks

Learning When to Attend for Neural Machine Translation

Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

Attention module improves both performance and interpretability of 4D fMRI decoding neural network

Predicting Task-Driven Attention Via Integrating Bottom-Up Stimulus and Top-Down Guidance

Attention module improves both performance and interpretability of four‐dimensional functional magnetic resonance imaging decoding neural network

Inferring Human Attention by Learning Latent Intentions.

Attending Via Both Fine-tuning and Compressing.

Enhancing spiking neural networks with hybrid top-down attention

Inferring Shared Attention In Social Scene Videos

EyeTrans: Merging Human and Machine Attention for Neural Code Summarization

An Intersubject Brain-Computer Interface Based on Domain-Adversarial Training of Convolutional Neural Network for Online Attention Decoding

Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension

Human brain activity for machine attention