Abstract:A visual scanpath represents the human eye movements when scanning the visual field for acquiring and receiving visual information. Predicting visual scanpaths when a certain stimulus is presented plays an important role in modeling overt human visual attention and search behavior. In this paper, we presented an 'Inhibition of Return - Region of Interest' (IOR-ROI) recurrent mixture density network based framework learning to produce human-like visual scanpaths under task-free viewing conditions. The proposed model simultaneously predicts a sequence of ordered fixation positions and their corresponding fixation durations. Our model integrates bottom-up features and semantic features extracted by convolutional neural networks. Then the integrated feature maps are fed into the IOR-ROI Long Short-Term Memory (LSTM) which is the core component of the proposed model. The IOR-ROI LSTM is a dual LSTM unit, i.e., the IOR-LSTM and the ROI-LSTM, capturing IOR dynamics and gaze shift behavior simultaneously. IOR-LSTM simulates the visual working memory to adaptively maintain and update visual information regarding previously fixated regions. ROI-LSTM is responsible for predicting the next possible ROIs given the spatially inhibited image feature maps on the feature-wise basis. Fixation duration is predicted by a regression neural network given the viewing history and image feature maps corresponding to currently fixated ROI. Considering the eye movement pattern variations among subjects, a mixture density network is adopted to model the next fixation distribution as Gaussian mixtures and the fixation duration is also modeled using Gaussian distribution. Our model is evaluated on the OSIE and MIT low resolution eye-tracking datasets and experimental results indicate that the proposed method can achieve superior performance in predicting visual scanpaths. The code will be publicly available on URL: https://github.com/sunwj/scanpath.

Simulating human saccadic scanpaths on natural images

Modeling of Human Saccadic Scanpaths Based on Visual Saliency.

Simulate Human Saccadic Scan-Paths in Target Searching

Predicting Human Saccadic Scanpaths Based on Iterative Representation Learning

Human Scanpath Prediction Based on Deep Convolutional Saccadic Model

Saliency Map Generation Based On Saccade Target Theory

Toward Statistical Modeling of Saccadic Eye-Movement and Visual Saliency.

What Are We Looking For: Towards Statistical Modeling of Saccadic Eye Movements and Visual Saliency

Scanpath Estimation Based on Foveated Image Saliency

Human Scanpath Estimation Based on Semantic Segmentation Guided by Common Eye Fixation Behaviors

Imitating the Human Visual System for Scanpath Predicting.

Saccade Sequence Prediction: Beyond Static Saliency Maps

AN ITERATIVE REPRESENTATION LEARNING FRAMEWORK TO PREDICT THE SEQUENCE OF EYE FIXATIONS

Improving saliency models' predictions of the next fixation with humans' intrinsic cost of gaze shifts

Spatiotemporal evidence accumulation through saccadic sampling for object recognition

Simulation of Mono-eye Blind Spot Effect Based on Visual System

Visual Scanpath Prediction Using IOR-ROI Recurrent Mixture Density Network.

Evaluation of Saccadic Scanpath Prediction: Subjective Assessment Database and Recurrent Neural Network Based Metric

Saccade Mechanisms for Image Classification, Object Detection and Tracking

A Robotics-Inspired Scanpath Model Reveals the Importance of Uncertainty and Semantic Object Cues for Gaze Guidance in Dynamic Scenes

Energy-Efficient Visual Search by Eye Movement and Low-Latency Spiking Neural Network