Abstract:Eye movements have long been studied as a window into the attentional mechanisms of the human brain and made accessible as novelty style human-machine interfaces. However, not everything that we gaze upon, is something we want to interact with; this is known as the Midas Touch problem for gaze interfaces. To overcome the Midas Touch problem, present interfaces tend not to rely on natural gaze cues, but rather use dwell time or gaze gestures. Here we present an entirely data-driven approach to decode human intention for object manipulation tasks based solely on natural gaze cues. We run data collection experiments where 16 participants are given manipulation and inspection tasks to be performed on various objects on a table in front of them. The subjects' eye movements are recorded using wearable eye-trackers allowing the participants to freely move their head and gaze upon the scene. We use our Semantic Fovea, a convolutional neural network model to obtain the objects in the scene and their relation to gaze traces at every frame. We then evaluate the data and examine several ways to model the classification task for intention prediction. Our evaluation shows that intention prediction is not a naive result of the data, but rather relies on non-linear temporal processing of gaze cues. We model the task as a time series classification problem and design a bidirectional Long-Short-Term-Memory (LSTM) network architecture to decode intentions. Our results show that we can decode human intention of motion purely from natural gaze cues and object relative position, with $91.9\%$ accuracy. Our work demonstrates the feasibility of natural gaze as a Zero-UI interface for human-machine interaction, i.e., users will only need to act naturally, and do not need to interact with the interface itself or deviate from their natural eye movement patterns.

Application Layer Operation Intention Recognition Based on Mouse Behavior

Better Than Humans: a Method for Inferring Consumer Shopping Intentions by Reading Facial Expressions

Research on Human-Computer Interaction Intention Recognition Based on EEG and Eye Movement

3D Intuitive Gesture Interaction via Motion Sensing

Emotion-Age-Gender-Nationality Based Intention Understanding in Human–Robot Interaction Using Two-Layer Fuzzy Support Vector Regression

Interact with Robot: an Efficient Approach Based on Finite State Machine and Mouse Gesture Recognition

Context-Aware Mouse Behavior Recognition Using Hidden Markov Models

A novel mice interaction recognition method by using machine learning

MouseIdentity: Modeling Mouse-Interaction Behavior for a User Verification System

MIDAS: Deep learning human action intention prediction from natural eye movement patterns

Feature analysis of mouse dynamics in identity authentication and monitoring

An Insider Threat Detection Approach Based on Mouse Dynamics and Deep Learning

User Authentication Through Mouse Dynamics

Intent Recognition in Smart Living Through Deep Recurrent Neural Networks

Pattern-Growth Based Mining Mouse-Interaction Behavior for an Active User Authentication System.

Enabling Pointing Assistance in Adaptive Interfaces Using Mouse Pointing Intention Prediction

User Authentication and Monitoring Based on Mouse Behavioral Features

Multimodal human attention detection for reading from facial expression, eye gaze, and mouse dynamics

A Deep Learning Approach to Web Bot Detection Using Mouse Behavioral Biometrics

Murine Motion Behavior Recognition Based on DeepLabCut and Convolutional Long Short-Term Memory Network

Evaluating Focused Attention Level Using Mouse Operation Data