Abstract:Human eye gaze plays a significant role in many virtual and augmented reality (VR/AR) applications, such as gaze-contingent rendering, gaze-based interaction, or eye-based activity recognition. However, prior works on gaze analysis and prediction have only explored eye-head coordination and were limited to human-object interactions. We first report a comprehensive analysis of eye-body coordination in various human-object and human-human interaction activities based on four public datasets collected in real-world (MoGaze), VR (ADT), as well as AR (GIMO and EgoBody) environments. We show that in human-object interactions, e.g. pick and place, eye gaze exhibits strong correlations with full-body motion while in human-human interactions, e.g. chat and teach, a person's gaze direction is correlated with the body orientation towards the interaction partner. Informed by these analyses we then present Pose2Gaze, a novel eye-body coordination model that uses a convolutional neural network and a spatio-temporal graph convolutional neural network to extract features from head direction and full-body poses, respectively, and then uses a convolutional neural network to predict eye gaze. We compare our method with state-of-the-art methods that predict eye gaze only from head movements and show that Pose2Gaze outperforms these baselines with an average improvement of 24.0% on MoGaze, 10.1% on ADT, 21.3% on GIMO, and 28.6% on EgoBody in mean angular error, respectively. We also show that our method significantly outperforms prior methods in the sample downstream task of eye-based activity recognition. These results underline the significant information content available in eye-body coordination during daily activities and open up a new direction for gaze prediction.

Gaze Estimation in Children's Peer-Play Scenarios

Eye Gaze Perception and Gaze Cueing Effect

ChildPlay: A New Benchmark for Understanding Children's Gaze Behaviour

PerimetryNet: A Multiscale Fine Grained Deep Network for Three-Dimensional Eye Gaze Estimation Using Visual Field Analysis

Specialized Gaze Estimation for Children by Convolutional Neural Network and Domain Adaptation.

Video Based Children'S Social Behavior Classification in Peer-Play Scenarios

Vision-based Gaze Estimation: A Review

Detecting Gaze Towards Eyes in Natural Social Interactions and its Use in Child Assessment

Computer vision analysis for children's social play classification in peer-play scenarios.

Joint Estimation of Head Pose and Visual Focus of Attention

A Practical, Robust, Accurate Gaze-Based Intention Inference Method for Everyday Human-Robot Interaction

An Attention Detection System Based on Gaze Estimation Using Self-supervised Learning

Gaze Target Estimation inspired by Interactive Attention

Enhanced Gaze Following via Object Detection and Human Pose Estimation

Improving Children'S Gaze Prediction Via Separate Facial Areas And Attention Shift Cue

Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses

Evaluating User Experience and Data Quality in Gamified Data Collection for Appearance-Based Gaze Estimation

Multi-Person Gaze-Following with Numerical Coordinate Regression

Gaze Estimation Based on Deep Learning Method

DGaze: CNN-Based Gaze Prediction in Dynamic Scenes.

Gaze Estimation Method Combining Facial Feature Extractor with Pyramid Squeeze Attention Mechanism