Abstract:Human eye gaze plays a significant role in many virtual and augmented reality (VR/AR) applications, such as gaze-contingent rendering, gaze-based interaction, or eye-based activity recognition. However, prior works on gaze analysis and prediction have only explored eye-head coordination and were limited to human-object interactions. We first report a comprehensive analysis of eye-body coordination in various human-object and human-human interaction activities based on four public datasets collected in real-world (MoGaze), VR (ADT), as well as AR (GIMO and EgoBody) environments. We show that in human-object interactions, e.g. pick and place, eye gaze exhibits strong correlations with full-body motion while in human-human interactions, e.g. chat and teach, a person's gaze direction is correlated with the body orientation towards the interaction partner. Informed by these analyses we then present Pose2Gaze, a novel eye-body coordination model that uses a convolutional neural network and a spatio-temporal graph convolutional neural network to extract features from head direction and full-body poses, respectively, and then uses a convolutional neural network to predict eye gaze. We compare our method with state-of-the-art methods that predict eye gaze only from head movements and show that Pose2Gaze outperforms these baselines with an average improvement of 24.0% on MoGaze, 10.1% on ADT, 21.3% on GIMO, and 28.6% on EgoBody in mean angular error, respectively. We also show that our method significantly outperforms prior methods in the sample downstream task of eye-based activity recognition. These results underline the significant information content available in eye-body coordination during daily activities and open up a new direction for gaze prediction.

Speech Driven Gaze in a Face-to-Face Interaction

Charting the Silent Signals of Social Gaze: Automating Eye Contact Assessment in Face-to-Face Conversations

Are you really looking at me? A Feature-Extraction Framework for Estimating Interpersonal Eye Gaze from Conventional Video

Impact of Simulated Gaze Gestures on Social Interaction for People with Visual Impairments

Social Glasses: Simulating Interactive Gaze for Visually Impaired People in Face-to-Face Communication

Vision-based Gaze Estimation: A Review

Gaze in a real-world social interaction: A dual eye-tracking study

I Can See it in Your Eyes: Gaze as an Implicit Cue of Uncanniness and Task Performance in Repeated Interactions

Gaze-action coupling, gaze-gesture coupling, and exogenous attraction of gaze in dyadic interactions

Eyes on Emotion: Dynamic Gaze Allocation During Emotion Perception From Speech-Like Stimuli.

Measuring and modeling the perception of natural and unconstrained gaze in humans and machines

Pose2Gaze: Eye-body Coordination during Daily Activities for Gaze Prediction from Full-body Poses

Gaze modeling in multi-party dialogues and extraversion expression through gaze aversion control

It's Written All Over Your Face: Full-Face Appearance-Based Gaze Estimation

Multi-Modal Gaze Following in Conversational Scenarios

The semantics of gaze in person perception: a novel qualitative-quantitative approach

Quantifying Gaze Behavior During Real-World Interactions Using Automated Object, Face, and Fixation Detection

DGaze: CNN-Based Gaze Prediction in Dynamic Scenes.

Judging by the Look: The Impact of Robot Gaze Strategies on Human Cooperation

Multimodal Across Domains Gaze Target Detection

Attention-based Mechanism for Head-Body Fusion Gaze Estimation in Dynamic Scenes