Abstract:In this paper, we are concerned with the task of gaze following. Given a scene (e.g. a girl playing soccer on the field) and a human subject’s head position, this task aims to infer where she is looking (e.g. at the soccer ball). An existing method adopts a saliency model conditioned on the head position. However, this methodology is intrinsically troubled with dataset bias issues, which we will reveal in detail. In order to resolve these issues, we argue that the right methodology is to simulate how human beings follow gazes. Specifically, we propose the hypothesis that a human follows gazes by searching for salient objects along the subject’s sight line direction. To algorithmically embody this hypothesis, a two-stage method is proposed, which is dubbed as learning to draw sight lines. In the first stage, a fully convolutional network is trained to directly regress the existence strength of sight lines. It may seem counterintuitive at a first glance as these so-called sight lines do not really exist in the form of learnable image gradients. However, with the large-scale dataset GazeFollow, we demonstrate that this highly abstract concept can be grounded into neural network activations. An extensive study is conducted on the design of this sight line grounding network. We show that the best model we visited can already outperform the state-of-the-arts by a large margin, using a naive greedy inference strategy. We attribute these improvements to modern architecture design philosophies. However, no matter how strong the sight line grounding network is, the greedy inference strategy cannot handle a bunch of failure cases caused by dataset bias issues. We identify these issues and demonstrate that those grounded sight lines, which is a unique ingredient of our method, is the key to overcome them. Specifically, an algorithm termed as RASP is introduced as a second stage. RASP has five intriguing features: (1) it explicitly embodies the aforementioned hypothesis; (2) it involves no hyper-parameters, thus guaranteeing its robustness; (3) if needed, it can be implemented as an integrated layer for end-to-end inference; (4) it improves the performances of all sight line grounding networks we inspected; (5) further analyses confirm that RASP works by alleviating those spotted dataset biases. Strong results are achieved on the GazeFollow benchmark. Combining RASP and the best sight line grounding network can bring mean distance, minimum distance and mean angle difference 45.85%, 42.60%, and 49.23% closer towards human performance when compared with state-of-the-arts. We also contribute a video gaze following benchmark called GazeShift, on which we further demonstrate the importance of RASP in video applications. Codes and models will be released, encouraging further research on the important task of gaze following. Along with our implementation, we contribute a well-engineered toolbox for joint subject tracking and gaze following.

Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

Controllable Continuous Gaze Redirection

PerimetryNet: A Multiscale Fine Grained Deep Network for Three-Dimensional Eye Gaze Estimation Using Visual Field Analysis

GazeGaussian: High-Fidelity Gaze Redirection with 3D Gaussian Splatting

GazeNeRF: 3D-Aware Gaze Redirection with Neural Radiance Fields

NeRF-Gaze: A Head-Eye Redirection Parametric Model for Gaze Estimation

GazeDirector: Fully Articulated Eye Gaze Redirection in Video

Angle Range and Identity Similarity Enhanced Gaze and Head Redirection based on Synthetic data

GazeCorrection:Self-Guided Eye Manipulation in the Wild Using Self-Supervised Generative Adversarial Networks

Towards Pixel-Level Prediction for Gaze Following: Benchmark and Approach

Multi-Person Gaze-Following with Numerical Coordinate Regression

Depth-aware gaze-following via auxiliary networks for robotics

Dual In-painting Model for Unsupervised Gaze Correction and Animation in the Wild

Purifying real images with an attention-guided style transfer network for gaze estimation

Unsupervised High-Resolution Portrait Gaze Correction and Animation

LG-Gaze: Learning Geometry-aware Continuous Prompts for Language-Guided Gaze Estimation

PureGaze: Purifying Gaze Feature for Generalizable Gaze Estimation

Different gaze direction (DGNet) collaborative learning for iris segmentation

Learning to Draw Sight Lines.

DiffGaze: A Diffusion Model for Continuous Gaze Sequence Generation on 360° Images