Abstract:With prior knowledge of seen objects, humans have a remarkable ability to recognize novel objects using shared and distinct local attributes. This is significant for the challenging tasks of zero-shot learning (ZSL) and fine-grained visual classification (FGVC), where the discriminative attributes of objects have played an important role. Inspired by human visual attention, neural networks have widely exploited the attention mechanism to learn the locally discriminative attributes for challenging tasks. Though greatly promoted the development of these fields, existing works mainly focus on learning the region embeddings of different attribute features and neglect the importance of discriminative attribute localization. It is also unclear whether the learned attention truly matches the real human attention. To tackle this problem, this paper proposes to employ real human gaze data for visual recognition networks to learn from human attention. Specifically, we design a unified Attribute Attention Network (A 2 Net) that learns from human attention for both ZSL and FGVC tasks. The overall model consists of an attribute attention branch and a baseline classification network. On top of the image feature maps provided by the baseline classification network, the attribute attention branch employs attribute prototypes to produce attribute attention maps and attribute features. The attribute attention maps are converted to gaze-like attentions to be aligned with real human gaze attention. To guarantee the effectiveness of attribute feature learning, we further align the extracted attribute features with attribute-defined class embeddings. To facilitate learning from human gaze attention for the visual recognition problems, we design a bird classification game to collect real human gaze data using the CUB dataset via an eye-tracker device. Experiments on ZSL and FGVC tasks without/with real human gaze data validate the benefits and accuracy of our proposed model. This work supports the promising benefits of collecting human gaze datasets and automatic gaze estimation algorithms learning from human attention for high-level computer vision tasks.

AGIL: Learning Attention from Human for Visuomotor Tasks

GVGNet: Gaze-Directed Visual Grounding for Learning Under-Specified Object Referring Intention

Attention Guided Imitation Learning and Reinforcement Learning

Efficiently Guiding Imitation Learning Agents with Human Gaze

Imitation Learning with Human Eye Gaze via Multi-Objective Prediction

Gaze Target Estimation inspired by Interactive Attention

Gaze-based Human Intention Prediction in the Hybrid Foraging Search Task

Gaze-Guided Graph Neural Network for Action Anticipation Conditioned on Intention

Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning

A Gaze Model Improves Autonomous Driving

Learning From Human Attention for Attribute-Assisted Visual Recognition

Exploring Coarse-grained Pre-guided Attention to Assist Fine-grained Attention Reinforcement Learning Agents

Eyes on the Game: Deciphering Implicit Human Signals to Infer Human Proficiency, Trust, and Intent

MIDAS: Deep learning human action intention prediction from natural eye movement patterns

Attend Before you Act: Leveraging human visual attention for continual learning

Learning Actions and Control of Focus of Attention with a Log-Polar-like Sensor

Human Visual Attention Prediction Boosts Learning & Performance of Autonomous Driving Agents

Gaze estimation learning architecture as support to affective, social and cognitive studies in natural human-robot interaction

Understanding Teacher Gaze Patterns for Robot Learning

Imitator Learning: Achieve Out-of-the-Box Imitation Ability in Variable Environments

Voila-A: Aligning Vision-Language Models with User's Gaze Attention