Abstract:With prior knowledge of seen objects, humans have a remarkable ability to recognize novel objects using shared and distinct local attributes. This is significant for the challenging tasks of zero-shot learning (ZSL) and fine-grained visual classification (FGVC), where the discriminative attributes of objects have played an important role. Inspired by human visual attention, neural networks have widely exploited the attention mechanism to learn the locally discriminative attributes for challenging tasks. Though greatly promoted the development of these fields, existing works mainly focus on learning the region embeddings of different attribute features and neglect the importance of discriminative attribute localization. It is also unclear whether the learned attention truly matches the real human attention. To tackle this problem, this paper proposes to employ real human gaze data for visual recognition networks to learn from human attention. Specifically, we design a unified Attribute Attention Network (A 2 Net) that learns from human attention for both ZSL and FGVC tasks. The overall model consists of an attribute attention branch and a baseline classification network. On top of the image feature maps provided by the baseline classification network, the attribute attention branch employs attribute prototypes to produce attribute attention maps and attribute features. The attribute attention maps are converted to gaze-like attentions to be aligned with real human gaze attention. To guarantee the effectiveness of attribute feature learning, we further align the extracted attribute features with attribute-defined class embeddings. To facilitate learning from human gaze attention for the visual recognition problems, we design a bird classification game to collect real human gaze data using the CUB dataset via an eye-tracker device. Experiments on ZSL and FGVC tasks without/with real human gaze data validate the benefits and accuracy of our proposed model. This work supports the promising benefits of collecting human gaze datasets and automatic gaze estimation algorithms learning from human attention for high-level computer vision tasks.

Deriving Machine Attention from Human Rationales

Attention in Reasoning: Dataset, Analysis, and Modeling

Exploring Distantly-Labeled Rationales in Neural Network Models

What to Learn, and How: Toward Effective Learning from Rationales

AiR: Attention with Reasoning Capability

Understanding More about Human and Machine Attention in Deep Neural Networks

Data-Centric Human Preference Optimization with Rationales

Beyond Accuracy: Ensuring Correct Predictions With Correct Rationales

An Analysis of Attention via the Lens of Exchangeability and Latent Variable Models

Leveraging Machine-Generated Rationales to Facilitate Social Meaning Detection in Conversations

Making a (Counterfactual) Difference One Rationale at a Time

Human Vs Machine Attention in Neural Networks: A Comparative Study.

Rethinking the role of attention mechanism: a causality perspective

Attention: Marginal Probability is All You Need?

Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension

An Introductory Survey on Attention Mechanisms in NLP Problems

Inferring Human Attention by Learning Latent Intentions.

Enhancing the Rationale-Input Alignment for Self-explaining Rationalization

Faithful Attention Explainer: Verbalizing Decisions Based on Discriminative Features

Attention Is (not) All You Need for Commonsense Reasoning

Learning From Human Attention for Attribute-Assisted Visual Recognition