Abstract:With prior knowledge of seen objects, humans have a remarkable ability to recognize novel objects using shared and distinct local attributes. This is significant for the challenging tasks of zero-shot learning (ZSL) and fine-grained visual classification (FGVC), where the discriminative attributes of objects have played an important role. Inspired by human visual attention, neural networks have widely exploited the attention mechanism to learn the locally discriminative attributes for challenging tasks. Though greatly promoted the development of these fields, existing works mainly focus on learning the region embeddings of different attribute features and neglect the importance of discriminative attribute localization. It is also unclear whether the learned attention truly matches the real human attention. To tackle this problem, this paper proposes to employ real human gaze data for visual recognition networks to learn from human attention. Specifically, we design a unified Attribute Attention Network (A 2 Net) that learns from human attention for both ZSL and FGVC tasks. The overall model consists of an attribute attention branch and a baseline classification network. On top of the image feature maps provided by the baseline classification network, the attribute attention branch employs attribute prototypes to produce attribute attention maps and attribute features. The attribute attention maps are converted to gaze-like attentions to be aligned with real human gaze attention. To guarantee the effectiveness of attribute feature learning, we further align the extracted attribute features with attribute-defined class embeddings. To facilitate learning from human gaze attention for the visual recognition problems, we design a bird classification game to collect real human gaze data using the CUB dataset via an eye-tracker device. Experiments on ZSL and FGVC tasks without/with real human gaze data validate the benefits and accuracy of our proposed model. This work supports the promising benefits of collecting human gaze datasets and automatic gaze estimation algorithms learning from human attention for high-level computer vision tasks.

Visual Attention Consistency for Human Attribute Recognition

Learning From Human Attention for Attribute-Assisted Visual Recognition

Improving Person Re-Identification Via Attribute-Identity Representation and Visual Attention Mechanism

Spatial and Semantic Consistency Regularizations for Pedestrian Attribute Recognition

Deep Attention Aware Feature Learning for Person Re-Identification

Image Visual Attention Computation and Application Via the Learning of Object Attributes

ATCON: Attention Consistency for Vision Models

VAC-Net: Visual Attention Consistency Network for Person Re-identification

Learning Scale-Consistent Attention Part Network for Fine-grained Image Recognition

Understanding More about Human and Machine Attention in Deep Neural Networks

Attentional Alignment Networks.

Harmonious Attention Network for Person Re-Identification

Fine-grained Image Recognition Via Attention Interaction and Counterfactual Attention Network

An Orientation-Aware Attention Network for Person Re-Identification

Human Vs Machine Attention in Neural Networks: A Comparative Study.

FAN: Focused Attention Networks

Attend and Guide (AG-Net): A Keypoints-driven Attention-based Deep Network for Image Recognition

Learning Predictable And Discriminative Attributes For Visual Recognition

Where is the Model Looking At?--Concentrate and Explain the Network Attention

BI-AVAN: A Brain-Inspired Adversarial Visual Attention Network for Characterizing Human Visual Attention from Neural Activity

Human Interaction Understanding With Consistency-Aware Learning