Abstract:With prior knowledge of seen objects, humans have a remarkable ability to recognize novel objects using shared and distinct local attributes. This is significant for the challenging tasks of zero-shot learning (ZSL) and fine-grained visual classification (FGVC), where the discriminative attributes of objects have played an important role. Inspired by human visual attention, neural networks have widely exploited the attention mechanism to learn the locally discriminative attributes for challenging tasks. Though greatly promoted the development of these fields, existing works mainly focus on learning the region embeddings of different attribute features and neglect the importance of discriminative attribute localization. It is also unclear whether the learned attention truly matches the real human attention. To tackle this problem, this paper proposes to employ real human gaze data for visual recognition networks to learn from human attention. Specifically, we design a unified Attribute Attention Network (A 2 Net) that learns from human attention for both ZSL and FGVC tasks. The overall model consists of an attribute attention branch and a baseline classification network. On top of the image feature maps provided by the baseline classification network, the attribute attention branch employs attribute prototypes to produce attribute attention maps and attribute features. The attribute attention maps are converted to gaze-like attentions to be aligned with real human gaze attention. To guarantee the effectiveness of attribute feature learning, we further align the extracted attribute features with attribute-defined class embeddings. To facilitate learning from human gaze attention for the visual recognition problems, we design a bird classification game to collect real human gaze data using the CUB dataset via an eye-tracker device. Experiments on ZSL and FGVC tasks without/with real human gaze data validate the benefits and accuracy of our proposed model. This work supports the promising benefits of collecting human gaze datasets and automatic gaze estimation algorithms learning from human attention for high-level computer vision tasks.

Learning high-level concepts by training a deep network on eye fixations

Learning to attend in a brain-inspired deep neural network

Inferring Salient Objects from Human Fixations

Learning what and where to attend

Deep Visual Attention Prediction

Deep supervised visual saliency model addressing low-level features

Deep saliency models learn low-, mid-, and high-level features to predict scene attention

Learning to Model Task-Oriented Attention

What Do Deep Saliency Models Learn about Visual Attention?

Low-level and High-Level Prior Learning for Visual Saliency Estimation.

Object Detection Based on Saturation of Visual Perception

Learning From Human Attention for Attribute-Assisted Visual Recognition

Two-Stage Learning to Predict Human Eye Fixations Via SDAEs

Salient Object Detection Driven By Fixation Prediction

Image Visual Attention Computation and Application Via the Learning of Object Attributes

Visual Attention driven by Convolutional Features

Weakly Supervised Visual Saliency Prediction

A Deep Model of Visual Attention for Saliency Detection on 3D Objects

Predicting eye fixations with higher-level visual features.

Contour-guided saliency detection with long-range interactions

Attend Before you Act: Leveraging human visual attention for continual learning