Abstract:Zero-shot learning (ZSL) aims to recognize objects without seeing any visual instances by learning knowledge transfer between seen and unseen classes. Attributes, which denote high-level visual entities or visual characteristics, have been widely utilized as intermediate embedding space for knowledge transfer in a majority of existing ZSL approaches and have shown impressive performance. Attribute based ZSL approaches, which introduce an intermediate embedding space of attributes for knowledge transfer, have shown impressive performance. However, providing attribute annotations for unseen classes at test time is time-consuming and labor-intensive. Besides, directly using attributes as intermediation (embedding space) for knowledge transfer and zero-shot prediction inevitably leads to the projection domain shift and hubness problems. In this paper, we propose a novel multi-view deep neural network, termed Fusion by Synthesis (FS), which leverages word embeddings of classes as complementary for attributes and performs zero-shot prediction by fusing the word embeddings of unseen classes and the synthesized attributes in the visual feature space. Specifically, in the training phase, by considering the visual features, attributes and word embeddings as three different views of visual instances, FS allocates each view with a denoising auto-encoder to simultaneously ensures robust view-specific reconstructions and cross-view synthesizing, while preserving the discrimination of class labels. During testing, FS can synthesize the absent attributes for unseen classes and fuse them with word embeddings to the visual feature space to perform zero-shot prediction. Besides, FS is flexible to learn with partial views, where either attribute view or word embedding view is missing during training. Moreover, FS is flexible to synthesize either the missing view of attributes or word embeddings from the provided view. Extensive experiments on six benchmark datasets on both image classification and action recognition show that FS is advantaged to fuse multi-view data by synthesis and achieves superior performance compared with the state-of-the-art ZSL methods. (C) 2019 Elsevier B.V. All rights reserved.

Motion-Decoupled Spiking Transformer for Audio-Visual Zero-Shot Learning

Modality-Fusion Spiking Transformer Network for Audio-Visual Zero-Shot Learning

Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning

Transformer-Based Approach Via Contrastive Learning for Zero-Shot Detection.

A Novel Spike Transformer Network for Depth Estimation from Event Cameras via Cross-modality Knowledge Distillation

Deep CovDenseSNN: A Hierarchical Event-Driven Dynamic Framework with Spiking Neurons in Noisy Environment

Transformer-based Spiking Neural Networks for Multimodal Audio-Visual Classification

Hybrid Routing Transformer for Zero-Shot Learning

Temporal–Semantic Aligning and Reasoning Transformer for Audio-Visual Zero-Shot Learning

Spike Transformer: Monocular Depth Estimation for Spiking Camera

Transductive Zero-Shot Learning With Adaptive Structural Embedding

MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object Segmentation.

ZeroMamba: Exploring Visual State Space Model for Zero-Shot Learning

Learning Stereo Depth Estimation with Bio-Inspired Spike Cameras

TransZero: Attribute-guided Transformer for Zero-Shot Learning

CHANNEL-WISE MIX-FUSION DEEP NEURAL NETWORKS FOR ZERO-SHOT LEARNING

Deep Semantic-Visual Alignment for Zero-Shot Remote Sensing Image Scene Classification

Fusion by Synthesizing: A Multi-View Deep Neural Network for Zero-Shot Recognition

Discrepancy-Aware Attention Network for Enhanced Audio-Visual Zero-Shot Learning

Object-Aware Image Augmentation for Audio-Visual Zero-Shot Learning

Zero-Shot Learning Using Synthesised Unseen Visual Data with Diffusion Regularisation