Abstract:The transfer of a neural network (CNN) trained to recognize objects to the task of scene classification is considered. A Bag-of-Semantics (BoS) representation is first induced, by feeding scene image patches to the object CNN, and representing the scene image by the ensuing bag of posterior class probability vectors (semantic posteriors). The encoding of the BoS with a Fisher vector (FV) is then studied. A link is established between the FV of any probabilistic model and the Q-function of the expectation-maximization (EM) algorithm used to estimate its parameters by maximum likelihood. This enables 1) immediate derivation of FVs for any model for which an EM algorithm exists, and 2) leveraging efficient implementations from the EM literature for the computation of FVs. It is then shown that standard FVs, such as those derived from Gaussian or even Dirichlet mixtures, are unsuccessful for the transfer of semantic posteriors, due to the highly non-linear nature of the probability simplex. The analysis of these FVs shows that significant benefits can ensue by 1) designing FVs in the natural parameter space of the multinomial distribution, and 2) adopting sophisticated probabilistic models of semantic feature covariance. The combination of these two insights leads to the encoding of the BoS in the natural parameter space of the multinomial, using a vector of Fisher scores derived from a mixture of factor analyzers (MFA). A network implementation of the MFA Fisher Score (MFA-FS), denoted as the MFAFSNet, is finally proposed to enable end-to-end training. Experiments with various object CNNs and datasets show that the approach has state-of-the-art transfer performance. Somewhat surprisingly, the scene classification results are superior to those of a CNN explicitly trained for scene classification, using a large scene dataset (Places). This suggests that holistic analysis is insufficient for scene classification. The modeling of local object semantics appears to be at least equally important. The two approaches are also shown to be strongly complementary, leading to very large scene classification gains when combined, and outperforming all previous scene classification approaches by a sizable margin.

Semantic Clustering for Robust Fine-Grained Scene Recognition

Domain Adaptive Remote Sensing Scene Recognition via Semantic Relationship Knowledge Transfer

Semantic-aware scene recognition

A Study of RobustNet, a Domain Generalization Method for Semantic Segmentation

Clustering-based Image-Text Graph Matching for Domain Generalization

Generalizable model-agnostic semantic segmentation via target-specific normalization

Semantic-guided modeling of spatial relation and object co-occurrence for indoor scene recognition

Exploiting Object-based and Segmentation-based Semantic Features for Deep Learning-based Indoor Scene Classification

Fine-Grained Recognition With Learnable Semantic Data Augmentation

Semantic Understanding of Scenes Through the ADE20K Dataset

Domain Adaptive Semantic Segmentation by Optimal Transport

Domain-Invariant Information Aggregation for Domain Generalization Semantic Segmentation

Semantic-embedded Similarity Prototype for Scene Recognition

Semantic Fisher Scores for Task Transfer: Using Objects to Classify Scenes

Inter-Class and Inter-Domain Semantic Augmentation for Domain Generalization

Semantic Relation Model and Dataset for Remote Sensing Scene Understanding

Soft Semantic Representation for Cross-Domain Face Recognition

Visual representations with texts domain generalization for semantic segmentation

Scaling up Multi-domain Semantic Segmentation with Sentence Embeddings

Constrained Parametric Proposals and Pooling Methods for Semantic Segmentation in RGB-D Images

Domain Generalization with Domain-Specific Aggregation Modules