Abstract:The transfer of a neural network (CNN) trained to recognize objects to the task of scene classification is considered. A Bag-of-Semantics (BoS) representation is first induced, by feeding scene image patches to the object CNN, and representing the scene image by the ensuing bag of posterior class probability vectors (semantic posteriors). The encoding of the BoS with a Fisher vector (FV) is then studied. A link is established between the FV of any probabilistic model and the Q-function of the expectation-maximization (EM) algorithm used to estimate its parameters by maximum likelihood. This enables 1) immediate derivation of FVs for any model for which an EM algorithm exists, and 2) leveraging efficient implementations from the EM literature for the computation of FVs. It is then shown that standard FVs, such as those derived from Gaussian or even Dirichlet mixtures, are unsuccessful for the transfer of semantic posteriors, due to the highly non-linear nature of the probability simplex. The analysis of these FVs shows that significant benefits can ensue by 1) designing FVs in the natural parameter space of the multinomial distribution, and 2) adopting sophisticated probabilistic models of semantic feature covariance. The combination of these two insights leads to the encoding of the BoS in the natural parameter space of the multinomial, using a vector of Fisher scores derived from a mixture of factor analyzers (MFA). A network implementation of the MFA Fisher Score (MFA-FS), denoted as the MFAFSNet, is finally proposed to enable end-to-end training. Experiments with various object CNNs and datasets show that the approach has state-of-the-art transfer performance. Somewhat surprisingly, the scene classification results are superior to those of a CNN explicitly trained for scene classification, using a large scene dataset (Places). This suggests that holistic analysis is insufficient for scene classification. The modeling of local object semantics appears to be at least equally important. The two approaches are also shown to be strongly complementary, leading to very large scene classification gains when combined, and outperforming all previous scene classification approaches by a sizable margin.

Deep FisherNet for Object Classification

Deep FisherNet for Image Classification

Fisher Vector Based CNN Architecture for Image Classification.

Deep Marginal Fisher Analysis Based CNN for Image Representation and Classification.

Image Classification with CNN-based Fisher Vector Coding

Deep-based Fisher Vector for Mobile Visual Search

Foreground Fisher Vector: Encoding Class-Relevant Foreground to Improve Image Classification

Towards compact visual descriptor via deep fisher network with binary embedding

ConvFishNet: an Efficient Backbone for Fish Classification from Composited Underwater Images

FPCANet: Fisher Discrimination for Principal Component Analysis Network

Hybrid RGB-D object recognition using Convolutional Neural Network and Fisher Vector

Enhancing Small Object Encoding in Deep Neural Networks: Introducing Fast&Focused-Net with Volume-wise Dot Product Layer

Rifd-Cnn: Rotation-Invariant And Fisher Discriminative Convolutional Neural Networks For Object Detection

ℱ3-Net: Feature Fusion and Filtration Network for Object Detection in Optical Remote Sensing Images

Encoding High Dimensional Local Features By Sparse Coding Based Fisher Vectors

Locally-Transferred Fisher Vectors For Texture Classification

FDCNet: Filtering Deep Convolutional Network for Marine Organism Classification.

Analyzing Classifiers: Fisher Vectors and Deep Neural Networks

Fisher Kernel for Deep Neural Activations

Semantic Fisher Scores for Task Transfer: Using Objects to Classify Scenes

Scalable Object Detection Using Deep But Lightweight Cnn With Features Fusion