Abstract:Deriving from the gradient vector of a generative model of local features, Fisher vector coding (FVC) has been identified as an effective coding method for image classification. Most, if not all, FVC implementations employ the Gaussian mixture model (GMM) as the generative model for local features. However, the representative power of a GMM can be limited because it essentially assumes that local features can be characterized by a fixed number of feature prototypes, and the number of prototypes is usually small in FVC. To alleviate this limitation, in this work, we break the convention which assumes that a local feature is drawn from one of a few Gaussian distributions. Instead, we adopt a compositional mechanism which assumes that a local feature is drawn from a Gaussian distribution whose mean vector is composed as a linear combination of multiple key components, and the combination weight is a latent random variable. In doing so we greatly enhance the representative power of the generative model underlying FVC. To implement our idea, we design two particular generative models following this compositional approach. In our first model, the mean vector is sampled from the subspace spanned by a set of bases and the combination weight is drawn from a Laplace distribution. In our second model, we further assume that a local feature is composed of a discriminative part and a residual part. As a result, a local feature is generated by the linear combination of discriminative part bases and residual part bases. The decomposition of the discriminative and residual parts is achieved via the guidance of a pre-trained supervised coding method. By calculating the gradient vector of the proposed models, we derive two new Fisher vector coding strategies. The first is termed Sparse Coding-based Fisher Vector Coding (SCFVC) and can be used as the substitute of traditional GMM based FVC. The second is termed Hybrid Sparse Coding-based Fisher vector coding (HSCFVC) since it combines the merits of both pre-trained supervised coding methods and FVC. Using pre-trained Convolutional Neural Network (CNN) activations as local features, we experimentally demonstrate that the proposed methods are superior to traditional GMM based FVC and achieve state-of-the-art performance in various image classification tasks.

Locally-Transferred Fisher Vectors For Texture Classification

Fisher Vector Based CNN Architecture for Image Classification.

Image Classification with CNN-based Fisher Vector Coding

Encoding High Dimensional Local Features By Sparse Coding Based Fisher Vectors

Deep FisherNet for Image Classification

Compositional Model Based Fisher Vector Coding for Image Classification

Deep FisherNet for Object Classification

Fusion of Complex Networks-based Global and Local Features for Texture Classification

A multilevel pooling scheme in convolutional neural networks for texture image recognition

C-CNN: Contourlet Convolutional Neural Networks

Bone texture characterization with fisher encoding of local descriptors

Hybrid RGB-D object recognition using Convolutional Neural Network and Fisher Vector

Spatial Weighted Fisher Vector for Image Retrieval

A Neighbourhood Feature-Based Local Binary Pattern for Texture Classification

Foreground Fisher Vector: Encoding Class-Relevant Foreground to Improve Image Classification

Efficient Image Categorization with Sparse Fisher Vector.

Deep-based Fisher Vector for Mobile Visual Search

Descriptor Learning Based on Fisher Separation Criterion for Texture Classification

Towards compact visual descriptor via deep fisher network with binary embedding

Exploiting Convolutional Neural Networks with Deeply Local Description for Remote Sensing Image Classification.

Fvct: Finger Vein Authentication Based on the Combination of Cnn and Transformer