Abstract:While convolutional neural networks (CNNs) have been excellent for object recognition, the greater spatial variability in scene images typically means that the standard full-image CNNfeatures are suboptimal for scene classification. In this article, we investigate a framework allowing greater spatial flexibility, in which the Fisher vector (FV)-encoded distribution of local CNN features, obtained from a multitude of region proposals per image, is considered instead. The CNN features are computed from an augmented pixel-wise representation consisting of multiple modalities of RGB, HHA, and surface normals, as extracted from RGB-D data. More significantly, we make two postulates: (1) component sparsity-that only a small variety of region proposals and their corresponding FV GMM components contribute to scene discriminability, and (2) modal nonsparsity-that features from all modalities are encouraged to coexist. In our proposed feature fusion framework, these are implemented through regularization terms that apply group lasso to GMM components and exclusive group lasso across modalities. By learning and combining regressors for both proposal-based FV features and global CNN features, we are able to achieve state-of-the-art scene classification performance on the SUNRGBD Dataset and NYU Depth Dataset V2. Moreover, we further apply our feature fusion framework on an action recognition task to demonstrate that our framework can be generalized for other multimodal well-structured features. In particular, for action recognition, we enforce interpart sparsity to choose more discriminative body parts, and intermodal nonsparsity to make informative features from both appearance and motion modalities coexist. Experimental results on the JHMDB and MPII Cooking Datasets show that our feature fusion is also very effective for action recognition, achieving very competitive performance compared with the state of the art.

AdaNFF: A New Method for Adaptive Nonnegative Multi-Feature Fusion to Scene Classification.

Sentiment Analysis Using Deep Robust Complementary Fusion of Multi-Features and Multi-Modalities.

AttentionBased Deep Feature Fusion for the Scene Classification of HighResolution Remote Sensing Images

RETRACTED: Attention-Based Deep Feature Fusion for the Scene Classification of High-Resolution Remote Sensing Images

AMFF-net: Adaptive Multi-Modal Feature Fusion Network for Image Classification

Deep Feature Fusion for High-Resolution Aerial Scene Classification

Deep feature fusion through adaptive discriminative metric learning for scene recognition

A Global-Local Feature Adaptive Fusion Network for Image Scene Classification

An Adaptive Multilayer Feature Fusion Strategy for Remote Sensing Scene Classification

AFDFusion: an Adaptive Frequency Decoupling Fusion Network for Multi-Modality Image

A dual-branch multi-feature deep fusion network framework for hyperspectral image classification

An adaptive feature fusion framework for multi-class classification based on SVM

Decision-Level Fusion with a Pluginable Importance Factor Generator for Remote Sensing Image Scene Classification

A Multi-Feature Fusion Model Based on Denoising Convolutional Neural Network and Attention Mechanism for Image Classification

AFN: Adaptive Fusion Normalization via an Encoder-Decoder Framework

Structure-Aware Multimodal Feature Fusion for RGB-D Scene Classification and Beyond

Multi-Focus Image Fusing Based on Non-Negative Matrix Factorization

Multi-feature Fusion Deep Networks

Adaptive Ensemble Learning: Boosting Model Performance through Intelligent Feature Fusion in Deep Neural Networks

Deep Mutual Attention Network for Acoustic Scene Classification

Improving Object Detection with Feature Fusion Methods.