Abstract:While convolutional neural networks (CNNs) have been excellent for object recognition, the greater spatial variability in scene images typically means that the standard full-image CNNfeatures are suboptimal for scene classification. In this article, we investigate a framework allowing greater spatial flexibility, in which the Fisher vector (FV)-encoded distribution of local CNN features, obtained from a multitude of region proposals per image, is considered instead. The CNN features are computed from an augmented pixel-wise representation consisting of multiple modalities of RGB, HHA, and surface normals, as extracted from RGB-D data. More significantly, we make two postulates: (1) component sparsity-that only a small variety of region proposals and their corresponding FV GMM components contribute to scene discriminability, and (2) modal nonsparsity-that features from all modalities are encouraged to coexist. In our proposed feature fusion framework, these are implemented through regularization terms that apply group lasso to GMM components and exclusive group lasso across modalities. By learning and combining regressors for both proposal-based FV features and global CNN features, we are able to achieve state-of-the-art scene classification performance on the SUNRGBD Dataset and NYU Depth Dataset V2. Moreover, we further apply our feature fusion framework on an action recognition task to demonstrate that our framework can be generalized for other multimodal well-structured features. In particular, for action recognition, we enforce interpart sparsity to choose more discriminative body parts, and intermodal nonsparsity to make informative features from both appearance and motion modalities coexist. Experimental results on the JHMDB and MPII Cooking Datasets show that our feature fusion is also very effective for action recognition, achieving very competitive performance compared with the state of the art.

Multimodal Remote Sensing Data Classification Based on Gaussian Mixture Variational Dynamic Fusion Network

Multi-Level Interactive Fusion Network Based on Adversarial Learning for Fusion Classification of Hyperspectral and Lidar Data

GCFnet: Global Collaborative Fusion Network for Multispectral and Panchromatic Image Classification

Multisource Remote Sensing Data Classification with Graph Fusion Network

Fractional Gabor Convolutional Network for Multisource Remote Sensing Data Classification

Modality Fusion Vision Transformer for Hyperspectral and LiDAR Data Collaborative Classification

Classification of Multisource Remote Sensing Images Using Multimodal Equilateral Absorption Network

Multifrequency Graph Convolutional Network With Cross-Modality Mutual Enhancement for Multisource Remote Sensing Data Classification

MGFN: A Multi-Granularity Fusion Convolutional Neural Network for Remote Sensing Scene Classification

Multi-Frequency Graph Convolutional Network with Cross-Modality Mutual Enhancement for Multisource Remote Sensing Data Classification

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data

Attention Fusion of Transformer-Based and Scale-Based Method for Hyperspectral and LiDAR Joint Classification

MHFNet: an Improved HGR Multimodal Network for Informative Correlation Fusion in Remote Sensing Image Classification

Multimodal Hyperspectral Image Classification via Interconnected Fusion

Multiple Information Collaborative Fusion Network for Joint Classification of Hyperspectral and LiDAR Data

Reinforcement Learning Based Markov Edge Decoupled Fusion Network for Fusion Classification of Hyperspectral and LiDAR

FDGSNet: A Multi-modal Gated Segmentation Network for Remote Sensing Image Based on Frequency Decomposition

Dual-Branch Dynamic Modulation Network for Hyperspectral and LiDAR Data Classification.

Enhancing Multimodal Fusion with Only Unimodal Data

Structure-Aware Multimodal Feature Fusion for RGB-D Scene Classification and Beyond

Deep Multi-Level Fusion Network for Multi-Source Image Pixel-Wise Classification