Abstract:While convolutional neural networks (CNNs) have been excellent for object recognition, the greater spatial variability in scene images typically means that the standard full-image CNNfeatures are suboptimal for scene classification. In this article, we investigate a framework allowing greater spatial flexibility, in which the Fisher vector (FV)-encoded distribution of local CNN features, obtained from a multitude of region proposals per image, is considered instead. The CNN features are computed from an augmented pixel-wise representation consisting of multiple modalities of RGB, HHA, and surface normals, as extracted from RGB-D data. More significantly, we make two postulates: (1) component sparsity-that only a small variety of region proposals and their corresponding FV GMM components contribute to scene discriminability, and (2) modal nonsparsity-that features from all modalities are encouraged to coexist. In our proposed feature fusion framework, these are implemented through regularization terms that apply group lasso to GMM components and exclusive group lasso across modalities. By learning and combining regressors for both proposal-based FV features and global CNN features, we are able to achieve state-of-the-art scene classification performance on the SUNRGBD Dataset and NYU Depth Dataset V2. Moreover, we further apply our feature fusion framework on an action recognition task to demonstrate that our framework can be generalized for other multimodal well-structured features. In particular, for action recognition, we enforce interpart sparsity to choose more discriminative body parts, and intermodal nonsparsity to make informative features from both appearance and motion modalities coexist. Experimental results on the JHMDB and MPII Cooking Datasets show that our feature fusion is also very effective for action recognition, achieving very competitive performance compared with the state of the art.

Spark Challenge: Multimodal Classifier for Space Target Recognition

RGB-D based multi-modal deep learning for spacecraft and debris recognition

A Depth Estimation Framework Based on Unsupervised Learning and Cross-Modal Translation

Localization and classification of space objects using EfficientDet detector for space situational awareness

Hyperspectral Anomaly Detection with Auto-Encoder and Independent Target

Look and Listen: A Multi-modality Late Fusion Approach to Scene Classification for Autonomous Machines

Pattern Recognition in Multispectral Satellite Images Using Concurrent Self-Organizing Modular Neural Networks

Multimodal deep learning for robust RGB-D object recognition

SpectraNet: Learned Recognition of Artificial Satellites From High Contrast Spectroscopic Imagery

MRSCAtt: A Spatio-Channel Attention-Guided Network for Mars Rover Image Classification

Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks

Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios

Multi-Task Learning of Height and Semantics from Aerial Images

Multi-Modal Domain Fusion for Multi-modal Aerial View Object Classification

CubifAE-3D: Monocular Camera Space Cubification for Auto-Encoder based 3D Object Detection

SpaRC: Sparse Radar-Camera Fusion for 3D Object Detection

GEM: Glare or Gloom, I Can Still See You -- End-to-End Multimodal Object Detection

Random Subspace Ensembles for Hyperspectral Image Classification With Extended Morphological Attribute Profiles

Structure-Aware Multimodal Feature Fusion for RGB-D Scene Classification and Beyond

3-D Deep Learning Approach for Remote Sensing Image Classification

Self-supervised Sparse-to-Dense: Self-supervised Depth Completion from LiDAR and Monocular Camera