Abstract:While convolutional neural networks (CNNs) have been excellent for object recognition, the greater spatial variability in scene images typically means that the standard full-image CNNfeatures are suboptimal for scene classification. In this article, we investigate a framework allowing greater spatial flexibility, in which the Fisher vector (FV)-encoded distribution of local CNN features, obtained from a multitude of region proposals per image, is considered instead. The CNN features are computed from an augmented pixel-wise representation consisting of multiple modalities of RGB, HHA, and surface normals, as extracted from RGB-D data. More significantly, we make two postulates: (1) component sparsity-that only a small variety of region proposals and their corresponding FV GMM components contribute to scene discriminability, and (2) modal nonsparsity-that features from all modalities are encouraged to coexist. In our proposed feature fusion framework, these are implemented through regularization terms that apply group lasso to GMM components and exclusive group lasso across modalities. By learning and combining regressors for both proposal-based FV features and global CNN features, we are able to achieve state-of-the-art scene classification performance on the SUNRGBD Dataset and NYU Depth Dataset V2. Moreover, we further apply our feature fusion framework on an action recognition task to demonstrate that our framework can be generalized for other multimodal well-structured features. In particular, for action recognition, we enforce interpart sparsity to choose more discriminative body parts, and intermodal nonsparsity to make informative features from both appearance and motion modalities coexist. Experimental results on the JHMDB and MPII Cooking Datasets show that our feature fusion is also very effective for action recognition, achieving very competitive performance compared with the state of the art.

Multi-Modal Local Receptive Field Extreme Learning Machine for Object Recognition.

Correntropy Extreme Learning Machine Based on Spatial Pyramid Matching and Local Receptive Field

Multiple-kernel-learning-based Extreme Learning Machine for Classification Design

Extreme Learning Machine Combining Hidden-Layer Feature Weighting and Batch Training for Classification

Correntropy-based Fusion Extreme Learning Machine for Representation Level Feature Fusion and Classification

Robotic Grasping Recognition Using Multi-Modal Deep Extreme Learning Machine

An Adaptive Fuzzy Fusion Framework for Face Recognition under Illumination Variation Based on Local Multiple Patterns.

Advancing the incremental fusion of robotic sensory features using online multi-kernel extreme learning machine

Combining 2D Gabor and Local Binary Pattern for Facial Expression Recognition Using Extreme Learning Machine

MMSS: Multi-modal Sharable and Specific Feature Learning for RGB-D Object Recognition.

Extreme learning machine with autoencoding receptive fields for image classification

Surface Material Recognition Using Active Multi-modal Extreme Learning Machine.

Optimization-based Multikernel Extreme Learning for Multimodal Object Image Classification

Large-Margin Multi-Modal Deep Learning for RGB-D Object Recognition

Haptic Recognition Using Hierarchical Extreme Learning Machine with Local-Receptive-field

A Novel Bio-inspired Image Recognition Network with Extreme Learning Machine

Weakly paired multimodal fusion using multilayer extreme learning machine

Active Object Recognition Using Hierarchical Local-Receptive-field-based Extreme Learning Machine

Structure-Aware Multimodal Feature Fusion for RGB-D Scene Classification and Beyond

Local Receptive Field Based Extreme Learning Machine with Three Channels for Histopathological Image Classification

More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification