Abstract:While convolutional neural networks (CNNs) have been excellent for object recognition, the greater spatial variability in scene images typically means that the standard full-image CNNfeatures are suboptimal for scene classification. In this article, we investigate a framework allowing greater spatial flexibility, in which the Fisher vector (FV)-encoded distribution of local CNN features, obtained from a multitude of region proposals per image, is considered instead. The CNN features are computed from an augmented pixel-wise representation consisting of multiple modalities of RGB, HHA, and surface normals, as extracted from RGB-D data. More significantly, we make two postulates: (1) component sparsity-that only a small variety of region proposals and their corresponding FV GMM components contribute to scene discriminability, and (2) modal nonsparsity-that features from all modalities are encouraged to coexist. In our proposed feature fusion framework, these are implemented through regularization terms that apply group lasso to GMM components and exclusive group lasso across modalities. By learning and combining regressors for both proposal-based FV features and global CNN features, we are able to achieve state-of-the-art scene classification performance on the SUNRGBD Dataset and NYU Depth Dataset V2. Moreover, we further apply our feature fusion framework on an action recognition task to demonstrate that our framework can be generalized for other multimodal well-structured features. In particular, for action recognition, we enforce interpart sparsity to choose more discriminative body parts, and intermodal nonsparsity to make informative features from both appearance and motion modalities coexist. Experimental results on the JHMDB and MPII Cooking Datasets show that our feature fusion is also very effective for action recognition, achieving very competitive performance compared with the state of the art.

Compositional action recognition with multi-view feature fusion

B2C-AFM: Bi-Directional Co-Temporal and Cross-Spatial Attention Fusion Model for Human Action Recognition.

Home Action Genome: Cooperative Compositional Action Understanding

Cross-modality Online Distillation for Multi-View Action Recognition

Multi-view key information representation and multi-modal fusion for single-subject routine action recognition

Progressive Instance-Aware Feature Learning for Compositional Action Recognition.

View-invariant Human Action Recognition Via Robust Locally Adaptive Multi-View Learning

Skeleton Sequence and RGB Frame Based Multi-Modality Feature Fusion Network for Action Recognition

Human action recognition via multi-view learning.

Hypergraph-Based Multi-View Action Recognition Using Event Cameras

Multi-view daily action recognition based on Hooke balanced matrix and broad learning system

Graph-Based Multi-Modal Multi-View Fusion for Facial Action Unit Recognition

Structure-Aware Multimodal Feature Fusion for RGB-D Scene Classification and Beyond

Rethinking Fusion Baselines for Multi-modal Human Action Recognition

M$^3$Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition

Multi-view Super Vector for Action Recognition

Deep Fusion of Multiple Semantic Cues for Complex Event Recognition

Early, intermediate and late fusion strategies for robust deep learning-based multimodal action recognition

Multi-layer Representation for Cross-view Action Recognition

MultiFuser: Multimodal Fusion Transformer for Enhanced Driver Action Recognition