Abstract:While convolutional neural networks (CNNs) have been excellent for object recognition, the greater spatial variability in scene images typically means that the standard full-image CNNfeatures are suboptimal for scene classification. In this article, we investigate a framework allowing greater spatial flexibility, in which the Fisher vector (FV)-encoded distribution of local CNN features, obtained from a multitude of region proposals per image, is considered instead. The CNN features are computed from an augmented pixel-wise representation consisting of multiple modalities of RGB, HHA, and surface normals, as extracted from RGB-D data. More significantly, we make two postulates: (1) component sparsity-that only a small variety of region proposals and their corresponding FV GMM components contribute to scene discriminability, and (2) modal nonsparsity-that features from all modalities are encouraged to coexist. In our proposed feature fusion framework, these are implemented through regularization terms that apply group lasso to GMM components and exclusive group lasso across modalities. By learning and combining regressors for both proposal-based FV features and global CNN features, we are able to achieve state-of-the-art scene classification performance on the SUNRGBD Dataset and NYU Depth Dataset V2. Moreover, we further apply our feature fusion framework on an action recognition task to demonstrate that our framework can be generalized for other multimodal well-structured features. In particular, for action recognition, we enforce interpart sparsity to choose more discriminative body parts, and intermodal nonsparsity to make informative features from both appearance and motion modalities coexist. Experimental results on the JHMDB and MPII Cooking Datasets show that our feature fusion is also very effective for action recognition, achieving very competitive performance compared with the state of the art.

Recurrent Volume-Based 3-D Feature Fusion for Real-Time Multiview Object Pose Estimation.

Recurrent Volume-based 3D Feature Fusion for Real-time Multi-view Object Pose Estimation

MoreFusion: Multi-object Reasoning for 6D Pose Estimation from Volumetric Fusion

Temporal Consistent Object Pose Estimation from Monocular Videos

Zero-Shot 3d Pose Estimation of Unseen Object by Two-Step Rgb-D Fusion

Towards Two-view 6D Object Pose Estimation: A Comparative Study on Fusion Strategy

PA-Pose: Partial Point Cloud Fusion Based on Reliable Alignment for 6D Pose Tracking

An Iterative Attention Fusion Network for 6D Object Pose Estimation

An Efficient Color and Geometric Feature Fusion Module for 6D Object Pose Estiamtion

RGB-Fusion: Monocular 3D reconstruction with learned depth prediction

MixedFusion: 6D Object Pose Estimation from Decoupled RGB-Depth Features.

DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion

6D Object Pose Estimation Based on Cross-Modality Feature Fusion

TemporalFusion: Temporal Motion Reasoning with Multi-Frame Fusion for 6D Object Pose Estimation

Homologous multimodal fusion network with geometric constraint keypoints selection for 6D pose estimation

Cascaded Multi-3D-view Fusion for 3D-Oriented Object Detection

Structure-Aware Multimodal Feature Fusion for RGB-D Scene Classification and Beyond

A Novel Depth and Color Feature Fusion Framework for 6D Object Pose Estimation.

A Transformer-based multi-modal fusion network for 6D pose estimation

A Lightweight Color and Geometry Feature Extraction and Fusion Module for End-to-end 6D Pose Estimation

A Pose Estimation Algorithm for Multimodal Data Fusion