Abstract:Motion recognition is a promising direction in computer vision, but the training of video classification models is much harder than images due to insufficient data and considerable parameters. To get around this, some works strive to explore multimodal cues from RGB-D data. Although improving motion recognition to some extent, these methods still face sub-optimal situations in the following aspects: (i) Data augmentation, i.e., the scale of the RGB-D datasets is still limited, and few efforts have been made to explore novel data augmentation strategies for videos; (ii) Optimization mechanism, i.e., the tightly space-time-entangled network structure brings more challenges to spatiotemporal information modeling; And (iii) cross-modal knowledge fusion, i.e., the high similarity between multimodal representations caused to insufficient late fusion. To alleviate these drawbacks, we propose to improve RGB-D-based motion recognition both from data and algorithm perspectives in this paper. In more detail, firstly, we introduce a novel video data augmentation method dubbed ShuffleMix, which acts as a supplement to MixUp, to provide additional temporal regularization for motion recognition. Secondly, a Unified Multimodal De-coupling and multi-stage Re-coupling framework, termed UMDR, is proposed for video representation learning. Finally, a novel cross-modal Complement Feature Catcher (CFCer) is explored to mine potential commonalities features in multimodal information as the auxiliary fusion stream, to improve the late fusion results. The seamless combination of these novel designs forms a robust spatiotemporal representation and achieves better performance than state-of-the-art methods on four public motion datasets. Specifically, UMDR achieves unprecedented improvements of +4.5% on the Chalearn IsoGD dataset. Our code is available at <a class="link-external link-https" href="https://github.com/zhoubenjia/MotionRGBD-PAMI" rel="external noopener nofollow">this https URL</a>.

Visual Recognition in RGB Images and Videos by Learning from RGB-D Data.

Improving RGB-D Face Recognition via Transfer Learning from a Pretrained 2D Network.

Semi-supervised Learning for RGB-D Object Recognition.

Uniform and Variational Deep Learning for RGB-D Object Recognition and Person Re-Identification.

A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition

RGB×D: Learning Depth-Weighted RGB Patches for RGB-D Indoor Semantic Segmentation

Unsupervised Domain Adaptation Approach for Vision-Based Semantic Understanding of Bridge Inspection Scenes Without Manual Annotations

Multi-Modal Unsupervised Domain Adaptation for Semantic Image Segmentation

Adversarial Unsupervised Domain Adaptation for 3D Semantic Segmentation with 2D Image Fusion of Dense Depth

Depth Images Could Tell Us More: Enhancing Depth Discriminability for RGB-D Scene Recognition

Unsupervised Domain Adaptation for Video Object Grounding with Cascaded Debiasing Learning

Learning Feature Decomposition for Domain Adaptive Monocular Depth Estimation

Robust Multiview Feature Learning for RGB-D Image Understanding

Recurrent Convolutional Fusion for RGB-D Object Recognition

Multichannel Semantic Segmentation with Unsupervised Domain Adaptation

Enhancing Visual Domain Adaptation with Source Preparation

Multimodal deep learning for robust RGB-D object recognition

Large-Margin Multi-Modal Deep Learning for RGB-D Object Recognition

Video domain adaptation for semantic segmentation using perceptual consistency matching

Recognizing Multi-View Objects with Occlusions Using a Deep Architecture

Unified Domain Generalization and Adaptation for Multi-View 3D Object Detection