Abstract:Recently, some multi-modality features are introduced to the multi-view action recognition methods in order to obtain a more robust performance. However, it is intuitive that not all modalities are avail- able in real applications. For example, daily scenes lack depth modal data and capture RGB sequences only. Thus comes the challenge of learning critical features from multi-modality data at train time, while still getting robust performance based on RGB sequences at test time. To address this chal- lenge, our paper presents a novel two-stage teacher-student framework. The teacher network takes advantage of multi-view geometry-and-texture features during training, while the student network is given only RGB sequences at test time. Specifically, in the first stage, Cross-modality Aggregated Transfer (CAT) network is proposed to transfer multi-view cross-modality aggregated features from the teacher network to the student network. Moreover, we design a Viewpoint-Aware Attention (VAA) module which captures discriminative information across different views to combine multi-view fea- tures effectively. In the second stage, Multi-view Features Strengthen (MFS) network with the VAA module further strengthens the global view-invariance features of the student network. Besides, both of CAT and MFS learn in an online distillation manner, so that the teacher and the student network can be trained jointly. Extensive experiments on IXMAS and Northwestern-UCLA demonstrate the effectiveness of our proposed method.

Multi-task Information Bottleneck Co-clustering for Unsupervised Cross-view Human Action Categorization

Multi-View Clustering Via Simultaneously Learning Shared Subspace And Affinity Matrix

Cross-modality Online Distillation for Multi-View Action Recognition

View-invariant Human Action Recognition Via Robust Locally Adaptive Multi-View Learning

Live and Learn: Continual Action Clustering with Incremental Views

Hierarchical Mutual Information Analysis: Towards Multi-view Clustering in The Wild

A Matrix-Based Approach to Unsupervised Human Action Categorization

Multi-Modal Multi-Action Video Recognition.

Cross-view action recognition via view knowledge transfer

Multi-view daily action recognition based on Hooke balanced matrix and broad learning system

An Information Maximization Multi-task Clustering Method for egocentric temporal segmentation

Multi-View Information-Theoretic Co-Clustering for Co-Occurrence Data

A Joint Matrix Factorization Approach to Unsupervised Action Categorization

Human action recognition via multi-view learning.

Representing Videos As Discriminative Sub-graphs for Action Recognition*

Multi-Task Co-Clustering Via Nonnegative Matrix Factorization

A 3D-CNN and LSTM Based Multi-Task Learning Architecture for Action Recognition.

Action Recognition with Multi-stream Motion Modeling and Mutual Information Maximization

Multi-View Matrix Completion for Clustering with Side Information.

Multi-layer Representation for Cross-view Action Recognition

Constrained Bilinear Factorization Multi-view Subspace Clustering