Abstract:Human actions in video sequences are characterized by the complex interplay between spatial features and their temporal dynamics. In this paper, we propose novel tensor representations for compactly capturing such higher-order relationships between visual features for the task of action recognition. We propose two tensor-based feature representations, viz. (i) sequence compatibility kernel (SCK) and (ii) dynamics compatibility kernel (DCK). SCK builds on the spatio-temporal correlations between features, whereas DCK explicitly models the action dynamics of a sequence. We also explore generalization of SCK, coined SCK ⊕, that operates on subsequences to capture the local-global interplay of correlations, which can incorporate multi-modal inputs e.g., skeleton 3D body-joints and per-frame classifier scores obtained from deep learning models trained on videos. We introduce linearization of these kernels that lead to compact and fast descriptors. We provide experiments on (i) 3D skeleton action sequences, (ii) fine-grained video sequences, and (iii) standard non-fine-grained videos. As our final representations are tensors that capture higher-order relationships of features, they relate to co-occurrences for robust fine-grained recognition (Lin, 2017), (Koniusz, 2018). We use higher-order tensors and so-called Eigenvalue Power Normalization (EPN) which have been long speculated to perform spectral detection of higher-order occurrences (Koniusz, 2013), (Koniusz, 2017), thus detecting fine-grained relationships of features rather than merely count features in action sequences. We prove that a tensor of order r, built from Z* dimensional features, coupled with EPN indeed detects if at least one higher-order occurrence is 'projected' into one of its [Formula: see text] subspaces of dim. r represented by the tensor, thus forming a Tensor Power Normalization metric endowed with [Formula: see text] such 'detectors'.

Tensorial kernel principal component analysis for action recognition

High-Order PCA of Video Volume Tensors for Human Action Representation and Recognition Shu Kong And

Tensor Canonical Correlation Analysis for Action Classification

Tensor Robust Kernel PCA for Multidimensional Data

Tensor Representations for Action Recognition

STPCA: Sparse tensor Principal Component Analysis for feature extraction

Tensor discriminant analysis on grassmann manifold with application to video based human action recognition

Action Recognition Using Subtensor Constraint

Robust Tensor Principal Component Analysis in All Modes

Iterative Block Tensor Singular Value Thresholding for Extraction of Low Rank Component of Image Data

On Multilinear Principal Component Analysis of Order-Two Tensors

Online Multilinear Principal Component Analysis.

Tensor-based Projection Using Ridge Regression and Its Application to Action Classification

Extension of PCA to Higher Order Data Structures: An Introduction to Tensors, Tensor Decompositions, and Tensor PCA

Cosine Multilinear Principal Component Analysis for Recognition

Effective Fusing the Factored Matrices in Dual Tensors for Action Recognition

Multilinear Sparse Principal Component Analysis

Incremental Multi-Linear Discriminant Analysis Using Canonical Correlations for Action Recognition

Low-Rank Tensor Learning with Discriminant Analysis for Action Classification and Image Recovery

Tensor Robust Principal Component Analysis Via Non-convex Low-Rank Approximation Based on the Laplace Function

Human Action Recognition Based on Tensor Shape Descriptor.