Abstract:Statistical classification of actions in videos is mostly performed by extracting relevant features, particularly covariance features, from image frames and studying time series associated with temporal evolutions of these features. A natural mathematical representation of activity videos is in form of parameterized trajectories on the covariance manifold, i.e. the set of symmetric, positive-definite matrices (SPDMs). The variable execution-rates of actions implies variable parameterizations of the resulting trajectories, and complicates their classification. Since action classes are invariant to execution rates, one requires rate-invariant metrics for comparing trajectories. A recent paper represented trajectories using their transported square-root vector fields (TSRVFs), defined by parallel translating scaled-velocity vectors of trajectories to a reference tangent space on the manifold. To avoid arbitrariness of selecting the reference and to reduce distortion introduced during this mapping, we develop a purely intrinsic approach where SPDM trajectories are represented by redefining their TSRVFs at the starting points of the trajectories, and analyzed as elements of a vector bundle on the manifold. Using a natural Riemannain metric on vector bundles of SPDMs, we compute geodesic paths and geodesic distances between trajectories in the quotient space of this vector bundle, with respect to the re-parameterization group. This makes the resulting comparison of trajectories invariant to their re-parameterization. We demonstrate this framework on two applications involving video classification: visual speech recognition or lip-reading and hand-gesture recognition. In both cases we achieve results either comparable to or better than the current literature.

Action Recognition Based on Spatio-temporal Log-Euclidean Covariance Matrix

Human Action Recognition under Log-Euclidean Riemannian Metric.

Action Recognition Using Low-Rank Sparse Representation

Action Recognition with Spatial-Temporal Representation Analysis Across Grassmannian Manifold and Euclidean Space

Action Recognition Using Discriminative Spatio-Temporal Neighborhood Features

Spatio-temporal Laplacian Pyramid Coding for Action Recognition.

Cross-covariance matrix: Time-shifted correlations for 3D action recognition

Local Spatiotemporal Coding and Sparse Representation Based Human Action Recognition

Modeling Geometric-Temporal Context with Directional Pyramid Co-Occurrence for Action Recognition

Multiscale Spatial Position Coding under Locality Constraint for Action Recognition

Action Recognition Using Local Consistent Group Sparse Coding with Spatio-Temporal Structure.

Online Human Action Recognition Based on Incremental Learning of Weighted Covariance Descriptors.

Video-Based Action Recognition Using Rate-Invariant Analysis of Covariance Trajectories

Action Recognition Based on Spatial-Temporal Pyramid Sparse Coding.

A Non-negative Low Rank and Sparse Model for Action Recognition.

Hessian Regularized Sparse Coding for Human Action Recognition.

Action Recognition Using Spatiotemporal Features and Hybrid Generative/Discriminative Models

Discriminative Saliency-pose-attention Covariance for Action Recognition.

Human Action Recognition with Trajectory Based Covariance Descriptor in Unconstrained Videos

Research on Local Spatio-Temporal Features for Action Recognition

Human Action Recognition Algorithm Based on Spatial Temporal Depth Feature