Abstract:For video recognition task, a global representation summarizing the whole contents of the video snippets plays an important role for the final performance. However, existing video architectures usually generate it by using a simple, global average pooling (GAP) method, which has limited ability to capture complex dynamics of videos. For image recognition task, there exist evidences showing that covariance pooling has stronger representation ability than GAP. Unfortunately, such plain covariance pooling used in image recognition is an orderless representative, which cannot model spatio-temporal structure inherent in videos. Therefore, this paper proposes a Temporal-attentive Covariance Pooling(TCP), inserted at the end of deep architectures, to produce powerful video representations. Specifically, our TCP first develops a temporal attention module to adaptively calibrate spatio-temporal features for the succeeding covariance pooling, approximatively producing attentive covariance representations. Then, a temporal covariance pooling performs temporal pooling of the attentive covariance representations to characterize both intra-frame correlations and inter-frame cross-correlations of the calibrated features. As such, the proposed TCP can capture complex temporal dynamics. Finally, a fast matrix power normalization is introduced to exploit geometry of covariance representations. Note that our TCP is model-agnostic and can be flexibly integrated into any video architectures, resulting in TCPNet for effective video recognition. The extensive experiments on six benchmarks (e.g., Kinetics, Something-Something V1 and Charades) using various video architectures show our TCPNet is clearly superior to its counterparts, while having strong generalization ability. The source code is publicly available.

Fast Calculation of Covariance Matrices for Arbitrary Size Cuboids

Feature Based Inter Prediction Optimization for Non-Translational Video Coding in Cloud

Blind CFA Interpolation Detection Based on Covariance Matrix

Identification of Source Cameras Based on CFA Interpolation Algorithm Detection Using Covariance Matrix

Compact Approximation for Polynomial of Covariance Feature

Human-Machine Collaborative Video Coding Through Cuboidal Partitioning

Texture and Correlation Based Fast Intra Prediction Algorithm for HEVC

The Feature and Spatial Covariant Kernel

Efficient depth coding in 3D video to minimize coding bitrate and complexity

Fast disparity estimation using spatio-temporal correlation of disparity field for multiview video coding

Incremetal Spatio-Temporal Feature Extraction and Retrieval for Large Video Database

Fast disparity and motion estimation based on correlations for multiview video coding

A time-slice optimization based weak feature association algorithm for video condensation

Temporal-attentive Covariance Pooling Networks for Video Recognition

Rapid Numerical Approximation Method for Integrated Covariance Functions Over Irregular Data Regions

Fast VVC Intra Encoding for Video Coding for Machines

A Covariance-Free Iterative Principal Component Analysis for High Dimensional and Large Scale Data

A novel fast search method to find disparity vectors in multiview video coding

A Bayesian Factorised Covariance Model for Image Analysis.

Fast Covariance Estimation for Sparse Functional Data

Video Frame Interpolation with Densely Queried Bilateral Correlation