Abstract:Recent developments in Transformers have achieved notable strides in enhancing video comprehension. Nonetheless, the O($N^2$) computation complexity associated with attention mechanisms presents substantial computational hurdles when dealing with the high dimensionality of videos. This challenge becomes particularly pronounced when striving to increase the frames per second (FPS) to enhance the motion capturing capabilities. Such a pursuit is likely to introduce redundancy and exacerbate the existing computational limitations. In this paper, we initiate by showcasing the enhanced performance achieved through an escalation in the FPS rate. Additionally, we present a novel approach, Motion Guided Token Compression (MGTC), to empower Transformer models to utilize a smaller yet more representative set of tokens for comprehensive video representation. Consequently, this yields substantial reductions in computational burden and remains seamlessly adaptable to increased FPS rates. Specifically, we draw inspiration from video compression algorithms and scrutinize the variance between patches in consecutive video frames across the temporal dimension. The tokens exhibiting a disparity below a predetermined threshold are then masked. Notably, this masking strategy effectively addresses video redundancy while conserving essential information. Our experiments, conducted on widely examined video recognition datasets, Kinetics-400, UCF101 and HMDB51, demonstrate that elevating the FPS rate results in a significant top-1 accuracy score improvement of over 1.6, 1.6 and 4.0. By implementing MGTC with the masking ratio of 25\%, we further augment accuracy by 0.1 and simultaneously reduce computational costs by over 31\% on Kinetics-400. Even within a fixed computational budget, higher FPS rates paired with MGTC sustain performance gains when compared to lower FPS settings.

MARS: Masked Automatic Ranks Selection in Tensor Decompositions

Motion Guided Token Compression for Efficient Masked Video Modeling

Multi-Dictionary Tensor Decomposition

Tensor Ring Decomposition with Rank Minimization on Latent Space: An Efficient Approach for Tensor Completion

Bayesian tensorized neural networks with automatic rank selection

Maestro: Uncovering Low-Rank Structures via Trainable Decomposition

Accelerating the Low-Rank Decomposed Models

Model based Multi-agent Reinforcement Learning with Tensor Decompositions

Unified Framework for Neural Network Compression via Decomposition and Optimal Rank Selection

Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition

Rank Minimization on Tensor Ring: A New Paradigm in Scalable Tensor Decomposition and Completion

DeepTensor: Low-Rank Tensor Decomposition with Deep Network Priors

Multiple Low-Ranks Plus Sparsity Based Tensor Reconstruction for Dynamic MRI.

FRAPPE: fast rank approximation with explainable features for tensors

Tensor rank learning in CP decomposition via convolutional neural network

Towards Compact Neural Networks via End-to-End Training: A Bayesian Tensor Approach with Automatic Rank Determination

MACH: Fast Randomized Tensor Decompositions

Incremental multi-domain learning with network latent tensor factorization

Latent Matrices for Tensor Network Decomposition and to Tensor Completion

Compression of Recurrent Neural Networks using Matrix Factorization

A Practical Approach for Employing Tensor Train Decomposition in Edge Devices