Abstract:Recent developments in Transformers have achieved notable strides in enhancing video comprehension. Nonetheless, the O($N^2$) computation complexity associated with attention mechanisms presents substantial computational hurdles when dealing with the high dimensionality of videos. This challenge becomes particularly pronounced when striving to increase the frames per second (FPS) to enhance the motion capturing capabilities. Such a pursuit is likely to introduce redundancy and exacerbate the existing computational limitations. In this paper, we initiate by showcasing the enhanced performance achieved through an escalation in the FPS rate. Additionally, we present a novel approach, Motion Guided Token Compression (MGTC), to empower Transformer models to utilize a smaller yet more representative set of tokens for comprehensive video representation. Consequently, this yields substantial reductions in computational burden and remains seamlessly adaptable to increased FPS rates. Specifically, we draw inspiration from video compression algorithms and scrutinize the variance between patches in consecutive video frames across the temporal dimension. The tokens exhibiting a disparity below a predetermined threshold are then masked. Notably, this masking strategy effectively addresses video redundancy while conserving essential information. Our experiments, conducted on widely examined video recognition datasets, Kinetics-400, UCF101 and HMDB51, demonstrate that elevating the FPS rate results in a significant top-1 accuracy score improvement of over 1.6, 1.6 and 4.0. By implementing MGTC with the masking ratio of 25\%, we further augment accuracy by 0.1 and simultaneously reduce computational costs by over 31\% on Kinetics-400. Even within a fixed computational budget, higher FPS rates paired with MGTC sustain performance gains when compared to lower FPS settings.

Efficient Transformer Inference for Extremely Weak Edge Devices Using Masked Autoencoders.

Adaptive Offloading of Transformer Inference for Weak Edge Devices with Masked Autoencoders

Adaptive Offloading of Transformer Inference for Weak Edge Devices with Masked AutoencodersJust Accepted

Motion Guided Token Compression for Efficient Masked Video Modeling

Understanding Sensor Data Using Deep Learning Methods on Resource-Constrained Edge Devices.

AyE-Edge: Automated Deployment Space Search Empowering Accuracy Yet Efficient Real-Time Object Detection on the Edge

Masked Autoencoders As Image Processors.

Video Surveillance on Mobile Edge Networks—A Reinforcement-Learning-Based Approach

Intelligent Video Surveillance Based on Mobile Edge Networks

Efficient Transformer Encoders for Mask2Former-style models

Reinforcement Learning-based Mobile Edge Computing and Transmission Scheduling for Video Surveillance

Masked autoencoders are effective solution to transformer data-hungry

Transformer Inference Acceleration in Edge Computing Environment

The analysis of intelligent real-time image recognition technology based on mobile edge computing and deep learning

Lightweight Monocular Depth Estimation on Edge Devices

Accelerating Mobile Edge Generation (MEG) by Constrained Learning

Enhancing Security-Problem-Based Deep Learning in Mobile Edge Computing

Lightweight and Unobtrusive Data Obfuscation at IoT Edge for Remote Inference

Efficient Privacy Preserving Edge Computing Framework for Image Classification

Efficient Transformer-based Edge Detector

Masked Autoencoders Are Scalable Vision Learners