MaskVD: Region Masking for Efficient Video Object Detection

Sreetama Sarkar,Gourav Datta,Souvik Kundu,Kai Zheng,Chirayata Bhattacharyya,Peter A. Beerel

2024-07-16

Abstract:Video tasks are compute-heavy and thus pose a challenge when deploying in real-time applications, particularly for tasks that require state-of-the-art Vision Transformers (ViTs). Several research efforts have tried to address this challenge by leveraging the fact that large portions of the video undergo very little change across frames, leading to redundant computations in frame-based video processing. In particular, some works leverage pixel or semantic differences across frames, however, this yields limited latency benefits with significantly increased memory overhead. This paper, in contrast, presents a strategy for masking regions in video frames that leverages the semantic information in images and the temporal correlation between frames to significantly reduce FLOPs and latency with little to no penalty in performance over baseline models. In particular, we demonstrate that by leveraging extracted features from previous frames, ViT backbones directly benefit from region masking, skipping up to 80% of input regions, improving FLOPs and latency by 3.14x and 1.5x. We improve memory and latency over the state-of-the-art (SOTA) by 2.3x and 1.14x, while maintaining similar detection performance. Additionally, our approach demonstrates promising results on convolutional neural networks (CNNs) and provides latency improvements over the SOTA up to 1.3x using specialized computational kernels.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The paper primarily addresses the computationally intensive challenges in video object detection tasks, especially when using state-of-the-art Vision Transformers (ViTs) in real-time applications. The paper proposes a new method called MaskVD, which efficiently reduces computation by masking regions in video frames, thereby significantly lowering the number of floating-point operations (FLOPs) and latency while maintaining detection performance close to the baseline model.

Specifically, the core contribution of MaskVD lies in proposing a region masking strategy that considers the semantic information of the image and the temporal correlation between frames. This method can process only important regions, effectively improving detection performance and reducing computational costs. The input region masking strategy can be applied to Convolutional Neural Networks (CNNs) and ViTs. Unlike existing methods, MaskVD requires only a single region selection at the entry point of the detection pipeline, rather than repeatedly selecting regions at intermediate layers.

Additionally, MaskVD optimizes the computation process of the ViT backend by utilizing features extracted from previous frames. By storing a single tensor for the windowed transformer blocks, it reduces memory usage without sacrificing performance. Experimental results show that MaskVD can effectively reduce up to 80% of the input regions, while reducing FLOPs by 3.14 times and latency by 1.5 times, with only a 0.9% drop in detection performance. For non-windowed backends, although accuracy slightly decreases, higher latency improvements can be achieved.

In summary, MaskVD aims to improve the efficiency of frame-based video object detection through a novel region masking strategy that is applicable to both static and moving cameras. It can be seamlessly integrated during inference without retraining the model. This method enables efficient utilization of computational resources while maintaining similar detection performance.