Abstract:Large-scale video-language pre-training has made remarkable strides in advancing video-language understanding tasks. However, the heavy computational burden of video encoding remains a formidable efficiency bottleneck, particularly for long-form videos. These videos contain massive visual tokens due to their inherent 3D properties and spatiotemporal redundancy, making it challenging to capture complex temporal and spatial relationships. To tackle this issue, we propose an efficient method called TEmporal-Spatial Token Aggregation (TESTA). TESTA condenses video semantics by adaptively aggregating similar frames, as well as similar patches within each frame. TESTA can reduce the number of visual tokens by 75% and thus accelerate video encoding. Building upon TESTA, we introduce a pre-trained video-language model equipped with a divided space-time token aggregation module in each video encoder block. We evaluate our model on five datasets for paragraph-to-video retrieval and long-form VideoQA tasks. Experimental results show that TESTA improves computing efficiency by 1.7 times, and achieves significant performance gains from its scalability in processing longer input frames, e.g., +13.7 R@1 on QuerYD and +6.5 R@1 on Condensed Movie.

What problem does this paper attempt to address?

The paper aims to address the efficiency bottleneck in video encoding when processing long videos. Specifically, due to the three-dimensional nature and spatiotemporal redundancy of videos, a large number of visual tokens are generated during the video encoding process. This makes it difficult to capture complex spatiotemporal relationships and imposes a heavy computational burden. To tackle this challenge, the paper proposes an efficient method called Temporal-Spatial Token Aggregation (TESTA). TESTA condenses video semantics by adaptively aggregating similar frames and similar patches within each frame, thereby significantly reducing the number of visual tokens and accelerating the video encoding process. Experimental results show that TESTA not only improves computational efficiency (by approximately 1.7 times) but also demonstrates significant performance improvements when handling longer input frames. For example, it achieves a 13.7R@1 improvement on the QuerYD dataset and a 6.5R@1 improvement on the Condensed Movie dataset.

TESTA: Temporal-Spatial Token Aggregation for Long-form Video-Language Understanding