Can MLLMs Guide Weakly-Supervised Temporal Action Localization Tasks?

Quan Zhang,Yuxin Qi

2024-11-13

Abstract:Recent breakthroughs in Multimodal Large Language Models (MLLMs) have gained significant recognition within the deep learning community, where the fusion of the Video Foundation Models (VFMs) and Large Language Models(LLMs) has proven instrumental in constructing robust video understanding systems, effectively surmounting constraints associated with predefined visual tasks. These sophisticated MLLMs exhibit remarkable proficiency in comprehending videos, swiftly attaining unprecedented performance levels across diverse benchmarks. However, their operation demands substantial memory and computational resources, underscoring the continued importance of traditional models in video comprehension tasks. In this paper, we introduce a novel learning paradigm termed MLLM4WTAL. This paradigm harnesses the potential of MLLM to offer temporal action key semantics and complete semantic priors for conventional Weakly-supervised Temporal Action Localization (WTAL) methods. MLLM4WTAL facilitates the enhancement of WTAL by leveraging MLLM guidance. It achieves this by integrating two distinct modules: Key Semantic Matching (KSM) and Complete Semantic Reconstruction (CSR). These modules work in tandem to effectively address prevalent issues like incomplete and over-complete outcomes common in WTAL methods. Rigorous experiments are conducted to validate the efficacy of our proposed approach in augmenting the performance of various heterogeneous WTAL models.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve the accuracy of the Temporal Action Localization (TAL) task in videos under the weakly - supervised environment, especially for the problems of incomplete localization and over - complete localization. Specifically, the paper proposes a new learning paradigm called MLLM4WTAL, which utilizes Multimodal Large Language Models (MLLMs) to provide Key Semantic Matching (KSM) and Complete Semantic Reconstruction (CSR) modules to enhance the existing Weakly - supervised Temporal Action Localization (WTAL) methods. Through the interactive optimization strategies of these two modules, the paper aims to overcome the problems of incomplete and over - complete localization in current WTAL methods, thereby improving the performance of the model in identifying the temporal positions of action instances in videos.

The main contributions of the paper include:
1. Proposing a new learning paradigm MLLM4WTAL, which is the first to use MLLMs to guide the WTAL task and shows that this method can be easily extended to the existing state - of - the - art methods without introducing additional overhead in the inference stage.
2. Designing two modules, KSM and CSR, which enhance the WTAL method from the perspectives of key semantic matching and complete semantic reconstruction respectively.
3. Proposing a dual - prior interactive distillation strategy, which cleverly solves the problems of incomplete and over - complete localization in each module and realizes strong collaboration between KSM and CSR.
4. A large number of experiments show that this method achieves state - of - the - art performance on two popular benchmark datasets, THUMOS14 and ActivityNet1.2. Comprehensive ablation studies reveal the effectiveness of the proposed method.