Rethinking Video-Text Understanding: Retrieval from Counterfactually Augmented Data

Wufei Ma,Kai Li,Zhongshi Jiang,Moustafa Meshry,Qihao Liu,Huiyu Wang,Christian Häne,Alan Yuille

2024-07-18

Abstract:Recent video-text foundation models have demonstrated strong performance on a wide variety of downstream video understanding tasks. Can these video-text models genuinely understand the contents of natural videos? Standard video-text evaluations could be misleading as many questions can be inferred merely from the objects and contexts in a single frame or biases inherent in the datasets. In this paper, we aim to better assess the capabilities of current video-text models and understand their limitations. We propose a novel evaluation task for video-text understanding, namely retrieval from counterfactually augmented data (RCAD), and a new Feint6K dataset. To succeed on our new evaluation task, models must derive a comprehensive understanding of the video from cross-frame reasoning. Analyses show that previous video-text foundation models can be easily fooled by counterfactually augmented data and are far behind human-level performance. In order to narrow the gap between video-text models and human performance on RCAD, we identify a key limitation of current contrastive approaches on video-text data and introduce LLM-teacher, a more effective approach to learn action semantics by leveraging knowledge obtained from a pretrained large language model. Experiments and analyses show that our approach successfully learn more discriminative action embeddings and improves results on Feint6K when applied to multiple video-text models. Our Feint6K dataset and project page is available at <a class="link-external link-https" href="https://feint6k.github.io" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to solve several key problems faced by video - text understanding models when evaluating their ability to understand natural video content. Specifically, the paper aims to:

1. **Evaluate the true understanding ability of existing video - text models**: Although existing video - text models perform well in a variety of downstream tasks, whether these models can truly understand the semantics in videos (especially action semantics), and whether their performance is close to the human level, remains an unsolved problem.

2. **Eliminate shortcuts and biases in datasets**: Standard video - text evaluation methods can be misleading because many questions can be inferred from objects or context in a single frame, or by exploiting the inherent biases in the dataset (for example, the spurious association between "outdoor" and "football"). This allows the model to answer questions without performing cross - frame reasoning.

3. **Introduce a new evaluation paradigm**: In order to better evaluate the capabilities of video - text models and reveal their limitations, the authors propose a new evaluation task - Retrieval from Counterfactually Augmented Data (RCAD), and a new dataset Feint6K. In this new task, negative samples are generated by modifying the actions in the original captions while the object entities remain unchanged. The model must distinguish between positive and negative samples through an understanding of the entire video sequence, thereby avoiding reliance on shortcuts or biases.

4. **Improve the contrastive learning method**: In view of the limitations of existing contrastive learning methods on video - text data (such as the shortcut learning problem), the authors propose the LLM - teacher method. This method generates synthetic captions as additional learning resources by introducing the knowledge of pre - trained large - scale language models (LLMs), thereby learning action embeddings more effectively and improving the performance of the model on the RCAD task.

### Main contributions
- Propose a new evaluation task (RCAD) that requires cross - frame reasoning and its corresponding dataset Feint6K.
- Verify through experiments the limitations of existing video - text models in understanding video action semantics.
- Propose the LLM - teacher method, which combines the knowledge of pre - trained language models to learn action embeddings more effectively and significantly improves the performance of the model on the RCAD task.

### Conclusion
This research reveals the deficiencies of existing video - text models in understanding video content, and by introducing new evaluation tasks and improving learning methods, provides directions for future research.