Towards Surveillance Video-and-Language Understanding: New Dataset, Baselines, and Challenges

Tongtong Yuan,Xuange Zhang,Kun Liu,Bo Liu,Chen Chen,Jian Jin,Zhenzhen Jiao

2023-12-04

Abstract:Surveillance videos are an essential component of daily life with various critical applications, particularly in public security. However, current surveillance video tasks mainly focus on classifying and localizing anomalous events. Existing methods are limited to detecting and classifying the predefined events with unsatisfactory semantic understanding, although they have obtained considerable performance. To address this issue, we propose a new research direction of surveillance video-and-language understanding, and construct the first multimodal surveillance video dataset. We manually annotate the real-world surveillance dataset UCF-Crime with fine-grained event content and timing. Our newly annotated dataset, UCA (UCF-Crime Annotation), contains 23,542 sentences, with an average length of 20 words, and its annotated videos are as long as 110.7 hours. Furthermore, we benchmark SOTA models for four multimodal tasks on this newly created dataset, which serve as new baselines for surveillance video-and-language understanding. Through our experiments, we find that mainstream models used in previously publicly available datasets perform poorly on surveillance video, which demonstrates the new challenges in surveillance video-and-language understanding. To validate the effectiveness of our UCA, we conducted experiments on multimodal anomaly detection. The results demonstrate that our multimodal surveillance learning can improve the performance of conventional anomaly detection tasks. All the experiments highlight the necessity of constructing this dataset to advance surveillance AI. The link to our dataset is provided at: <a class="link-external link-https" href="https://xuange923.github.io/Surveillance-Video-Understanding" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The problem this paper attempts to address is that current surveillance video tasks mainly focus on classifying and locating anomalous events. Although existing methods have achieved significant performance in detecting and classifying predefined events, they lack in-depth semantic understanding of video content. To tackle this challenge, the authors propose a new research direction—surveillance video and language understanding, and construct the first multimodal surveillance video dataset.

Specifically, the paper aims to expand existing surveillance video datasets by introducing fine-grained event content and temporal annotations, thereby supporting more complex multimodal learning tasks such as moment retrieval and video captioning. This not only has the potential to improve the performance of existing surveillance applications but also to advance surveillance AI, particularly in detailed case investigations in the public safety domain. Through experimental validation, the authors found that mainstream models perform poorly on publicly available datasets, indicating the need to design new foundational learning architectures tailored to the unique characteristics of surveillance videos. Therefore, constructing such a dataset is of great significance for advancing the multimodal understanding of surveillance videos.