Abstract:Understanding movies and their structural patterns is a crucial task in decoding the craft of video editing. While previous works have developed tools for general analysis, such as detecting characters or recognizing cinematography properties at the shot level, less effort has been devoted to understanding the most basic video edit, the Cut. This paper introduces the Cut type recognition task, which requires modeling multi-modal information. To ignite research in this new task, we construct a large-scale dataset called MovieCuts, which contains 173,967 video clips labeled with ten cut types defined by professionals in the movie industry. We benchmark a set of audio-visual approaches, including some dealing with the problem's multi-modal nature. Our best model achieves 47.7% mAP, which suggests that the task is challenging and that attaining highly accurate Cut type recognition is an open research problem. Advances in automatic Cut-type recognition can unleash new experiences in the video editing industry, such as movie analysis for education, video re-editing, virtual cinematography, machine-assisted trailer generation, machine-assisted video editing, among others. Our data and code are publicly available: <a class="link-external link-https" href="https://github.com/PardoAlejo/MovieCuts" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/PardoAlejo/MovieCuts" rel="external noopener nofollow">this https URL</a>.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the problem of automatic identification of the **Cut (cutting)** type, which is the most fundamental but crucial editing technique in film editing. Specifically, the paper introduces a new task: **cut - type identification**, and constructs a large - scale dataset MovieCuts for this purpose. #### Main problem background: 1. **Understanding of film editing**: - Understanding films and their structural patterns is crucial for decoding video - editing techniques. Although previous research has developed tools for general analysis, such as detecting characters or identifying shot - level cinematographic properties, relatively little research has been done on understanding the most basic video - editing operation - the cut. 2. **Importance of the cut**: - The cut is the most commonly used transition method in films. It is not just a simple switch between two shots, but has specific purposes and meanings. Different types of cuts can convey different semantic information, such as emotional expression, change of perspective, etc. 3. **Deficiencies of existing tools**: - Although there are some tools that can analyze the cut frequency in films (such as Cinemetrics), challenges still remain in understanding the semantic meaning of these cuts. Existing computer vision methods mainly focus on shot - level analysis, and the automatic identification and understanding of cut types have not been fully studied. #### Specific objectives of the paper: - **Introducing the cut - type identification task**: Formally define and study this new task of cut - type identification, requiring the model to be able to handle multimodal information (audio and video). - **Constructing a large - scale dataset**: Create a large - scale dataset named MovieCuts, which contains 173,967 video segments with professionally - annotated cut - type labels, covering ten different cut types. - **Establishing a benchmark model**: Implement multiple audio - visual baseline models and establish a benchmark for the cut - type identification task. The experimental results show that the mAP (mean Average Precision) of the best model is 47.7%, indicating that this task is challenging and requires further research. #### Potential applications: - **Education and analysis**: Automated cut - type identification can assist in film analysis and education, providing a deeper understanding of film - editing styles. - **Video re - editing**: Support application scenarios such as virtual cinematography, machine - assisted trailer generation, and machine - assisted video editing. By solving these problems, this research provides new tools and methods for the automated understanding and analysis of film editing, promoting the progress in the field of video editing.

MovieCuts: A New Dataset and Benchmark for Cut Type Recognition

ActionCLIP: Adapting Language-Image Pretrained Models for Video Action Recognition.

The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing

Match Cutting: Finding Cuts with Smooth Visual Transitions

MovieCLIP: Visual Scene Recognition in Movies

Moviescope: Large-scale Analysis of Movies using Multiple Modalities

Discontinuity-aware Video Object Cutout

Where to look at the movies: Analyzing visual attention to understand movie editing

Exploring challenge and explainable shot type classification using SAM-guided approaches

Audio Match Cutting: Finding and Creating Matching Audio Transitions in Movies and Videos

MovieNet: A Holistic Dataset for Movie Understanding

Visual Motif Identification: Elaboration of a Curated Comparative Dataset and Classification Methods

PyCinemetrics: Computational film studies tool based on deep learning and PySide2

A Unified Framework for Shot Type Classification Based on Subject Centric Lens

Movie101: A New Movie Understanding Benchmark

A Novel Algorithm for Cut Shot Boundary Detection

CineScale: A dataset of cinematic shot scale in movies

A Benchmarking Campaign for the Multimodal Detection of Violent Scenes in Movies

Scene Segmentation and Categorization Using NCuts

Edit3K: Universal Representation Learning for Video Editing Components

Editing like Humans: A Contextual, Multimodal Framework for Automated Video Editing