Abstract:With the exponential growth of video content, the need for automated video highlight detection to extract key moments or highlights from lengthy videos has become increasingly pressing. This technology has the potential to significantly enhance user experiences by allowing quick access to relevant content across diverse domains. Existing methods typically rely either on expensive manually labeled frame-level annotations, or on a large external dataset of videos for weak supervision through category information. To overcome this, we focus on unsupervised video highlight detection, eliminating the need for manual annotations. We propose an innovative unsupervised approach which capitalizes on the premise that significant moments tend to recur across multiple videos of the similar category in both audio and visual modalities. Surprisingly, audio remains under-explored, especially in unsupervised algorithms, despite its potential to detect key moments. Through a clustering technique, we identify pseudo-categories of videos and compute audio pseudo-highlight scores for each video by measuring the similarities of audio features among audio clips of all the videos within each pseudo-category. Similarly, we also compute visual pseudo-highlight scores for each video using visual features. Subsequently, we combine audio and visual pseudo-highlights to create the audio-visual pseudo ground-truth highlight of each video for training an audio-visual highlight detection network. Extensive experiments and ablation studies on three highlight detection benchmarks showcase the superior performance of our method over prior work.

Highlight Detection For Video Content Analysis Through Double Filters

Highlight detection and removal based on chromaticity

Audiovisual Highlight Detection in Videos

Temporal Cue Guided Video Highlight Detection with Low-Rank Audio-Visual Fusion

A Semi-Automatic Feature Selecting Method For Sports Video Highlight Annotation

Detecting Highlighted Video Clips Through Emotion-Enhanced Audio-Visual Cues.

Highlight Extraction Based on Goal Detection in Soccer Video

Probing Visual-Audio Representation for Video Highlight Detection Via Hard-Pairs Guided Contrastive Learning

Visual-aural attention modeling for talk show video highlight detection

HighlightMe: Detecting Highlights from Human-Centric Videos

Two-Way Affective Modeling for Hidden Movie Highlights' Extraction

An Effective Audio-Visual Information Based Framework for Extracting Highlights in Basketball Games

Highlights Extraction in Soccer Videos Based on Goal-Mouth Detection

Unsupervised Video Highlight Detection by Learning from Audio and Visual Recurrence

Joint Visual and Audio Learning for Video Highlight Detection

Video Highlight Detection Via Deep Ranking Modeling

Image Highlight Removal Based on Double Edge-preserving Filter

Content-based Table Tennis Games Highlight Detection Utilizing Audiovisual Clues

Highlight Detection With Pairwise Deep Ranking For First-Person Video Summarization

Highlight Detection in Soccer Video Using Web-Casting Text

Show Me What I Like: Detecting User-Specific Video Highlights Using Content-Based Multi-Head Attention