Abstract:The video highlight detection task is to localize key elements (moments of user's major or special interest) in a video. Most of existing highlight detection approaches extract features from the video segment as a whole without considering the difference of local features both temporally and spatially. Due to the complexity of video content, this kind of mixed features will impact the final highlight prediction. In temporal extent, not all frames are worth watching because some of them only contain the background of the environment without human or other moving objects. In spatial extent, it is similar that not all regions in each frame are highlights especially when there are lots of clutters in the background. To solve the above problem, we propose a novel three-dimensional (3-D) (spatial+temporal) attention model that can automatically localize the key elements in a video without any extra supervised annotations. Specifically, the proposed attention model produces attention weights of local regions along both the spatial and temporal dimensions of the video segment. The regions of key elements in the video will be strengthened with large weights. Thus, the more effective feature of the video segment is obtained to predict the highlight score. The proposed 3-D attention scheme can be easily integrated into a conventional end-to-end deep ranking model that aims to learn a deep neural network to compute the highlight score of each video segment. Extensive experimental results on the YouTube and SumMe datasets demonstrate that the proposed approach achieves significant improvement over state-of-the-art methods. With the proposed 3-D attention model, video highlights can be accurately retrieved in spatial and temporal dimensions without human supervision in several domains, such as gymnastics, parkour, skating, skiing, surfing, and dog activities, on the public datasets.

Constructing a Highlight Classifier with an Attention Based LSTM Neural Network

HighlightMe: Detecting Highlights from Human-Centric Videos

Video Highlight Detection Via Deep Ranking Modeling

Action Classification and Highlighting in Videos

Show Me What I Like: Detecting User-Specific Video Highlights Using Content-Based Multi-Head Attention

Adaptive Video Highlight Detection by Learning from User History

Video Highlight Prediction Using Audience Chat Reactions

Unsupervised Modality-Transferable Video Highlight Detection With Representation Activation Sequence Learning

Towards Extracting Highlights From Recorded Live Videos: An Implicit Crowdsourcing Approach

Highlight Detection With Pairwise Deep Ranking For First-Person Video Summarization

Three-Dimensional Attention-Based Deep Ranking Model for Video Highlight Detection

Collaborative Noisy Label Cleaner: Learning Scene-aware Trailers for Multi-modal Highlight Detection in Movies

Unsupervised Video Highlight Detection by Learning from Audio and Visual Recurrence

Unleash the Potential of CLIP for Video Highlight Detection

Videogenic: Identifying Highlight Moments in Videos with Professional Photographs as a Prior

VideoLights: Feature Refinement and Cross-Task Alignment Transformer for Joint Video Highlight Detection and Moment Retrieval

Video Highlight Detection Via Region-Based Deep Ranking Model

Unsupervised Extraction of Video Highlights Via Robust Recurrent Auto-encoders

Indirect Match Highlights Detection with Deep Convolutional Neural Networks

Less Is More: Learning Highlight Detection From Video Duration