Abstract:The explosive growth of video data constitutes a series of new challenges in computer vision, and the function of video summarization (VS) is becoming more and more prominent. Recent works have shown the effectiveness of sparse dictionary selection (SDS) based VS, which selects a representative frame set to sufficiently reconstruct a given video. Existing SDS based VS methods use conventional handcrafted features or single-scale deep features, which could diminish their summarization performance due to the underutilization of frame feature representation. Deep learning techniques based on convolutional neural networks (CNNs) exhibit powerful capabilities among various vision tasks, as the CNN provides excellent feature representation. Therefore, in this paper, a multi-scale deep feature fusion based sparse dictionary selection (MSDFF-SDS) is proposed for VS. Specifically, multi-scale features include the directly extracted features from the last fully connected layer and the global average pooling (GAP) processed features from intermediate layers, then VS is formulated as a problem of minimizing the reconstruction error using the multi-scale deep feature fusion. In our formulation, the contribution of each scale of features can be adjusted by a balance parameter, and the row-sparsity consistency of the simultaneous reconstruction coefficient is used to select as few keyframes as possible. The resulting MSDFF-SDS model is solved by using an efficient greedy pursuit algorithm. Experimental results on two benchmark datasets demonstrate that the proposed MSDFF-SDS improves the F-score of keyframe based summarization more than 3% compared with the existing SDS methods, and performs better than most deep-learning methods for skimming based summarization.

Effective Video Summarization Using Channel Attention-Assisted Encoder–Decoder Framework

Creating Memorable Video Summaries That Satisfy the User's Intention for Taking the Videos.

Memorable and Rich Video Summarization

Creating Personalized Video Summaries Via Semantic Event Detection

A GAN Based Video Summarization Method with Representation Loss

An Unsupervised Video Summarization Method Based on Multimodal Representation.

Video Summarization with Attention-Based Encoder-Decoder Networks.

Video Summarization Via Weighted Neighborhood Based Representation.

Deep Multi-Scale Pyramidal Features Network for Supervised Video Summarization.

Video Summarization Techniques: A Comprehensive Review

Video Summarization Via Simultaneous Block Sparse Representation.

Video Summarization with a Dual-Path Attentive Network

Deep Semantic and Attentive Network for Unsupervised Video Summarization

Video Summarization Via Block Sparse Dictionary Selection

Unsupervised Video Summarization via Multi-source Features

Robust Video Summarization Using Collaborative Representation of Adjacent Frames

Unsupervised Video Summarization Based on An Encoder-Decoder Architecture

Supervised Video Summarization via Multiple Feature Sets with Parallel Attention

A Novel Visual Assisted Technique for Video Summarization Using Attention-Based Key-Frames Selection

Efficient video summarization through MobileNetSSD: a robust deep learning-based framework for efficient video summarization focused on objects of interest

Multi-scale Deep Feature Fusion Based Sparse Dictionary Selection for Video Summarization