Abstract:The proliferation of video content on platforms like YouTube and Vimeo presents significant challenges in efficiently locating relevant information. Automatic video summarization aims to address this by extracting and presenting key content in a condensed form. This thesis explores enhancing video summarization by integrating text-based queries and conditional modeling to tailor summaries to user needs. Traditional methods often produce fixed summaries that may not align with individual requirements. To overcome this, we propose a multi-modal deep learning approach that incorporates both textual queries and visual information, fusing them at different levels of the model architecture. Evaluation metrics such as accuracy and F1-score assess the quality of the generated summaries. The thesis also investigates improving text-based query representations using contextualized word embeddings and specialized attention networks. This enhances the semantic understanding of queries, leading to better video summaries. To emulate human-like summarization, which accounts for both visual coherence and abstract factors like storyline consistency, we introduce a conditional modeling approach. This method uses multiple random variables and joint distributions to capture key summarization components, resulting in more human-like and explainable summaries. Addressing data scarcity in fully supervised learning, the thesis proposes a segment-level pseudo-labeling approach. This self-supervised method generates additional data, improving model performance even with limited human-labeled datasets. In summary, this research aims to enhance automatic video summarization by incorporating text-based queries, improving query representations, introducing conditional modeling, and addressing data scarcity, thereby creating more effective and personalized video summaries.

Multiple Pairwise Ranking Networks for Personalized Video Summarization

Learning User Interest with Improved Triplet Deep Ranking and Web-Image Priors for Topic-Related Video Summarization.

Creating Personalized Video Summaries Via Semantic Event Detection

An Unsupervised Video Summarization Method Based on Multimodal Representation.

A Human-Machine Collaborative Video Summarization Framework Using Pupillary Response Signals

An Interactive Personalized Video Summarization Based on Sketches.

A GAN Based Video Summarization Method with Representation Loss

Personalized Video Summarization by Multimodal Video Understanding

User-Ranking Video Summarization with Multi-Stage Spatio-Temporal Representation.

Personalized Video Summarization using Text-Based Queries and Conditional Modeling

Query-based video summarization with multi-label classification network

Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach

Conditional Modeling Based Automatic Video Summarization

Highlight Detection With Pairwise Deep Ranking For First-Person Video Summarization

Multi-View Video Summarization

DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization

Video Summarization Based on User Log Enhanced Link Analysis

Unsupervised Video Summarization via Multi-source Features

Supervised Video Summarization via Multiple Feature Sets with Parallel Attention

Multi-View Metric Learning for Multi-View Video Summarization

Dynamic graph convolutional network for multi-video summarization