Abstract:The proliferation of video content on platforms like YouTube and Vimeo presents significant challenges in efficiently locating relevant information. Automatic video summarization aims to address this by extracting and presenting key content in a condensed form. This thesis explores enhancing video summarization by integrating text-based queries and conditional modeling to tailor summaries to user needs. Traditional methods often produce fixed summaries that may not align with individual requirements. To overcome this, we propose a multi-modal deep learning approach that incorporates both textual queries and visual information, fusing them at different levels of the model architecture. Evaluation metrics such as accuracy and F1-score assess the quality of the generated summaries. The thesis also investigates improving text-based query representations using contextualized word embeddings and specialized attention networks. This enhances the semantic understanding of queries, leading to better video summaries. To emulate human-like summarization, which accounts for both visual coherence and abstract factors like storyline consistency, we introduce a conditional modeling approach. This method uses multiple random variables and joint distributions to capture key summarization components, resulting in more human-like and explainable summaries. Addressing data scarcity in fully supervised learning, the thesis proposes a segment-level pseudo-labeling approach. This self-supervised method generates additional data, improving model performance even with limited human-labeled datasets. In summary, this research aims to enhance automatic video summarization by incorporating text-based queries, improving query representations, introducing conditional modeling, and addressing data scarcity, thereby creating more effective and personalized video summaries.

Query-based video summarization with multi-label classification network

An Unsupervised Video Summarization Method Based on Multimodal Representation.

Learning User Interest with Improved Triplet Deep Ranking and Web-Image Priors for Topic-Related Video Summarization.

Creating Memorable Video Summaries That Satisfy the User's Intention for Taking the Videos.

Creating Personalized Video Summaries Via Semantic Event Detection

Memorable and Rich Video Summarization

A GAN Based Video Summarization Method with Representation Loss

Query-focused Multi-document Summarization: Combining a Novel Topic Model with Graph-based Semi-supervised Learning

Query-controllable Video Summarization

Query-Biased Self-Attentive Network for Query-Focused Video Summarization

Personalized Video Summarization using Text-Based Queries and Conditional Modeling

Query-Aware Sparse Coding for Multi-Video Summarization

Personalized Video Summarization by Multimodal Video Understanding

Multiple Pairwise Ranking Networks for Personalized Video Summarization

Query-Oriented Micro-Video Summarization

Convolutional Hierarchical Attention Network for Query-Focused Video Summarization.

Dynamic graph convolutional network for multi-video summarization

Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach

DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization

MHSCNet: A Multimodal Hierarchical Shot-aware Convolutional Network for Video Summarization

Multi-View Video Summarization