Abstract:The proliferation of video content on platforms like YouTube and Vimeo presents significant challenges in efficiently locating relevant information. Automatic video summarization aims to address this by extracting and presenting key content in a condensed form. This thesis explores enhancing video summarization by integrating text-based queries and conditional modeling to tailor summaries to user needs. Traditional methods often produce fixed summaries that may not align with individual requirements. To overcome this, we propose a multi-modal deep learning approach that incorporates both textual queries and visual information, fusing them at different levels of the model architecture. Evaluation metrics such as accuracy and F1-score assess the quality of the generated summaries. The thesis also investigates improving text-based query representations using contextualized word embeddings and specialized attention networks. This enhances the semantic understanding of queries, leading to better video summaries. To emulate human-like summarization, which accounts for both visual coherence and abstract factors like storyline consistency, we introduce a conditional modeling approach. This method uses multiple random variables and joint distributions to capture key summarization components, resulting in more human-like and explainable summaries. Addressing data scarcity in fully supervised learning, the thesis proposes a segment-level pseudo-labeling approach. This self-supervised method generates additional data, improving model performance even with limited human-labeled datasets. In summary, this research aims to enhance automatic video summarization by incorporating text-based queries, improving query representations, introducing conditional modeling, and addressing data scarcity, thereby creating more effective and personalized video summaries.

Hierarchical Variational Network for User-Diversified & Query-Focused Video Summarization.

Learning User Interest with Improved Triplet Deep Ranking and Web-Image Priors for Topic-Related Video Summarization.

Memorable and Rich Video Summarization

Convolutional Hierarchical Attention Network for Query-Focused Video Summarization.

Hierarchical organization for medical video summarization using latent visual and semantic analysis

Update Summarization using a Multi-level Hierarchical Dirichlet Process Model.

An Unsupervised Video Summarization Method Based on Multimodal Representation.

DeepQAMVS: Query-Aware Hierarchical Pointer Networks for Multi-Video Summarization

Exploring global diverse attention via pairwise temporal relation for video summarization

Query-Biased Self-Attentive Network for Query-Focused Video Summarization

Feature fusion over hyperbolic graph convolution networks for video summarisation

Learning Multiscale Hierarchical Attention for Video Summarization

Deep Semantic and Attentive Network for Unsupervised Video Summarization

Query-Focused Video Summarization: Dataset, Evaluation, and A Memory Network Based Approach

Enhancing Video Summarization with Context Awareness

Dynamic graph convolutional network for multi-video summarization

Personalized Video Summarization using Text-Based Queries and Conditional Modeling

Hierarchical multi‐modal video summarization with dynamic sampling

A Hierarchical Spatial–Temporal Cross-Attention Scheme for Video Summarization Using Contrastive Learning

MHSCNet: A Multimodal Hierarchical Shot-aware Convolutional Network for Video Summarization

Query-controllable Video Summarization