Abstract:Summarization tasks aim to summarize multiple pieces of information into a short description or representative information. A text summarization task summarizes textual information into a short description, whereas an image collection summarization task summarizes an image collection into images or textual representation in which the challenge is to understand the relationship between images. In recent years, scene-graph generation has shown the advantage of describing the visual contexts of a single-image, and incorporating external knowledge into the scene-graph generation model has also given effective directions for unseen single-image scene-graph generation. While external knowledge has been implemented in related work, it is still challenging to use this information efficiently for relationship estimation during the summarization. Following this trend, in this paper, we propose a novel scene-graph-based image-collection summarization model that aims to generate a summarized scene-graph of an image collection. The key idea of the proposed method is to enhance the relation predictor toward relationships between images in an image collection incorporating knowledge graphs as external knowledge for training a model. With this approach, we build an end-to-end framework that can generate a summarized scene graph of an image collection. To evaluate the proposed method, we also build an extended annotated MS-COCO dataset for this task and introduce an evaluation process that focuses on estimating the similarity between a summarized scene graph and ground-truth scene graphs. Traditional evaluation focuses on calculating precision and recall scores, which involve true positive predictions without balancing precision and recall. Meanwhile, the proposed evaluation process focuses on calculating the F-score of the similarity between a summarized scene graph and ground-truth scene graphs, which aims to balance both false positives and false negatives. Experimental results show that using external knowledge to enhance the relation predictor achieves better results than existing methods.

A Joint Optimization Model for Image Summarization Based on Image Content and Tags.

Learning User Interest with Improved Triplet Deep Ranking and Web-Image Priors for Topic-Related Video Summarization.

Memorable and Rich Video Summarization

Creating Memorable Video Summaries That Satisfy the User's Intention for Taking the Videos.

A Novel Compact Yet Rich Key Frame Creation Method for Compressed Video Summarization

A GAN Based Video Summarization Method with Representation Loss

An Unsupervised Video Summarization Method Based on Multimodal Representation.

A Topic-aware Summarization Framework with Different Modal Side Information

Bridging the Semantic Gap Between Image Contents and Tags

CFSum: A Coarse-to-Fine Contribution Network for Multimodal Summarization

Perceptual Attributes Optimization For Multivideo Summarization

Image-Collection Summarization Using Scene-Graph Generation With External Knowledge

Multi-Image Summarization: Textual Summary from a Set of Cohesive Images

Conditional Modeling Based Automatic Video Summarization

Multi-Modal Summary Generation using Multi-Objective Optimization

Multi-Document Summarization Based On Two-Level Sparse Representation Model

SummScore: A Comprehensive Evaluation Metric for Summary Quality Based on Cross-Encoder

ImageHive: Interactive Content-Aware Image Summarization

Image Tag Refinement Towards Low-Rank, Content-Tag Prior and Error Sparsity.

Video Joint Modelling Based on Hierarchical Transformer for Co-summarization

Semantics-Consistent Cross-domain Summarization via Optimal Transport Alignment