Abstract:Video memorability measures the degree to which a video is remembered by different viewers and has shown great potential in various contexts, including advertising, education, and health care. While extensive research has been conducted on image memorability, the study of video memorability is still in its early stages. Existing methods in this field primarily focus on coarse-grained spatial feature representation and decision fusion strategies, overlooking the crucial interactions between spatial and temporal domains. Therefore, we propose an end-to-end collaborative spatial-temporal network called VMemNet, which incorporates targeted attention mechanisms and intermediation fusion strategies. This enables VMemNet to capture the intricate relationships between spatial and temporal information and uncover more elements of memorability within video visual features. VMemNet integrates spatially and semantically guided attention modules into a dual-stream network architecture, allowing it to simultaneously capture static local cues and dynamic global cues in videos. Specifically, the spatial attention module is used to aggregate more memorable elements from spatial locations, and the semantically guided attention module is used to achieve semantic alignment and intermediate fusion of the local and global cues. In addition, two types of loss functions with complementary decision rules are associated with the corresponding attention modules to guide the training process of the proposed network. Experimental results obtained on a publicly available dataset verify that the proposed VMemNet approach outperforms all current single- and multi-modal methods in terms of video memorability prediction.

Aesthetics-Assisted Multi-task Learning with Attention for Image Memorability Prediction.

Image Aesthetics Assessment With Attribute-Assisted Multimodal Memory Network

Media Memorability Prediction Based on Machine Learning

Confidence-Based Dynamic Cross-Modal Memory Network for Image Aesthetic Assessment

Image Aesthetic Assessment Based on Emotion-Assisted Multi-Task Learning Network

Beyond Vision: A Multimodal Recurrent Attention Convolutional Neural Network for Unified Image Aesthetic Prediction Tasks

AMNet: Memorability Estimation with Attention

Multi-Output Learning Based on Multimodal GCN and Co-Attention for Image Aesthetics and Emotion Analysis

Personality Driven Multi-task Learning for Image Aesthetic Assessment

Towards Unified Aesthetics and Emotion Prediction in Images.

VMemNet: A Deep Collaborative Spatial-Temporal Network with Attention Representation for Video Memorability Prediction

Multi-Task Convolutional Neural Network for Image Aesthetic Assessment

Personality-assisted Multi-task Learning for Generic and Personalized Image Aesthetics Assessment.

MSCAN: Multimodal Self-and-Collaborative Attention Network for Image Aesthetic Prediction Tasks

Deep Neural Framework with Visual Attention and Global Context for Predicting Image Aesthetics

Multimodal Image Aesthetic Prediction with Missing Modality

Attribute-assisted Multimodal Network for Image Aesthetics Assessment.

Unified Image Aesthetic and Emotional Prediction Based on Deep Multi-task Learning

Context-aware Attention Network for Predicting Image Aesthetic Subjectivity

Multi-view Attention with Memory Assistant for Image Captioning

Multi-Modal Memory Enhancement Attention Network for Image-Text Matching