Abstract:Due to a variety of motions across different frames, it is highly challenging to learn an effective spatiotemporal representation for accurate video saliency prediction (VSP). To address this issue, we develop an effective spatiotemporal feature alignment network tailored to VSP, mainly including two key sub-networks: a multi-scale deformable convolutional alignment network (MDAN) and a bidirectional convolutional Long Short-Term Memory (Bi-ConvLSTM) network. The MDAN learns to align the features of the neighboring frames to the reference one in a coarse-to-fine manner, which can well handle various motions. Specifically, the MDAN owns a pyramidal feature hierarchy structure that first leverages deformable convolution (Dconv) to align the lower-resolution features across frames, and then aggregates the aligned features to align the higher-resolution features, progressively enhancing the features from top to bottom. The output of MDAN is then fed into the Bi-ConvLSTM for further enhancement, which captures the useful long-time temporal information along forward and backward timing directions to effectively guide attention orientation shift prediction under complex scene transformation. Finally, the enhanced features are decoded to generate the predicted saliency map. The proposed model is trained end-to-end without any intricate post processing. Extensive evaluations on four VSP benchmark datasets demonstrate that the proposed method achieves favorable performance against state-of-the-art methods. The source codes and all the results will be released.

On the Consensus of Synchronous Temporal and Spatial Views: A Novel Multimodal Deep Learning Method for Social Video Prediction

SimVP: Simpler Yet Better Video Prediction

Going Beyond Multi-Task Dense Prediction with Synergy Embedding Models

MVSAS: Semantic-Aware Scheduling for Low Latency and High Precision in Wireless Multi-View Application.

SimVP: Towards Simple yet Powerful Spatiotemporal Predictive Learning

Predicting 360° Video Saliency: A ConvLSTM Encoder-Decoder Network with Spatio-temporal Consistency

Video Saliency Prediction Using Enhanced Spatiotemporal Alignment Network

From Single to Multiple: Leveraging Multi-level Prediction Spaces for Video Forecasting

Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification

Beyond Short-Term Snippet: Video Relation Detection With Spatio-Temporal Global Context

Pair-wise Layer Attention with Spatial Masking for Video Prediction

A Multi-Modal States based Vehicle Descriptor and Dilated Convolutional Social Pooling for Vehicle Trajectory Prediction

VLP2MSA: Expanding Vision-Language Pre-Training to Multimodal Sentiment Analysis

SDAPNet: End-to-End Multi-task Simultaneous Detection and Prediction Network.

Hybrid Attention Spatial-Temporal Network for Video Saliency Prediction

Visual Spatio-temporal Relation-enhanced Network for Cross-modal Text-Video Retrieval

A Spherical Convolution Approach for Learning Long Term Viewport Prediction in 360 Immersive Video

MMVP: Motion-Matrix-based Video Prediction

ASTM - an Attention Based Spatiotemporal Model for Video Prediction Using 3D Convolutional Neural Networks.

Multi-view Commercial Hotness Prediction Using Context-aware Neural Network Ensemble

Scene-Dependent Prediction in Latent Space for Video Anomaly Detection and Anticipation