Abstract:Predicting future scenes based on historical frames is challenging, especially when it comes to the complex uncertainty in nature. We observe that there is a divergence between spatial-temporal variations of active patterns and non-active patterns in a video, where these patterns constitute visual content and the former ones implicate more violent movement. This divergence enables active patterns the higher potential to act with more severe future uncertainty. Meanwhile, the existence of non-active patterns provides an opportunity for machines to examine some underlying rules with a mutual constraint between non-active patterns and active patterns. In order to solve this divergence, we provide a method called active patterns-perceived stochastic video prediction (ASVP) which allows active patterns to be perceived by neural networks during training. Our method starts with separating active patterns along with non-active ones from a video. Then, both scene-based prediction and active pattern-perceived prediction are conducted to respectively capture the variations within the whole scene and active patterns. Specially for active pattern-perceived prediction, a conditional generative adversarial network (CGAN) is exploited to model active patterns as conditions, with a variational autoencoder (VAE) for predicting the complex dynamics of active patterns. Additionally, a mutual constraint is designed to improve the learning procedure for the network to better understand underlying interacting rules among these patterns. Extensive experiments are conducted on both KTH human action and BAIR action-free robot pushing datasets with comparison to state-of-the-art works. Experimental results demonstrate the competitive performance of the proposed method as we expected. The released code and models are at https://github.com/tolearnmuch/ASVP.

ViPro: Enabling and Controlling Video Prediction for Complex Dynamical Scenarios using Procedural Knowledge

Guiding Video Prediction with Explicit Procedural Knowledge

VEDIT: Latent Prediction Architecture For Procedural Video Representation Learning

Probabilistic Future Prediction for Video Scene Understanding

Adaptive Hierarchical Motion-Focused Model for Video Prediction.

See, Plan, Predict: Language-guided Cognitive Planning with Video Prediction

VideoPro: A Visual Analytics Approach for Interactive Video Programming

Video Prediction by Modeling Videos as Continuous Multi-Dimensional Processes

Video Prediction via Example Guidance

Progressive Multi-granularity Analysis for Video Prediction.

ViP3D: End-to-end Visual Trajectory Prediction via 3D Agent Queries

Long-Term Prediction of Natural Video Sequences with Robust Video Predictors

Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction

Disentangling Propagation and Generation for Video Prediction

Zero-Shot Video Question Answering with Procedural Programs

Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representations

Active Patterns Perceived for Stochastic Video Prediction

Predicting Video with VQVAE

Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Prediction

Predicting Long-horizon Futures by Conditioning on Geometry and Time

Structure Preserving Video Prediction