Abstract:Predicting future frames using historical spatiotemporal data sequences is challenging and critical, and it is receiving a lot of attention these days from academic and industrial scholars. Most spatiotemporal predictive algorithms ignore the valuable backward reasoning ability and the disparate learning complexities among different layers and hence, cannot build good long-term dependencies and spatial correlations,resulting in suboptimal solutions. To address the aforementioned issues, we propose a two-stage coarse-to-fine spatiotemporal predictive model with bidirectional distillation and level-specific meta-adaptation(See More)in this paper, which includes a bidirectional distillation network(BDN) and a level-specific meta-adapter(LMA), to gain bidirectional multilevel reasoning. In the first stage, BDN concentrates on bidirectional dynamics modeling and coarsely constructs spatial correlations of different layers, while LMA is introduced in the second fine-tuning stage to refine the multilevel spatial correlations from a meta-learning perspective.In particular, BDN mimics the forward and backward reasoning abilities of humans in a distillation manner,which aids in the development of long-term dependencies. The LMA views learning of different layers as disparate but related tasks and guides the transfer of learning experiences among these tasks through learning complexities. Thus, each layer could be closer to its solutions and could extract more informative spatial correlations. By capturing the enhanced short-term spatial correlations and long-term temporal dependencies,the proposed model could extract adequate knowledge from sequential historical observations and accurately predict future frames whose backtracking preconditions are consistent with the historical sequence. Our work is general and robust enough to be integrated into most spatiotemporal predictive models without requiring additional computation or memory cost during inference. Extensive experiments on four widely used predictive learning benchmarks validated the proposed model's effectiveness in comparison to state-of-the-art approaches(e.g., 10.6% improvement of Mean Squared Error on the Moving MNIST dataset).

PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs

PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning

PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning

RSNN: Recurrent Spiking Neural Networks for Dynamic Spatial-Temporal Information Processing

ISA-PredRNN: an Improved Self-Attention PredRNN Network for Spatiotemporal Predictive Learning

Temporal Attention Unit: Towards Efficient Spatiotemporal Predictive Learning

Bi-PredRNN: An Enhanced PredRNN++ with a Bidirectional Network for Spatiotemporal Sequence Prediction

PredCNN: Predictive Learning with Cascade Convolutions

Wavelet-Driven Spatiotemporal Predictive Learning: Bridging Frequency and Time Variations

Revisiting the Temporal Modeling in Spatio-Temporal Predictive Learning under A Unified View

Z-Order Recurrent Neural Networks For Video Prediction

MS-RNN: A Flexible Multi-Scale Framework for Spatiotemporal Predictive Learning

Triplet Attention Transformer for Spatiotemporal Predictive Learning

PredFormer: Transformers Are Effective Spatial-Temporal Predictive Learners

SeeMore: a Spatiotemporal Predictive Model with Bidirectional Distillation and Level-Specific Meta-Adaptation

Modernn: towards fine-grained motion details for spatiotemporal predictive learning

PSRUNet: a recurrent neural network for spatiotemporal sequence forecasting based on parallel simple recurrent unit

Enhancing spatiotemporal predictive learning: an approach with nested attention module

VMRNN: Integrating Vision Mamba and LSTM for Efficient and Accurate Spatiotemporal Forecasting

MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions

ModeRNN: Harnessing Spatiotemporal Mode Collapse in Unsupervised Predictive Learning