Abstract:Human beings are endowed with a complementary learning system, which bridges the slow learning of general world dynamics with fast storage of episodic memory from a new experience. Previous video generation models, however, primarily focus on slow learning by pre-training on vast amounts of data, overlooking the fast learning phase crucial for episodic memory storage. This oversight leads to inconsistencies across temporally distant frames when generating longer videos, as these frames fall beyond the model's context window. To this end, we introduce SlowFast-VGen, a novel dual-speed learning system for action-driven long video generation. Our approach incorporates a masked conditional video diffusion model for the slow learning of world dynamics, alongside an inference-time fast learning strategy based on a temporal LoRA module. Specifically, the fast learning process updates its temporal LoRA parameters based on local inputs and outputs, thereby efficiently storing episodic memory in its parameters. We further propose a slow-fast learning loop algorithm that seamlessly integrates the inner fast learning loop into the outer slow learning loop, enabling the recall of prior multi-episode experiences for context-aware skill learning. To facilitate the slow learning of an approximate world model, we collect a large-scale dataset of 200k videos with language action annotations, covering a wide range of scenarios. Extensive experiments show that SlowFast-VGen outperforms baselines across various metrics for action-driven video generation, achieving an FVD score of 514 compared to 782, and maintaining consistency in longer videos, with an average of 0.37 scene cuts versus 0.89. The slow-fast learning loop algorithm significantly enhances performances on long-horizon planning tasks as well. Project Website: <a class="link-external link-https" href="https://slowfast-vgen.github.io" rel="external noopener nofollow">this https URL</a>

Video Generation with Learned Action Prior

Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge.

Action-conditioned video data improves predictability

Conditional Temporal Variational AutoEncoder for Action Video Prediction

Pose-guided Generative Adversarial Net for Novel View Action Synthesis

Deep Video Generation, Prediction and Completion of Human Action Sequences

Collaboratively Self-supervised Video Representation Learning for Action Recognition

Early Action Prediction with Generative Adversarial Networks

Learning Variational Motion Prior for Video-based Motion Capture

LaMD: Latent Motion Diffusion for Video Generation

Autoencoding Video Latents for Adversarial Video Generation

Learning a Generative Model for Multi‐Step Human‐Object Interactions from Videos

Explainable Video Action Reasoning via Prior Knowledge and State Transitions

Simple Video Generation using Neural ODEs

Video Action Understanding

Hierarchical Graph-Convolutional Variational AutoEncoding for Generative Modelling of Human Motion

Generating Action-conditioned Prompts for Open-vocabulary Video Action Recognition

SlowFast-VGen: Slow-Fast Learning for Action-Driven Long Video Generation

Action2video: Generating Videos of Human 3D Actions

Scaling Autoregressive Video Models

Collaborative Spatio-temporal Feature Learning for Video Action Recognition