Abstract:Learning a robust video Variational Autoencoder (VAE) is essential for reducing video redundancy and facilitating efficient video generation. Directly applying image VAEs to individual frames in isolation can result in temporal inconsistencies and suboptimal compression rates due to a lack of temporal compression. Existing Video VAEs have begun to address temporal compression; however, they often suffer from inadequate reconstruction performance. In this paper, we present a novel and powerful video autoencoder capable of high-fidelity video encoding. First, we observe that entangling spatial and temporal compression by merely extending the image VAE to a 3D VAE can introduce motion blur and detail distortion artifacts. Thus, we propose temporal-aware spatial compression to better encode and decode the spatial information. Additionally, we integrate a lightweight motion compression model for further temporal compression. Second, we propose to leverage the textual information inherent in text-to-video datasets and incorporate text guidance into our model. This significantly enhances reconstruction quality, particularly in terms of detail preservation and temporal stability. Third, we further improve the versatility of our model through joint training on both images and videos, which not only enhances reconstruction quality but also enables the model to perform both image and video autoencoding. Extensive evaluations against strong recent baselines demonstrate the superior performance of our method. The project website can be found at~\href{<a class="link-external link-https" href="https://yzxing87.github.io/vae/" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://yzxing87.github.io/vae/" rel="external noopener nofollow">this https URL</a>}.

Video Representation Learning with Joint-Embedding Predictive Architectures

Revisiting Feature Prediction for Learning Visual Representations from Video

Predicting Video with VQVAE

Variance-Covariance Regularization Improves Representation Learning

Nonparametric Variational Auto-encoders for Hierarchical Representation Learning.

Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

Video Prediction Models as General Visual Encoders

Large Motion Video Autoencoding with Cross-modal Video VAE

Continual Learning of Predictive Models in Video Sequences via Variational Autoencoders

Probabilistic Representations for Video Contrastive Learning

3D-JEPA: A Joint Embedding Predictive Architecture for 3D Self-Supervised Representation Learning

Improved Video VAE for Latent Video Diffusion Model

Fine-grained Key-Value Memory Enhanced Predictor for Video Representation Learning

Temporal Enhanced Hybrid Neural Representation for Video Compression

Unsupervised Learning From Video With Deep Neural Embeddings

V-VIPE: Variational View Invariant Pose Embedding

Visual Representation Learning with Stochastic Frame Prediction

Memory-augmented Dense Predictive Coding for Video Representation Learning

Deep video representation learning: a survey

Dynamical Textures Modeling Via Joint Video Dictionary Learning