LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior

Hanyu Wang,Saksham Suri,Yixuan Ren,Hao Chen,Abhinav Shrivastava

2024-10-29

Abstract:We present LARP, a novel video tokenizer designed to overcome limitations in current video tokenization methods for autoregressive (AR) generative models. Unlike traditional patchwise tokenizers that directly encode local visual patches into discrete tokens, LARP introduces a holistic tokenization scheme that gathers information from the visual content using a set of learned holistic queries. This design allows LARP to capture more global and semantic representations, rather than being limited to local patch-level information. Furthermore, it offers flexibility by supporting an arbitrary number of discrete tokens, enabling adaptive and efficient tokenization based on the specific requirements of the task. To align the discrete token space with downstream AR generation tasks, LARP integrates a lightweight AR transformer as a training-time prior model that predicts the next token on its discrete latent space. By incorporating the prior model during training, LARP learns a latent space that is not only optimized for video reconstruction but is also structured in a way that is more conducive to autoregressive generation. Moreover, this process defines a sequential order for the discrete tokens, progressively pushing them toward an optimal configuration during training, ensuring smoother and more accurate AR generation at inference time. Comprehensive experiments demonstrate LARP's strong performance, achieving state-of-the-art FVD on the UCF101 class-conditional video generation benchmark. LARP enhances the compatibility of AR models with videos and opens up the potential to build unified high-fidelity multimodal large language models (MLLMs).

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper attempts to address the limitations of existing video tokenization methods in video generation tasks, particularly their suboptimal performance in autoregressive (AR) generation models. Specifically:

1. **Limitations of Existing Video Tokenization Methods**:
- **Localized Features**: Traditional patch-based tokenization methods directly encode local regions of the input video into discrete tokens, which limits the tokenizer's ability to capture global and semantic information.
- **Serialization Challenges**: In autoregressive models, it is necessary to convert locally encoded tokens into a 1-dimensional sequence, but current methods lack clear guidance on determining the optimal serialization order.
- **Gap Between Reconstruction and Generation Performance**: Although the reconstruction quality of the tokenizer sets an upper limit on the generation quality, the gap between the two and its influencing factors remain unclear.

2. **Proposed Solutions in the Paper**:
- **Introduction of LARP**: LARP (Learned AutoRegressive Prior) is a novel video tokenizer that captures global information of the video by introducing a set of learned holistic queries, thereby generating more semantic tokens.
- **Autoregressive Generation Prior Model**: LARP integrates a lightweight autoregressive Transformer as a prior model during training to optimize the tokenizers' latent space, making it more suitable for autoregressive generation tasks.
- **Flexible Token Quantity**: LARP supports an arbitrary number of discrete tokens, which can be adjusted according to the specific needs of the task, enhancing the flexibility and efficiency of tokenization.

Through these designs, LARP aims to improve the quality of video generation, achieving higher generation fidelity under different token sequence lengths, and has achieved state-of-the-art FVD scores on the UCF101 class-conditional video generation benchmark.