Parallelized Autoregressive Visual Generation

Yuqing Wang,Shuhuai Ren,Zhijie Lin,Yujin Han,Haoyuan Guo,Zhenheng Yang,Difan Zou,Jiashi Feng,Xihui Liu

2024-12-20

Abstract:Autoregressive models have emerged as a powerful approach for visual generation but suffer from slow inference speed due to their sequential token-by-token prediction process. In this paper, we propose a simple yet effective approach for parallelized autoregressive visual generation that improves generation efficiency while preserving the advantages of autoregressive modeling. Our key insight is that parallel generation depends on visual token dependencies-tokens with weak dependencies can be generated in parallel, while strongly dependent adjacent tokens are difficult to generate together, as their independent sampling may lead to inconsistencies. Based on this observation, we develop a parallel generation strategy that generates distant tokens with weak dependencies in parallel while maintaining sequential generation for strongly dependent local tokens. Our approach can be seamlessly integrated into standard autoregressive models without modifying the architecture or tokenizer. Experiments on ImageNet and UCF-101 demonstrate that our method achieves a 3.6x speedup with comparable quality and up to 9.5x speedup with minimal quality degradation across both image and video generation tasks. We hope this work will inspire future research in efficient visual generation and unified autoregressive modeling. Project page: <a class="link-external link-https" href="https://epiphqny.github.io/PAR-project" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the slow speed of autoregressive visual generation models during the generation process. Specifically, traditional autoregressive models generate images or videos by predicting visual tokens one by one, which results in a very slow generation process and limits their efficiency in practical applications.

### Core problems of the paper:
1. **Slow generation speed**: Since autoregressive models need to predict tokens sequentially one by one, the generation process is very time - consuming, especially when dealing with high - resolution images or videos.
2. **Maintaining generation quality**: While increasing the generation speed, how to ensure that the quality of the generated images or videos does not significantly decline.

### Overview of the solution:
To solve the above problems, the author proposes a parallelized autoregressive visual generation method, which can significantly improve the generation speed without modifying the existing model architecture while maintaining the generation quality. Specifically:

- **Parallel generation strategy**: By identifying and grouping weakly - dependent visual tokens for parallel prediction, while maintaining the sequential generation of strongly - dependent local tokens.
- **Non - local parallel generation**: Divide the image into multiple regions, first generate the initial tokens of each region sequentially to establish the global structure, and then generate subsequent tokens in parallel between different regions.
- **Maintaining autoregressive characteristics**: Through appropriate organization and attention mechanisms, ensure that each prediction step can access all previously generated tokens, thereby maintaining the characteristics of the autoregressive model.

### Key innovation points:
1. **Parallel generation of weakly - dependent tokens**: By performing parallel prediction on weakly - dependent tokens, the number of generation steps is reduced, thus accelerating the generation process.
2. **Combination of local and non - local**: Maintain sequential generation within local regions to maintain high - quality local details, and perform parallel generation between different regions to improve efficiency.
3. **Seamless integration**: This method can be directly applied to existing autoregressive models without significant modification to the model architecture or tokenizer.

Through these improvements, the paper shows the experimental results on the ImageNet and UCF - 101 datasets, proving that this method can achieve a significant speed increase in image and video generation tasks while maintaining a relatively high generation quality.