GameIR: A Large-Scale Synthesized Ground-Truth Dataset for Image Restoration over Gaming Content

Lebin Zhou,Kun Han,Nam Ling,Wei Wang,Wei Jiang

2024-08-30

Abstract:Image restoration methods like super-resolution and image synthesis have been successfully used in commercial cloud gaming products like NVIDIA's DLSS. However, restoration over gaming content is not well studied by the general public. The discrepancy is mainly caused by the lack of ground-truth gaming training data that match the test cases. Due to the unique characteristics of gaming content, the common approach of generating pseudo training data by degrading the original HR images results in inferior restoration performance. In this work, we develop GameIR, a large-scale high-quality computer-synthesized ground-truth dataset to fill in the blanks, targeting at two different applications. The first is super-resolution with deferred rendering, to support the gaming solution of rendering and transferring LR images only and restoring HR images on the client side. We provide 19200 LR-HR paired ground-truth frames coming from 640 videos rendered at 720p and 1440p for this task. The second is novel view synthesis (NVS), to support the multiview gaming solution of rendering and transferring part of the multiview frames and generating the remaining frames on the client side. This task has 57,600 HR frames from 960 videos of 160 scenes with 6 camera views. In addition to the RGB frames, the GBuffers during the deferred rendering stage are also provided, which can be used to help restoration. Furthermore, we evaluate several SOTA super-resolution algorithms and NeRF-based NVS algorithms over our dataset, which demonstrates the effectiveness of our ground-truth GameIR data in improving restoration performance for gaming content. Also, we test the method of incorporating the GBuffers as additional input information for helping super-resolution and NVS. We release our dataset and models to the general public to facilitate research on restoration methods over gaming content.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to address the problem of achieving high-quality image restoration in cloud gaming content, particularly Super-Resolution (SR) and Novel View Synthesis (NVS). Specifically, the paper points out that current image restoration methods perform poorly when handling gaming content, mainly due to the lack of real gaming training data that matches the test scenarios. Gaming content has unique characteristics, such as clear and sharp low-resolution (LR) images, which make the method of generating pseudo-training data by downsampling high-resolution (HR) images unable to produce ideal restoration results. Therefore, the paper develops GameIR, a large-scale, high-quality computer-synthesized real dataset, aimed at filling this gap and supporting image restoration research for gaming content.

### Main Contributions:
1. **Development of the GameIR Dataset**: This dataset includes two main application datasets, namely GameIR-SR for super-resolution and GameIR-NVS for novel view synthesis. GameIR-SR contains 19,200 pairs of LR-HR paired real frames from 640 videos; GameIR-NVS contains 57,600 HR frames from 960 videos of 160 scenes, with 6 different camera views per scene.
2. **Evaluation of Existing Algorithms**: The paper evaluates several state-of-the-art super-resolution algorithms (such as Anime4K, RealESRGAN, AdaCode) and NeRF-based novel view synthesis algorithms (such as Instant-NGP, NeRFacto, DSNeRF, PyNeRF) on the GameIR dataset to provide benchmark performance, helping researchers understand the performance of existing methods on real gaming data.
3. **Exploration of Additional Information Utilization**: The paper further investigates how to use GBuffer (i.e., segmentation maps and depth maps from the deferred rendering stage) as additional input or generation conditions to improve the performance of super-resolution and novel view synthesis.

### Key Points of the Solution:
- **Importance of Real Data**: The paper emphasizes the importance of using real gaming data for training, as these data can better reflect the degradation characteristics in actual games, thereby improving the model's performance.
- **Utilization of GBuffer**: By incorporating GBuffer information (such as segmentation maps and depth maps) into the model, the quality of image restoration can be significantly improved, especially when dealing with complex gaming scenes.

### Experimental Results:
- **Super-Resolution**: On the GameIR-SR dataset, the fine-tuned models significantly outperform the pre-trained models on all metrics, particularly in restoring details and improving image clarity.
- **Novel View Synthesis**: On the GameIR-NVS dataset, PyNeRF performs excellently on all metrics, especially when using only front-facing views. Additionally, incorporating depth maps as supervision information can further enhance NVS performance.

Overall, this paper provides an important resource and reference for image restoration research in cloud gaming content by offering a large-scale real gaming dataset and detailed experimental evaluations.