Towards Real-world Video Face Restoration: A New Benchmark

Ziyan Chen,Jingwen He,Xinqi Lin,Yu Qiao,Chao Dong

2024-05-04

Abstract:Blind face restoration (BFR) on images has significantly progressed over the last several years, while real-world video face restoration (VFR), which is more challenging for more complex face motions such as moving gaze directions and facial orientations involved, remains unsolved. Typical BFR methods are evaluated on privately synthesized datasets or self-collected real-world low-quality face images, which are limited in their coverage of real-world video frames. In this work, we introduced new real-world datasets named FOS with a taxonomy of "Full, Occluded, and Side" faces from mainly video frames to study the applicability of current methods on videos. Compared with existing test datasets, FOS datasets cover more diverse degradations and involve face samples from more complex scenarios, which helps to revisit current face restoration approaches more comprehensively. Given the established datasets, we benchmarked both the state-of-the-art BFR methods and the video super resolution (VSR) methods to comprehensively study current approaches, identifying their potential and limitations in VFR tasks. In addition, we studied the effectiveness of the commonly used image quality assessment (IQA) metrics and face IQA (FIQA) metrics by leveraging a subjective user study. With extensive experimental results and detailed analysis provided, we gained insights from the successes and failures of both current BFR and VSR methods. These results also pose challenges to current face restoration approaches, which we hope stimulate future advances in VFR research.

Computer Vision and Pattern Recognition,Artificial Intelligence,Multimedia,Image and Video Processing

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

This paper attempts to address several key issues in Video Face Restoration (VFR):

1. **Lack of an Effective Evaluation System**: Currently, the task of video face restoration lacks an effective evaluation system. Existing evaluation metrics need to be validated and supplemented in video scenarios.
2. **Lack of Real-World Video Test Datasets**: Existing test datasets are mainly focused on images and lack the diversity of real-world videos, including different facial expressions, poses, and occlusions.
3. **Limitations of Existing Methods**: Although existing Blind Face Restoration (BFR) methods have made significant progress on images, they perform poorly in video face restoration tasks, especially in handling side faces, occluded faces, and maintaining inter-frame stability.

To address these issues, the authors introduce a new real-world dataset called FOS (Full, Occluded, and Side) and benchmark it to evaluate the potential and limitations of current state-of-the-art BFR and Video Super-Resolution (VSR) methods in video face restoration tasks. Additionally, the authors explore the effectiveness of commonly used Image Quality Assessment (IQA) and Face Image Quality Assessment (FIQA) metrics through subjective user studies and propose a new evaluation metric, VIDD, to measure the stability between video frames.