FLAIR: A Conditional Diffusion Framework with Applications to Face Video Restoration

Zihao Zou,Jiaming Liu,Shirin Shoushtari,Yubo Wang,Weijie Gan,Ulugbek S. Kamilov
2023-11-27
Abstract:Face video restoration (FVR) is a challenging but important problem where one seeks to recover a perceptually realistic face videos from a low-quality input. While diffusion probabilistic models (DPMs) have been shown to achieve remarkable performance for face image restoration, they often fail to preserve temporally coherent, high-quality videos, compromising the fidelity of reconstructed faces. We present a new conditional diffusion framework called FLAIR for FVR. FLAIR ensures temporal consistency across frames in a computationally efficient fashion by converting a traditional image DPM into a video DPM. The proposed conversion uses a recurrent video refinement layer and a temporal self-attention at different scales. FLAIR also uses a conditional iterative refinement process to balance the perceptual and distortion quality during inference. This process consists of two key components: a data-consistency module that analytically ensures that the generated video precisely matches its degraded observation and a coarse-to-fine image enhancement module specifically for facial regions. Our extensive experiments show superiority of FLAIR over the current state-of-the-art (SOTA) for video super-resolution, deblurring, JPEG restoration, and space-time frame interpolation on two high-quality face video datasets.
Computer Vision and Pattern Recognition,Image and Video Processing
What problem does this paper attempt to address?
The problem that this paper attempts to solve is Face Video Restoration (FVR). Specifically, researchers hope to restore perceptually realistic and high - quality face videos from low - quality inputs. Although Diffusion Probabilistic Models (DPMs) perform well in face image restoration, they are usually difficult to maintain the temporal coherence and high quality of videos, which will damage the realism of the reconstructed faces. For this reason, the paper proposes a new conditional diffusion framework - FLAIR, which is specifically used for FVR. It aims to ensure the temporal consistency between frames by transforming the traditional image DPM into a video DPM, and balance the perceptual quality and distortion quality through a conditional iterative refinement process, thereby improving the effect of face video restoration.