Learning an epipolar shift compensation for light field image super-resolution

Xinya Wang,Jiayi Ma,Peng Yi,Xin Tian,Junjun Jiang,Xiao-Ping Zhang

DOI: https://doi.org/10.1016/j.inffus.2021.10.005

IF: 18.6

2022-01-01

Information Fusion

Abstract:Light field imaging has drawn broad attention since the advent of practical light field capturing systems that facilitate a wide range of applications in computer vision. However, existing learning-based methods for improving the spatial resolution of light field images neglect the shifts in the sub-pixel domain that are widely used by super-resolution techniques, thus, fail in recovering rich high-frequency information. To fully exploit the shift information, our method attempts to learn an epipolar shift compensation for light field image super-resolution that allows the restored light field image to be angular coherent with the enhancement of spatial resolution. The proposed method first utilizes the rich surrounding views along some typical epipolar directions to explore the inter-view correlations. We then implement feature-level registration to capture accurate sub-pixel shifts of central view, which is constructed by the compensation module equipped with dynamic deformable convolution. Finally, the complementary information from different spatial directions is fused to provide high-frequency details for the target view. By taking each sub-aperture image as a central view, our method could be applied for light field images with any angular resolution. Extensive experiments on both synthetic and real scene datasets demonstrate the superiority of our method over the state-of-the-art qualitatively and quantitatively. Moreover, the proposed method shows good performance in preserving the inherent epipolar structures in light field images. Specifically, our LFESCN method outperforms the state-of-the-art method with about 0.7 dB (PSNR) on average.