HDR Video Reconstruction with a Large Dynamic Dataset in Raw and sRGB Domains

Huanjing Yue,Yubo Peng,Biting Yu,Xuanwu Yin,Zhenyu Zhou,Jingyu Yang

2023-04-12

Abstract:High dynamic range (HDR) video reconstruction is attracting more and more attention due to the superior visual quality compared with those of low dynamic range (LDR) videos. The availability of LDR-HDR training pairs is essential for the HDR reconstruction quality. However, there are still no real LDR-HDR pairs for dynamic scenes due to the difficulty in capturing LDR-HDR frames simultaneously. In this work, we propose to utilize a staggered sensor to capture two alternate exposure images simultaneously, which are then fused into an HDR frame in both raw and sRGB domains. In this way, we build a large scale LDR-HDR video dataset with 85 scenes and each scene contains 60 frames. Based on this dataset, we further propose a Raw-HDRNet, which utilizes the raw LDR frames as inputs. We propose a pyramid flow-guided deformation convolution to align neighboring frames. Experimental results demonstrate that 1) the proposed dataset can improve the HDR reconstruction performance on real scenes for three benchmark networks; 2) Compared with sRGB inputs, utilizing raw inputs can further improve the reconstruction quality and our proposed Raw-HDRNet is a strong baseline for raw HDR reconstruction. Our dataset and code will be released after the acceptance of this paper.

Image and Video Processing,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper primarily addresses the issue of high dynamic range (HDR) video reconstruction, specifically focusing on the following aspects:

1. **Dataset Problem**:
- There is currently a lack of real-world LDR-HDR video datasets. Existing datasets are usually generated through synthetic methods, which results in models that perform poorly when handling real-world scenarios.
- To overcome this problem, the authors propose a new method to capture real-world LDR-HDR video pairs and construct a large-scale real LDR-HDR video dataset.

2. **Alignment and Fusion Problem**:
- During the HDR video reconstruction process, significant exposure differences between adjacent frames can introduce severe ghosting artifacts during the alignment and fusion process.
- To address this, the authors propose a Pyramid Flow-Guided Deformable Convolution (PFD) alignment module to improve alignment accuracy and reduce ghosting artifacts.

3. **Advantages of Using RAW Format Input**:
- Compared to the traditional sRGB format, the RAW format has a higher bit depth, which can better preserve detail information.
- Based on this observation, the authors propose an HDR video reconstruction network specifically designed for RAW format input (Raw-HDRNet) and validate its superiority through experiments.

Through the above methods, this paper not only introduces a new real LDR-HDR video dataset but also develops a powerful benchmark model that significantly enhances the quality of HDR video reconstruction.