Improving Lens Flare Removal with General Purpose Pipeline and Multiple Light Sources Recovery

Yuyan Zhou,Dong Liang,Songcan Chen,Sheng-Jun Huang,Shuo Yang,Chongyi Li
2023-08-31
Abstract:When taking images against strong light sources, the resulting images often contain heterogeneous flare artifacts. These artifacts can importantly affect image visual quality and downstream computer vision tasks. While collecting real data pairs of flare-corrupted/flare-free images for training flare removal models is challenging, current methods utilize the direct-add approach to synthesize data. However, these methods do not consider automatic exposure and tone mapping in image signal processing pipeline (ISP), leading to the limited generalization capability of deep models training using such data. Besides, existing methods struggle to handle multiple light sources due to the different sizes, shapes and illuminance of various light sources. In this paper, we propose a solution to improve the performance of lens flare removal by revisiting the ISP and remodeling the principle of automatic exposure in the synthesis pipeline and design a more reliable light sources recovery strategy. The new pipeline approaches realistic imaging by discriminating the local and global illumination through convex combination, avoiding global illumination shifting and local over-saturation. Our strategy for recovering multiple light sources convexly averages the input and output of the neural network based on illuminance levels, thereby avoiding the need for a hard threshold in identifying light sources. We also contribute a new flare removal testing dataset containing the flare-corrupted images captured by ten types of consumer electronics. The dataset facilitates the verification of the generalization capability of flare removal methods. Extensive experiments show that our solution can effectively improve the performance of lens flare removal and push the frontier toward more general situations.
Image and Video Processing,Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
### Problems the Paper Aims to Solve This paper aims to address two main issues in lens flare removal: 1. **Synthesizing More Realistic Flare-Contaminated Images**: Existing methods overlook the automatic exposure and tone mapping functions in the image signal processing (ISP) pipeline when synthesizing data, resulting in less realistic generated data and limiting the model's generalization ability. This paper proposes a new data synthesis pipeline that uses pixel-level convex combinations to avoid global illumination shifts and local oversaturation issues. 2. **Restoring Multiple Light Sources**: Existing methods perform poorly when dealing with multiple light sources because the sizes, shapes, and illuminance of different light sources vary. This paper proposes a convex averaging strategy based on illuminance levels, which can naturally restore multiple light sources without the need for hard thresholds to identify light sources. Additionally, the authors contribute a new test dataset containing real flare images captured by various consumer electronic products to validate the generalization performance of different methods. Experiments show that this method can effectively improve lens flare removal and performs well in a wider range of scenarios.