Reflection Removal Using Recurrent Polarization-to-Polarization Network

Wenjiao Bian,Yusuke Monno,Masatoshi Okutomi

2024-02-28

Abstract:This paper addresses reflection removal, which is the task of separating reflection components from a captured image and deriving the image with only transmission components. Considering that the existence of the reflection changes the polarization state of a scene, some existing methods have exploited polarized images for reflection removal. While these methods apply polarized images as the inputs, they predict the reflection and the transmission directly as non-polarized intensity images. In contrast, we propose a polarization-to-polarization approach that applies polarized images as the inputs and predicts "polarized" reflection and transmission images using two sequential networks to facilitate the separation task by utilizing the interrelated polarization information between the reflection and the transmission. We further adopt a recurrent framework, where the predicted reflection and transmission images are used to iteratively refine each other. Experimental results on a public dataset demonstrate that our method outperforms other state-of-the-art methods.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the problem of separating reflection and transmission components from captured images to obtain a clear image containing only the transmission component. Specifically, the authors propose a deep learning-based method that utilizes polarized images to achieve this goal.

### Background and Motivation
In daily life, due to the presence of semi-reflective materials (such as glass), images captured by cameras usually contain both reflection and transmission components. However, most computer vision applications (such as object detection, segmentation, and depth estimation) assume that each pixel value comes only from the transmission part of the scene. Therefore, reflection removal is an important task that can improve the robustness of practical applications.

### Limitations of Existing Methods
- **Single Image Methods**: Most existing reflection removal methods are based on a single grayscale or color image, with both input and output being images in the intensity domain. Although these methods have made some progress, the problem is ill-posed, meaning there are countless combinations of reflection and transmission images that can generate the same mixed image, so the separation effect is still not ideal.
- **Multi-view Methods**: Some methods attempt to use multi-view color images to solve this problem, but these methods usually require preprocessing steps for image alignment, limiting their practical application.
- **Polarized Image Methods**: In recent years, with the decreasing cost of one-time polarized cameras, obtaining polarized images has become increasingly easy. Some methods use images from different polarization directions to remove reflections, but these methods usually take polarized images as input and directly predict non-polarized reflection and transmission images.

### Proposed Method
The authors propose a new polarization-to-polarization method, which not only takes polarized images as input but also predicts polarized reflection and transmission images. Specifically:
- **Network Structure**: The method employs two sequential networks, the first network (R-LSTM-Net) is used to predict reflection, and the second network (T-Net) is used to predict transmission. These two networks refine each other through an iterative framework.
- **Input Features**: The network input includes images from four polarization directions (0°, 45°, 90°, 135°), as well as intensity maps, degree of polarization (DoP) maps, and polarization difference images (Idiff) of these images.
- **Loss Functions**: To ensure the accuracy of the predicted results, the authors designed three types of loss functions: pixel-level L1 loss (Lpixel), perceptual loss (Lpercep), and perceptual normalized cross-correlation loss (Lpncc).

### Experimental Results
Experimental results show that this method outperforms other existing intensity domain and polarization domain methods on public datasets. Particularly in reflection removal, this method achieves significant improvements.

### Conclusion
By leveraging the interrelationships of polarized images and combining a recursive framework, the proposed RP2PN method performs excellently in the reflection removal task, surpassing other existing methods.