Towards Flexible Interactive Reflection Removal with Human Guidance

Xiao Chen,Xudong Jiang,Yunkang Tao,Zhen Lei,Qing Li,Chenyang Lei,Zhaoxiang Zhang

2024-06-04

Abstract:Single image reflection removal is inherently ambiguous, as both the reflection and transmission components requiring separation may follow natural image statistics. Existing methods attempt to address the issue by using various types of low-level and physics-based cues as sources of reflection signals. However, these cues are not universally applicable, since they are only observable in specific capture scenarios. This leads to a significant performance drop when test images do not align with their assumptions. In this paper, we aim to explore a novel flexible interactive reflection removal approach that leverages various forms of sparse human guidance, such as points and bounding boxes, as auxiliary high-level prior to achieve robust reflection removal. However, incorporating the raw user guidance naively into the existing reflection removal network does not result in performance gains. To this end, we innovatively transform raw user input into a unified form -- reflection masks using an Interactive Segmentation Foundation Model. Such a design absorbs the quintessence of the foundational segmentation model and flexible human guidance, thereby mitigating the challenges of reflection separations. Furthermore, to fully utilize user guidance and reduce user annotation costs, we design a mask-guided reflection removal network, comprising our proposed self-adaptive prompt block. This block adaptively incorporates user guidance as anchors and refines transmission features via cross-attention mechanisms. Extensive results on real-world images validate that our method demonstrates state-of-the-art performance on various datasets with the help of flexible and sparse user guidance. Our code and dataset will be publicly available here <a class="link-external link-https" href="https://github.com/ShawnChenn/FlexibleReflectionRemoval" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the problem of Single Image Reflection Removal (SIRR). Specifically:

1. **Limitations of Existing Methods**:
- Single image reflection removal is inherently a blurry and challenging problem because reflection and transmission components may follow similar natural image statistical features.
- Existing methods attempt to solve this problem by using various low-level and physics-based cues, but these cues are only effective in specific scenarios and perform poorly in others.

2. **Introducing Human Guidance**:
- This paper proposes a novel approach that utilizes flexible human guidance (such as points and bounding boxes) as auxiliary high-level prior knowledge to achieve more robust reflection removal.
- The authors found that directly inputting raw user guidance into existing reflection removal networks does not enhance performance. Therefore, they innovatively designed an interactive segmentation-based model that converts raw user input into a unified form—a reflection mask.
- To fully leverage user guidance and reduce user annotation costs, the authors designed a mask-guided reflection removal network that includes the proposed adaptive prompt block. This block can adaptively use user guidance as anchors and refine transmission features through a cross-attention mechanism.

3. **Main Contributions**:
- Proposed a robust reflection identification method based on sparse human guidance, extending the SAM model to the image reflection removal task for the first time, addressing "out-of-assumption" reflection issues that traditional fully automatic methods cannot handle.
- Reduced user annotation costs from 50 inputs per image to only 3-4 user inputs.
- Contributed an open-source interactive image reflection removal dataset, containing two forms of user input (points, boxes) as additional reflection signals.

Through the above methods, the paper aims to improve the accuracy and robustness of reflection removal, especially in cases with strong reflections.