See SIFT in a Rain

Wei Wu,Hao Chang,Zhu Li

2023-11-01

Abstract:Rain streaks bring complicated pixel intensity changes and additional gradients, greatly obstructing the extraction of image features from background. This causes serious performance degradation in feature-based applications. Thus, it is critical to remove rain streaks from a single rainy image to recover image features. Recently, many excellent image deraining methods have made remarkable progress. However, these human visual system-driven approaches mainly focus on improving image quality with pixel recovery as loss function, and neglect how to enhance image feature recovery ability. To address this issue, we propose a task-driven image deraining algorithm to strengthen image feature supply for subsequent feature-based applications. Due to the extensive use and strong practicability of Scale-Invariant Feature Transform (SIFT), we first propose two separate networks using distinct losses and modules to achieve two goals, respectively. One is difference of Gaussian (DoG) pyramid recovery network (DPRNet) for SIFT detection, and the other gradients of Gaussian images recovery network (GGIRNet) for SIFT description. Second, in the DPRNet we propose an alternative interest point loss that directly penalizes scale response extrema to recover the DoG pyramid. Third, we advance a gradient attention module in the GGIRNet to recover those gradients of Gaussian images. Finally, with the recovered DoG pyramid and gradients, we can regain SIFT key points. This divide-and-conquer scheme to set different objectives for SIFT detection and description leads to good robustness. Compared with state-of-the-art methods, experimental results demonstrate that our proposed algorithm achieves better performance in both the number of recovered SIFT key points and their accuracy.

Image and Video Processing

What problem does this paper attempt to address?

The paper attempts to address the problem of removing rain streaks from a single rainy image to restore image features, particularly Scale-Invariant Feature Transform (SIFT) keypoints. Specifically, the paper points out that many current de-raining methods mainly focus on improving image quality while neglecting how to enhance the ability to restore image features. Therefore, the authors propose a task-driven de-raining algorithm aimed at recovering as many SIFT keypoints as possible from the de-rained image, thereby improving the performance of subsequent feature-based applications.

The authors believe that traditional de-raining methods are mostly based on the Human Visual System (HVS), focusing on improving the objective and subjective quality of the image but failing to fully consider the needs of feature extraction. To this end, they propose a new framework consisting of two independent networks: the Difference of Gaussian Pyramid Restoration Network (DPRNet) for SIFT detection and the Gaussian Gradient Image Restoration Network (GGIRNet) for SIFT description. Additionally, they design an Alternative Interest Point Loss function (ALP loss) and a Gradient Attention Module (GAM) to better restore SIFT features. Experimental results show that this method outperforms existing state-of-the-art methods in terms of the number and accuracy of recovered SIFT keypoints.