ShapeMoiré: Channel-Wise Shape-Guided Network for Image Demoiréing

Jinming Cao,Sicheng Shen,Qiu Zhou,Yifang Yin,Yangyan Li,Roger Zimmermann

2024-04-28

Abstract:Photographing optoelectronic displays often introduces unwanted moiré patterns due to analog signal interference between the pixel grids of the display and the camera sensor arrays. This work identifies two problems that are largely ignored by existing image demoiréing approaches: 1) moiré patterns vary across different channels (RGB); 2) repetitive patterns are constantly observed. However, employing conventional convolutional (CNN) layers cannot address these problems. Instead, this paper presents the use of our recently proposed Shape concept. It was originally employed to model consistent features from fragmented regions, particularly when identical or similar objects coexist in an RGB-D image. Interestingly, we find that the Shape information effectively captures the moiré patterns in artifact images. Motivated by this discovery, we propose a ShapeMoiré method to aid in image demoiréing. Beyond modeling shape features at the patch-level, we further extend this to the global image-level and design a novel Shape-Architecture. Consequently, our proposed method, equipped with both ShapeConv and Shape-Architecture, can be seamlessly integrated into existing approaches without introducing additional parameters or computation overhead during inference. We conduct extensive experiments on four widely used datasets, and the results demonstrate that our ShapeMoiré achieves state-of-the-art performance, particularly in terms of the PSNR metric. We then apply our method across four popular architectures to showcase its generalization capabilities. Moreover, our ShapeMoiré is robust and viable under real-world demoiréing scenarios involving smartphone photographs.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper primarily addresses two major issues in the process of image demoiréing:

1. **Variation of moiré patterns in different channels**: Existing methods often overlook the different patterns of moiré that appear in different RGB channels. This variability makes it difficult for traditional convolutional layers to handle effectively.
2. **Presence of repetitive patterns**: The paper points out that repetitive moiré patterns frequently appear in images, which existing methods often ignore.

To solve these problems, the authors propose a new method called ShapeMoiré. This method leverages the recently introduced concept of "Shape," which was initially used for modeling consistent features in RGB-D segmentation tasks. The ShapeMoiré method captures these features at both the local patch level and the global image level by introducing ShapeConv layers and a Shape-Architecture framework. Experimental results show that this method achieves state-of-the-art performance on multiple benchmark datasets and can be seamlessly integrated into existing network architectures without adding extra parameters or computational overhead. Additionally, the method demonstrates good performance in real-world scenarios.