RenDetNet: Weakly-supervised Shadow Detection with Shadow Caster Verification

Nikolina Kubiak,Elliot Wortman,Armin Mustafa,Graeme Phillipson,Stephen Jolly,Simon Hadfield

2024-08-30

Abstract:Existing shadow detection models struggle to differentiate dark image areas from shadows. In this paper, we tackle this issue by verifying that all detected shadows are real, i.e. they have paired shadow casters. We perform this step in a physically-accurate manner by differentiably re-rendering the scene and observing the changes stemming from carving out estimated shadow casters. Thanks to this approach, the RenDetNet proposed in this paper is the first learning-based shadow detection model whose supervisory signals can be computed in a self-supervised manner. The developed system compares favourably against recent models trained on our data. As part of this publication, we release our code on github.

Computer Vision and Pattern Recognition,Graphics

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is that existing shadow detection models have difficulty distinguishing dark areas in images from real shadows. Specifically, the author points out that current shadow detection methods are prone to misjudging dark objects or areas as shadows when facing complex scenes. This is because most of these models rely on manually - labeled data for supervised learning, and these data sets lack diversity and complex texture changes.

To overcome these problems, this paper proposes a new weakly - supervised learning framework - RenDetNet (Render and Detection Network), which improves the accuracy of shadow detection by verifying whether each detected shadow has a corresponding shadow caster (i.e., the source of the shadow). The core of this method lies in physically accurately re - rendering the estimated shadow caster and observing the resulting differences, so as to ensure that the existence of a shadow is only confirmed when the re - rendered image actually shows the corresponding shadow change.

### Summary of main contributions:
1. **Proposed a weakly - supervised deep - learning method**: This method can generate accurate shadow masks and caster masks corresponding to real shadows, including shadows in the external environment and self - shadows.
2. **Introduced a self - supervised data - set generation pipeline with caster awareness**: Training data can be generated without manual labeling.
3. **Experimental results show that the new model outperforms existing shadow detection methods on the newly proposed data set with casters**.

Through this method, RenDetNet not only improves the accuracy of shadow detection, but also reduces the dependence on a large amount of manually - labeled data, enabling the model to be applied in a wider range of scenarios.