UPDExplainer: an Interpretable Transformer-Based Framework for Urban Physical Disorder Detection Using Street View Imagery
Chuanbo Hu,Shan Jia,Fan Zhang,Changjiang Xiao,Mindi Ruan,Jacob Thrasher,Xin Li
DOI: https://doi.org/10.1016/j.isprsjprs.2023.08.017
IF: 12.7
2023-01-01
ISPRS Journal of Photogrammetry and Remote Sensing
Abstract:Urban Physical Disorder (UPD), such as old or abandoned buildings, broken sidewalks, litter, and graffiti, has a negative impact on residents’ quality of life. They can also increase crime rates, cause social disorder, and pose a public health risk. Currently, there is a lack of efficient and reliable methods for detecting and understanding UPD. To bridge this gap, we propose UPDExplainer, an interpretable transformer-based framework for UPD detection. We first develop a UPD detection model based on the Swin Transformer architecture, which leverages readily accessible street view images to learn discriminative representations. To provide clear and comprehensible evidence and analysis, we subsequently introduce a UPD factor identification and ranking module that combines visual explanation maps with semantic segmentation maps. This novel integrated approach enables us to identify the exact objects within street view images that are responsible for physical disorders and gain insight into the underlying causes. Experimental results on the re-annotated Place Pulse 2.0 dataset demonstrate promising detection performance of the proposed method, surpassing the state-of-the-art with an increase of 6.54% in Accuracy and 7.05% in F1 score. For a comprehensive evaluation of the method’s ranking performance, we provide both quantitative and qualitative results, highlighting the model’s precise identification of one, two, three, and four UPD factors, respectively. We also present a case study of detecting and ranking physical disorders in the southern region of downtown Los Angeles, California, to demonstrate the practicality and effectiveness of our framework.