Abstract:This project uses machine learning and computer vision techniques and a novel interactive visualization tool to provide street-level characterization of urban spaces such as safety and maintenance in urban neighborhoods. This is achieved by collecting and annotating street-view images, extracting objective metrics through computer vision techniques, and using crowdsourcing to statistically model the perception of subjective metrics such as safety and maintenance. For modeling human perception and scaling it up with a predictive algorithm, we evaluate perception predictions across two points in time separated by economic changes in the urban core of Raleigh, North Carolina, in the aftermath of the 2008 Great Recession. We hypothesize specific socioeconomic processes can be substantially reflected in the built environment of cities and, thus, render themselves visible at the street level. This paper describes the process of incorporating subjective visual ratings across two datasets of temporally separated street-view images, an algorithm, and a visualization tool. This work serves as a case study for utilizing AI and visualization techniques in a richer characterization of urban spaces that includes both objective metrics such as income (that operates at a broader scale) and subjective metrics such as perception of individuals (that operates at a narrower scale at specific locations). We outline an interdisciplinary methodology to test this hypothesis in streetscape data from Raleigh, NC, from 2008 to 2020. We describe the results of training algorithms that utilized image features with crowdsourced human perception ratings. We provide a comparison of the results with income data. The analysis and interpretation of this comparison provide insight into the challenges and opportunities for using AI technology in characterizing changes in urban environments. One challenge is the ability of human domain experts to interpret the output of algorithms through manipulation and to integrate these results into their workflow. This is addressed with a novel interface designed for interactive analysis and visualization. We conclude with a discussion of some of the benefits and limitations of integrating AI models in the human expert's decision-making process in the presence of both subjective and objective metrics.

Automated Real-World Sustainability Data Generation from Images of Buildings

A Machine Learning Approach for Generating and Evaluating Forecasts on the Environmental Impact of the Buildings Sector

Building Usage Prediction in Complex Urban Scenes By Fusing Text and Facade Features from Street View Images Using Deep Learning

A Machine Learning-Based Method for the Large-Scale Evaluation of the Qualities of the Urban Environment.

Predicting Neighborhood-Level Residential Carbon Emissions from Street View Images Using Computer Vision and Machine Learning

Towards Automatic Generation of As-Built BIM: 3D Building Facade Modeling and Material Recognition from Images

BuildingView: Constructing Urban Building Exteriors Databases with Street View Imagery and Multimodal Large Language Mode

Using Social Media Images for Building Function Classification

Deep Learning Based Automated Estimation of Urban Green Space Index from Satellite Image: A Case Study

Building Information Modeling and Classification by Visual Learning At A City Scale

Development of an Automated Tool for Buildings' Sustainability Assessment in Early Design Stage

Simultaneous Extraction of Spatial and Attributional Building Information Across Large-Scale Urban Landscapes from High-Resolution Satellite Imagery

Embodied greenhouse gas emissions of buildings—Machine learning approach for early stage prediction

Visual Perception of Building and Household Vulnerability from Streets

Characterizing the perception of urban spaces from visual analytics of street-level imagery

Identifying every building's function in large-scale urban areas with multi-modality remote-sensing data

Understanding Building Energy Efficiency with Administrative and Emerging Urban Big Data by Deep Learning in Glasgow

Urban Built Environment Assessment Based on Scene Understanding of High-Resolution Remote Sensing Imagery

Zero-shot Building Attribute Extraction from Large-Scale Vision and Language Models

Encoding Urban Ecologies: Automated Building Archetype Generation through Self-Supervised Learning for Energy Modeling

Leveraging Internet of Things Network Metadata for Cost-Effective Automatic Smart Building Visualization