Abstract:Nowadays, advanced sensor technology and image processing algorithms allow us to measure different aspects of the objects on the Earth’s surface, from spectral characteristics in optical images, height information in LiDAR data, to spatial information generated by image processing technologies like commercial software eCognition®. However, automatic recognition of objects in remote sensed scenes remains challenging. It is clear that single technology might not be sufficient to obtain reliable classification results (Debes, 2014). Multisensor data, once combined, can contribute to a more comprehensive interpretation of objects on the ground. For example, spectral reflections from optical image cannot recognize objects under shadows, while they can often be easily detected by LiDAR data. On the other hand, LiDAR data alone may fail to discriminate between objects that are quite similar in height.Stacking multi-source data together is a widely applied data fusion technique for classification. These methods first apply feature extraction on each individual data source, after which all feature sources are concatenated into one stacked vector for classification. While such methods are appealing due to their simplicity, they do not always perform better than using a single data source. This is because the value of different components in the stacked feature vector can be significantly unbalanced. As a consequence, theinformation contained by different data sources is not equally represented or measured. Furthermore, the increase in the dimensionality of the stacked features, combined with the limited number of labelled samples, may together lead to the problem of the “curse of dimensionality” (Liao, 2015).Therefore, we present a local graph fusion method to fuse true orthophoto and LiDAR image for urban object recognition. First, object-based spatial and height information are generated on true orthophoto and LiDAR image, respectively. Second, we build a local fusion graph within a sliding window where only the data points with similar spatial and height characteristics are connected. Finally, we solve the problem of multisensor data fusion by projecting multisensor data into a subspace, on which the advantages ofdifferent data sources are well exploited. Experimental results on fusion of true orthophoto and LiDAR image from 'ISPRS Test Project on Urban Classification and 3D Building Reconstruction' demonstrate the potential of the proposed method. Compared to the methods using only single data source or stacking them together, our approach has significant improvements in overall classification accuracy. Both the method’s details and the results of a comprehensive test will be presented at GEOBIA 2016.

Comprehensive urban space representation with varying numbers of street-level images

CMM: LiDAR-Visual Fusion with Cross-Modality Module for Large-Scale Place Recognition

Profiling Urban Streets: A Semi-Supervised Prediction Model Based on Street View Imagery and Spatial Topology

Multi-level Urban Street Representation with Street-View Imagery and Hybrid Semantic

Multi-modal Fusion of Satellite and Street-View Images for Urban Village Classification Based on a Dual-Branch Deep Neural Network

Using Street-level Images and Deep Learning for Urban La ndscape STUDIES

Simultaneous Extraction of Spatial and Attributional Building Information Across Large-Scale Urban Landscapes from High-Resolution Satellite Imagery

A Deep-Learning-Based Multimodal Data Fusion Framework for Urban Region Function Recognition

StreetviewLLM: Extracting Geographic Information Using a Chain-of-Thought Multimodal Large Language Model

Multilevel Spatial-Channel Feature Fusion Network for Urban Village Classification by Fusing Satellite and Streetview Images

Mixed land use measurement and mapping with street view images and spatial context-aware prompts via zero-shot multimodal learning

A Spatial Analysis of Urban Streets under Deep Learning Based on Street View Imagery: Quantifying Perceptual and Elemental Perceptual Relationships

Integrating Aerial and Street View Images for Urban Land Use Classification

UrbanVLP: Multi-Granularity Vision-Language Pretraining for Urban Region Profiling

Understanding urban landuse from the above and ground perspectives: A deep learning, multimodal solution

Urban Land Use Classification Model Fusing Multimodal Deep Features

Fusion of optical and liDAR images for urban objects recognition

Large-Scale Street Space Quality Evaluation Based On Deep Learning Over Street View Image

Predicting Multi-level Socioeconomic Indicators from Structural Urban Imagery

Learning visual features from figure-ground maps for urban morphology discovery

Urban2Vec: Incorporating Street View Imagery and POIs for Multi-Modal Urban Neighborhood Embedding