Abstract:Remote sensing image scene classification is to annotate semantic categories for image areas covering multiple land cover types, reflecting the spatial aggregation of relevant social resources among feature objects, which is one of the remote sensing interpretation tasks with higher challenges for algorithms to understand the images. Nowadays, scene semantic information extraction of images using deep neural networks is also one of the hot research directions. In comparison to other algorithms, deep neural networks can better capture semantic information in images to achieve higher classification accuracy involved in applications such as urban planning. In recent years, multi-modal models represented by image-text have achieved satisfactory performance in downstream tasks. The introduction of "multi-modal" in the field of remote sensing research should not be limited to the use of multi-source data, but more importantly to the coding of diverse data and the extracted deep features based on the huge amount of data. Therefore, in this paper, based on an image-text matching model, we establish a multi-modal scene classification model (Fig. 1) for high spatial resolution aerial images which is dominated by image features and text provides facilitation for the representation of image features. The algorithm first employs self-supervised learning of the visual model, to align the expression domain of the image features obtained from training on natural images with that of our particular dataset, which will help to improve the feature extraction effectiveness of the aerial survey images on the visual model. The features generated by the pre-trained image encoding model and the text encoding model will be further aligned and some of the parameters in the image encoder will be iteratively updated during training. A valid classifier is designed at the end of the model to implement the scene classification task. Through experiments, it was found that our algorithm has a significant improvement effect on the task of scene categorization on aerial survey images compared to single visual models. The model presented in the article obtained precision and recall of above 90% on the test dataset, contained in the high spatial resolution aerial survey images dataset we built with 27 categories (Fig. 2). Fig 1. Diagram of the proposed model structure. Blue boxes are associated with the image, green boxes with the text, and red boxes with both image and text. Fig 2. Samples in our high spatial resolution aerial survey images dataset.

Research on High Resolution Synthetic Aperture Radar Image Urban Scene Classification Based on Local Semantic Representation

High Spatial-Resolution Classification of Urban Surfaces Using a Deep Learning Method

Urban Scene Classification with VHR Images

SAR Image Segmentation Based on Hierarchical Visual Semantic and Adaptive Neighborhood Multinomial Latent Model

Semantic and Spatial Co-Occurrence Analysis on Object Pairs for Urban Scene Classification.

Urban Landscape Classification Using Chinese Advanced High-Resolution Satellite Imagery and an Object-Oriented Multi-Variable Model

Massive-Scale Visual Information Retrieval towards City Residential Environment Surveillance

Semantic segmentation of urban land classes using a multi-scale dataset

A Multi-Modal High Spatial Resolution Aerial Imagery Scene Classification Model with Visual Enhancement

Semantic segmentation of deep learning remote sensing images based on band combination principle: Application in urban planning and land use

Multilevel object based image classification over urban area based hierarchical image segmentation and invariant moments

Comprehensively analyzing optical and polarimetric SAR features for land-use/land-cover classification and urban vegetation extraction in highly-dense urban area

Semantic Classification of Urban Buildings Combining Vhr Image and Gis Data: an Improved Random Forest Approach

Synthetic aperture radar remote sensing scene classification based on fuzzy co-occurrence networks

Segmentation-based VHR SAR images built-up area change detection: a coarse-to-fine approach

A Study on Segmentation-based, Multi-stage and Mixed Classification for Land Type Identification on High Resolution Remotely Sensed Imagery

Unsupervised Classification of High-Resolution SAR Images Using Multilayer Level Set Method.

Classification of High Resolution Urban Remote Sensing Images Using Deep Networks by Integration of Social Media Photos

Continuous Multi-Angle Remote Sensing and Its Application in Urban Land Cover Classification

Extracting Of Urban Features From High Resolution Remote Sensing Data Based On Multiscale Segmentation

High-resolution Multispectral Image Classification over Urban Areas by Image Segmentation and Extended Morphological Profile