Abstract:Semantic segmentation using cross-modal data is a hot topic in the field of Earth observation. Compared with single-modal strategies, cross-modal networks fuse multiaspect information and yield higher segmentation accuracy, which is widely used in urban planning, environmental monitoring and so on. In this study, an end-to-end adaptive cross-modal fusion network (ACFNet) is proposed for semantic segmentation task using high resolution and light detection and ranging images, because of the difference of sensor resolution, different modal data have different abilities of ground object expression. Therefore, multimodal data fusion should consider the features with different spatial scales, while most existing methods simply use the same spatial scale features for fusion. In this work, we first design an adaptive scale fusion module that can automatically choose the features with optimal spatial scales, making full use of the representation properties of ground object details. Second, the important feature guidance module is designed, which can evaluate the influence weights of deep semantic features and shallow spatial detailed features, achieving adaptive deep and shallow feature fusion, and reducing the semantic-spatial information dilution caused by layer-by-layer up and down sampling. Finally, we introduce a divide Fourier context learning (DFCL) module to transform the feature maps from spatial domain to frequency domain. Compared to the limited perception of current spatial convolution kernels, the DFCL module can easily model the contextual dependencies of cross-modal features, which will improve the segmentaion accuracy for complex ground objects of cities, especially for occlusion. To demonstrate the generalisation performance of our module, we conduct extensive experiments and ablation studies on three datasets: Potsdam, Vaihingen, and IEEE GRSS DFC 2018. Results show that the proposed ACFNet is effective in semantic segmentation.

Multilevel Context Feature Fusion for Semantic Segmentation of ALS Point Cloud

Deep Dual-Stream Network with Scale Context Selection Attention Module for Semantic Segmentation

Enhanced Multi-Scale Feature Adaptive Fusion Sparse Convolutional Network for Large-Scale Scenes Semantic Segmentation

MLFNet- Point Cloud Semantic Segmentation Convolution Network Based on Multi-Scale Feature Fusion

Multilevel Geometric Feature Embedding in Transformer Network for ALS Point Cloud Semantic Segmentation

A Crossmodal Multiscale Fusion Network for Semantic Segmentation of Remote Sensing Data

HR and LiDAR Data Collaborative Semantic Segmentation Based on Adaptive Cross-Modal Fusion Network

Large-scale ALS Point Clouds Segmentation Via Projection-based Context Embedding

Context and Boundary Guided Multi-Scale Feature Fusion Network for Semantic Segmentation

Large-Scale ALS Point Cloud Segmentation via Projection-Based Context Embedding

MFEAFN: Multi-scale feature enhanced adaptive fusion network for image semantic segmentation

MCFNet: Multi-scale Covariance Feature Fusion Network for Real-time Semantic Segmentation

Multispectral LiDAR Point Cloud Segmentation for Land Cover Leveraging Semantic Fusion in Deep Learning Network

Multiscale Fusion Convolutional Network in Real-time Semantic Segmentation

Encoder- and Decoder-Based Networks Using Multiscale Feature Fusion and Nonlocal Block for Remote Sensing Image Semantic Segmentation

MFSA-Net: Semantic Segmentation with Camera-LiDAR Cross-Attention Fusion Based on Fast Neighbor Feature Aggregation

Multi-Field Context Fusion Network for Semantic Segmentation of High-Spatial-Resolution Remote Sensing Images

Multilevel feature fusion dilated convolutional network for semantic segmentation

MSCFNet: A Lightweight Network with Multi-Scale Context Fusion for Real-Time Semantic Segmentation

Semantic Segmentation of LiDAR Point Cloud Based on CAFF-PointNet

GAF-Net: Geometric Contextual Feature Aggregation and Adaptive Fusion for Large-Scale Point Cloud Semantic Segmentation