Abstract:Multimodal remote sensing (RS) image segmentation aims to comprehensively utilize multiple RS modalities to assign pixel-level semantics to the studied scenes, which can provide a new perspective for global city understanding. Multimodal segmentation inevitably encounters the challenge of modeling intra- and inter-modal relationships, <inline-formula> <tex-math notation="LaTeX">$i.e$ </tex-math></inline-formula>., object diversity and modal gaps. However, the previous methods are usually designed for a single RS modality, limited by the noisy collection environment and poor discrimination information. Neuropsychology and neuroanatomy confirm that the human brain performs the guiding perception and integrative cognition of multimodal semantics through intuitive reasoning. Therefore, establishing a semantic understanding framework inspired by intuition to realize multimodal RS segmentation becomes the main motivation of this work. Drived by the superiority of hypergraphs in modeling high-order relationships, we propose an intuition-inspired hypergraph network (<inline-formula> <tex-math notation="LaTeX">$I^{2}HN$ </tex-math></inline-formula>) for multimodal RS segmentation. Specifically, we present a hypergraph parser to imitate guiding perception to learn intra-modal object-wise relationships. It parses the input modality into irregular hypergraphs to mine semantic clues and generate robust mono-modal representations. In addition, we also design a hypergraph matcher to dynamically update the hypergraph structure from the explicit correspondence of visual concepts, similar to integrative cognition, to improve cross-modal compatibility when fusing multimodal features. Extensive experiments on two multimodal RS datasets show that the proposed <inline-formula> <tex-math notation="LaTeX">$I^{2}HN$ </tex-math></inline-formula> outperforms the state-of-the-art models, achieving F1/mIoU accuracy 91.4%/82.9% on the ISPRS Vaihingen dataset, and 92.1%/84.2% on the MSAW dataset.

Remote sensing image segmentation using geodesic-kernel functions and multi-feature spaces

Learning an Image Manifold for Retrieval

Riemannian Manifold-Based Feature Space and Corresponding Image Clustering Algorithms

Geodesic Based Semi-supervised Multi-manifold Feature Extraction

Improved Mean-Shift Segmentation Approach for Remote Sensing Images

Robust Local Feature Descriptor for Multisource Remote Sensing Image Registration

A New Robust Approach for Remote Sensing Image Regional Classification

Multi-modal remote sensing image segmentation based on attention-driven dual-branch encoding framework

Segmentation for High-Resolution Optical Remote Sensing Imagery Using Improved Quadtree and Region Adjacency Graph Technique

Multimodal Remote Sensing Image Segmentation With Intuition-Inspired Hypergraph Modeling

Semi-supervised Remote Sensing Image Segmentation Using Dynamic Region Merging.

Optimizing Multiscale Segmentation with Local Spectral Heterogeneity Measure for High Resolution Remote Sensing Images

Toward Evaluating Multiscale Segmentations of High Spatial Resolution Remote Sensing Images

Segmentation Quality Evaluation Using Region-Based Precision and Recall Measures for Remote Sensing Images

Unsupervised Multi-Class Co-Segmentation via Joint-Cut Over $L_{1}$ -Manifold Hyper-Graph of Discriminative Image Regions

DOCNet: Dual-Domain Optimized Class-Aware Network for Remote Sensing Image Segmentation

Manifold Spectral Clustering Image Segmentation Algorithm Based on Local Geometry Features

An automatic region-based image segmentation algorithm for remote sensing applications

Adaptive Hierarchical Multinomial Latent Model with Hybrid Kernel Function for SAR Image Semantic Segmentation

Double-Variance Measures: A Potential Approach to Parameter Optimization of Remote Sensing Image Segmentation

Remote Sensing Image Registration Using Multiple Image Features