Abstract:Abstract. The semantic segmentation of the huge amount of acquired 3D data has become an important task in recent years. Meshes have evolved into a standard representation next to Point Clouds (PCs) – not least because of their great visualization possibilities. Compared to PCs, meshes have commonly smaller memory footprints while jointly providing geometrical and high-resolution textural information. For this reason, we opt for semantic mesh segmentation, which is a widely overlooked topic in photogrammetry and remote sensing yet. In this work, we perform an extensive ablation study on multi-modal handcrafted features adapting the Point Cloud Mesh Association (PCMA) (Laupheimer et al., 2020) which establishes explicit connections between faces and points. The multi-modal connections are used in a two-fold manner: (i) to extend per-face descriptors with features engineered on the PC and (ii) to annotate meshes semi-automatically by propagating the manually assigned labels from the PCs. In this way, we derive annotated meshes from the ISPRS benchmark data sets Vaihingen 3D (V3D) and Hessigheim 3D (H3D). To demonstrate the effectiveness of the multi-modal approach, we use well-established and fast Random Forest (RF) models deploying various feature vector compositions and analyze their performances for semantic mesh segmentation. The feature vector compositions consider features derived from the mesh, the PC or both. The results indicate that the combination of radiometric and geometric features outperforms feature sets of a single feature type only. Besides, we observe that relative height is the most crucial feature. The main finding is that the multi-modal feature vector integrates the complementary strengths of the underlying modalities. Whereas the mesh provides outstanding textural information, the dense PCs are superior in geometry. The multi-modal feature descriptor achieves the best performance on both data sets. It significantly outperforms feature sets that incorporate only features derived from the mesh by +7.37 pp and +2.38 pp for mF1 and Overall Accuracy (OA) on V3D. The registered improvement is +9.23 pp and +4.33 pp for mF1 and OA on H3D.

A deep-learning model for semantic segmentation of meshes from UAV oblique images

Semantic segmentation network of uav image based on improved U-net

An Image Segmentation Method Based on Transformer and Multi-Scale Feature Fusion for UAV Marine Environment Monitoring

MeshFormer: High-resolution Mesh Segmentation with Graph Transformer

SEMANTIC URBAN MESH SEGMENTATION BASED ON AERIAL OBLIQUE IMAGES AND POINT CLOUDS USING DEEP LEARNING

UAVformer: A Composite Transformer Network for Urban Scene Segmentation of UAV Images

UrbanSegNet: An urban meshes semantic segmentation network using diffusion perceptron and vertex spatial attention

3D Semantic Segmentation Using Deep Learning for Large-Scale Indoor Point Cloud

A transformer-based UAV instance segmentation model TF-YOLOv7

MULTI-MODAL SEMANTIC MESH SEGMENTATION IN URBAN SCENES

Image Semantic Segmentation Method Based on Deep Learning in UAV Aerial Remote Sensing Image

Semantic segmentation of oblique UAV video based on ConvLSTM in complex urban area

Semantic 3D Reconstruction with Learning MVS and 2D Segmentation of Aerial Images

Multi-scale Feature Extraction and Fusion Net: Research on UAVs Image Semantic Segmentation Technology

Automatic extraction and 3D modeling of real road scenes using UAV imagery and deep learning semantic segmentation

Meshformers: Transformer-Based Networks for Mesh Understanding

UNetFormer: A UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery

BFANet: Effective Segmentation Network for Low Altitude High-Resolution Urban Scene Image

SEMANTIC SEGMENTATION OF UAV IMAGES BASED ON U-NET IN URBAN AREA

UNeXt: An Efficient Network for the Semantic Segmentation of High-Resolution Remote Sensing Images

Advancing high-resolution remote sensing: a compact and powerful approach to semantic segmentation