Abstract:In recent years, deep learning and multimodal data have substantially propelled the development of building extraction models. However, prevailing multimodal methods are difficult to cope with two challenges: 1) modal laziness: the training error is minimized before the model has learned extensive unimodal patterns and 2) modal imbalance: the backpropagation process is easily dominated by a certain modality. As a result, the unimodal features learning is insufficient, leading to limited performance of the model when dealing with the intricate foreground and background contexts surrounding the buildings. In this article, we deal with this problem from the perspective of algorithm and model evaluation. At the algorithmic level, we propose a unimodal feature enhancement (UFE) framework. Specifically, UFE is model-agnostic, comprising two distinct components: adaptive gradient enhancement (AGE) for modal laziness and consistency constraint loss (CCL) for modal imbalance. AGE dynamically modulates the original gradient by monitoring the representation effects of unimodal features and multimodal fusion features. CCL imposes mutual constraints on diverse modal branches at the semantic level to reconcile the optimization process. At the model evaluation level, a new metric, named unimodal utilization ratio (UUR), is presented to assess models through the learning efficacy of unimodal features. The experimental results including the variants of UUR on two building extraction datasets demonstrate a substantial performance improvement by UFE. Moreover, UFE also exhibits its adaptability when integrated with various model components and its generalization on other multimodal image-related tasks.

Sparse Multigraph Embedding for Multimodal Feature Representation.

Hamming Embedding Sensitivity Guided Fusion Network for 3D Shape Representation.

Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion

Heterogeneous Visual Features Fusion via Sparse Multimodal Machine

Enhancing Unimodal Features Matters: A Multimodal Framework for Building Extraction.

Multimodal Feature Fusion Based Hypergraph Learning Model

Multimodal Graph Learning for Cross-Modal Retrieval.

An Efficient Approach to Informative Feature Extraction from Multimodal Data

Multi-Manifold Sparse Graph Embedding for Multi-Modal Image Classification

Fusion Learning of Multimodal Neuroimaging with Weighted Graph AutoEncoder.

Unsupervised feature selection via multiple graph fusion and feature weight learning

Graph Embedding Learning for Cross-Modal Information Retrieval.

Learning Joint Multimodal Representation Based On Multi-Fusion Deep Neural Networks

A multimodal framework for unsupervised feature fusion

Weakly paired multimodal fusion using multilayer extreme learning machine

Feature relationships hypergraph for multimodal recognition

Multimodal Graph for Unaligned Multimodal Sequence Analysis via Graph Convolution and Graph Pooling

Dense Multimodal Fusion for Hierarchically Joint Representation

Sparse Fusion for Multimodal Transformers

Multimodal sparse representation learning and applications

Adaptive Multimodal Robust Feature Learning Based on Dual Graph-regularization