Abstract:Weed competitiveness monitoring is crucial for field management at specific locations. Recent research in the fusion of multimodal data from unmanned aerial vehicles (UAVs) has propelled this advancement. However, these studies merely stack extracted features equivalently, neglecting the full utilization of fused information. This study utilizes hyperspectral and LiDAR data collected by UAVs to proposes a multimodal deep fusion model (MulDFNet) using Transformer and multi-layer residuals. It utilizes a comprehensive competitive index (CCI-A) based on multidimensional phenotypes of maize to assess the competitiveness of weeds in farmland ecosystems. To validate the effectiveness of this model, a series of ablation studies were conducted involving different modalities data, with/without the Transformer Encoder (TE) modules, and different fusion modules (shallow residual fusion module, deep feature fusion module). Additionally, a comparison was made with early/late stacking fusion models, traditional machine learning models, and deep learning models from relevant studies. The results indicate that the multimodal deep fusion model utilizing HSI, VI, and CHM data achieved a predictive effect of R 2 = 0.903 (RMSE = 0.078). Notably, the best performance was observed during the five-leaf stage. The combination of shallow and deep fusion modules demonstrated better predictive performance compared to a single fusion module. The positive impact of the TE module on model performance is evident, as its multi-head attention mechanism aids in better capturing the relationships and importance between feature maps and competition indices, thereby enhancing the model's predictive capability. In weed competition prediction, the multimodal deep fusion model proposed in this study has demonstrated significantly better predictive performance compared to early/late stacking fusion models and other machine learning models (RF, SVR, PLS, DNN-F2 and Multi-channel CNN). Overall, the multimodal deep fusion model developed in this study demonstrates outstanding performance in assessing weed competitiveness and can predict the competitive intensity of weeds in maize across various growth stages on a broad scale.

Deep multimodal fusion model for moisture content measurement of sand gravel using images, NIR spectra, and dielectric data

Interleaved Attention Convolutional Compression Network: an Effective Data Mining Method for the Fusion System of Gas Sensor and Hyperspectral

Multi-modal Fusion Deep Learning Model for Excavated Soil Heterogeneous Data with Efficient Classification

Quantitative Analysis of Moisture Content and Particle Size in a Fluidized Bed Granulation Process Using Near Infrared Spectroscopy and Acoustic Emission Combined with Data Fusion Strategies.

Deep Multimodal Fusion Network for Semantic Segmentation Using Remote Sensing Image and LiDAR Data

Spectral Fusion Modeling for Soil Organic Carbon by a Parallel Input-Convolutional Neural Network

Moisture Prediction of Biomass Fuel Based on Near-Infrared Spectroscopy and Deep Learning Algorithm

A systematic framework for DEM study of realistic gravel-sand mixture from particle recognition to macro- and micro-mechanical analysis

MoistNet: Machine vision-based deep learning models for wood chip moisture content measurement

Moisture content online detection system based on multi-sensor fusion and convolutional neural network

A Multi-Modal Approach to Rail Surface Condition Analysis: the MFDF-Net

Multimodal deep fusion model based on Transformer and multi-layer residuals for assessing the competitiveness of weeds in farmland ecosystems

More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification

A Deep Learning Data Fusion Model using Sentinel-1/2, SoilGrids, SMAP-USDA, and GLDAS for Soil Moisture Retrieval

A Deep Feature Fusion Method for Complex Ground Object Classification in the Land Cover Ecosystem Using ZY1-02D and Sentinel-1A

Geological Remote Sensing Interpretation Using Deep Learning Feature and an Adaptive Multisource Data Fusion Network

MSMDFusion: Fusing LiDAR and Camera at Multiple Scales with Multi-Depth Seeds for 3D Object Detection.

Detection of moisture content in salted sea cucumbers by hyperspectral and low field nuclear magnetic resonance based on deep learning network framework

MSFusion: Multistage for Remote Sensing Image Spatiotemporal Fusion Based on Texture Transformer and Convolutional Neural Network

Development of Multifrequency-Swept Microwave Sensing System for Moisture Measurement of Sweet Corn With Deep Neural Network

CMGFNet: A deep cross-modal gated fusion network for building extraction from very high-resolution remote sensing images