Abstract:In both plant breeding and crop management, interpretability plays a crucial role in instilling trust in AI-driven approaches and enabling the provision of actionable insights. The primary objective of this research is to explore and evaluate the potential contributions of deep learning network architectures that employ stacked LSTM for end-of-season maize grain yield prediction. A secondary aim is to expand the capabilities of these networks by adapting them to better accommodate and leverage the multi-modality properties of remote sensing data. In this study, a multi-modal deep learning architecture that assimilates inputs from heterogeneous data streams, including high-resolution hyperspectral imagery, LiDAR point clouds, and environmental data, is proposed to forecast maize crop yields. The architecture includes attention mechanisms that assign varying levels of importance to different modalities and temporal features that, reflect the dynamics of plant growth and environmental interactions. The interpretability of the attention weights is investigated in multi-modal networks that seek to both improve predictions and attribute crop yield outcomes to genetic and environmental variables. This approach also contributes to increased interpretability of the model's predictions. The temporal attention weight distributions highlighted relevant factors and critical growth stages that contribute to the predictions. The results of this study affirm that the attention weights are consistent with recognized biological growth stages, thereby substantiating the network's capability to learn biologically interpretable features. Accuracies of the model's predictions of yield ranged from 0.82-0.93 R2 ref in this genetics-focused study, further highlighting the potential of attention-based models. Further, this research facilitates understanding of how multi-modality remote sensing aligns with the physiological stages of maize. The proposed architecture shows promise in improving predictions and offering interpretable insights into the factors affecting maize crop yields, while demonstrating the impact of data collection by different modalities through the growing season. By identifying relevant factors and critical growth stages, the model's attention weights provide valuable information that can be used in both plant breeding and crop management. The consistency of attention weights with biological growth stages reinforces the potential of deep learning networks in agricultural applications, particularly in leveraging remote sensing data for yield prediction. To the best of our knowledge, this is the first study that investigates the use of hyperspectral and LiDAR UAV time series data for explaining/interpreting plant growth stages within deep learning networks and forecasting plot-level maize grain yield using late fusion modalities with attention mechanisms.

Developing a Deep Learning Network “Mscp-Net” to Generate Stalk Anatomical Traits Related with Crop Lodging and Yield in Maize

Measurement of Maize Stem Cross Section Parameters Based on Semantic Segmentation and Instance Segmentation

Automatic Segmentation of Stem and Leaf Components and Individual Maize Plants in Field Terrestrial LiDAR Data Using Convolutional Neural Networks

Dissecting the phenotypic components and genetic architecture of maize stem vascular bundles using high‐throughput phenotypic analysis

Separating the Structural Components of Maize for Field Phenotyping Using Terrestrial LiDAR Data and Deep Convolutional Neural Networks

A Two-Stage Leaf–Stem Separation Model for Maize With High Planting Density With Terrestrial, Backpack, and UAV-Based Laser Scanning

Stem–Leaf Segmentation and Phenotypic Trait Extraction of Individual Maize Using Terrestrial LiDAR Data

A spatiotemporal transcriptomic network dynamically modulates stalk development in maize

Deep Learning: Individual Maize Segmentation from Terrestrial Lidar Data Using Faster R-CNN and Regional Growth Algorithms

Maize plant growth period identification based on MobileNet and design of growth control system

Genome-Wide Association Study of Vascular Bundle-Related Traits in Maize Stalk

Three-dimensional branch segmentation and phenotype extraction of maize tassel based on deep learning

Pretrained Deep Learning Networks and Multispectral Imagery Enhance Maize LCC, FVC, and Maturity Estimation

Maize stem–leaf segmentation framework based on deformable point clouds

Stem-leaf segmentation and phenotypic trait extraction of maize shoots from three-dimensional point cloud

Recognition and Localization of Maize Leaf and Stalk Trajectories in RGB Images Based on Point-Line Net

Genetic Structure and Molecular Mechanism Underlying the Stalk Lodging Traits in Maize (zea Mays L.)

Image-based Root Phenotyping for Field-Grown Crops: an Example under Maize/soybean Intercropping

Simulated Plant Images Improve Maize Leaf Counting Accuracy

Integrating multi-modal remote sensing, deep learning, and attention mechanisms for yield prediction in plant breeding experiments

Detection of maize stem diameter by using RGB-D cameras' depth information under selected field condition