Abstract:The ability to automatically build 3D digital twins of plants from images has countless applications in agriculture, environmental science, robotics, and other fields. However, current 3D reconstruction methods fail to recover complete shapes of plants due to heavy occlusion and complex geometries. In this work, we present a novel method for 3D reconstruction of agricultural crops based on optimizing a parametric model of plant morphology via inverse procedural modeling. Our method first estimates depth maps by fitting a neural radiance field and then employs Bayesian optimization to estimate plant morphological parameters that result in consistent depth renderings. The resulting 3D model is complete and biologically plausible. We validate our method on a dataset of real images of agricultural fields, and demonstrate that the reconstructions can be used for a variety of monitoring and simulation applications.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to automatically construct a complete 3D digital twin of crop plants from images in agricultural scenarios. Specifically, current 3D reconstruction methods are unable to recover the complete shape of plants due to severe occlusion and complex geometric structures. To this end, the paper proposes a new method based on Inverse Procedural Modeling to achieve 3D reconstruction of agricultural crops by optimizing plant morphological parameters. ### Main Problems 1. **Challenges in 3D Reconstruction**: Existing 3D reconstruction methods face problems of severe occlusion and complex geometric structures when dealing with plants, resulting in incomplete reconstruction results. 2. **Limitations of Existing Methods**: - **Multi - view Reconstruction Methods** (such as methods based on neural rendering or multi - view stereo vision) are unable to reconstruct invisible areas, resulting in incomplete plant shapes. - **Learning Methods** require a large amount of ground - truth 3D data for training, and such data are difficult to collect in dense vegetation. ### Solutions The paper proposes a method that combines data - driven neural reconstruction methods and process - based generative model methods. The specific steps are as follows: 1. **Depth Map Estimation**: Use the Neural Radiance Field (NeRF) technology to estimate the geometric structure of the visible surface. 2. **Row - Aligned Camera Pose**: Estimate the plant row position through the RANSAC algorithm and determine a standard camera pose. 3. **Loss Function Definition**: Define a loss function based on depth - map histogram statistics to minimize the difference between the process - generative model parameters and the observed depth map. 4. **Bayesian Optimization**: Use the Bayesian optimization method to minimize the loss function, thereby estimating the optimal process - generative model parameters. ### Key Contributions 1. **Reconstruction of Complete 3D Morphological Models**: A new method is proposed that can reconstruct complete 3D morphological models of large - scale crop fields from a series of images. 2. **Image - Based Crop Productivity Quantification Framework**: For the first time, an image - based crop productivity quantification framework is introduced, making large - scale carbon - exchange monitoring possible. ### Experimental Verification The paper verifies the effectiveness of the method by conducting experiments on a multi - view image data set of real agricultural fields. The experimental results show that this method can successfully reconstruct real crop canopies at different growth stages and is more accurate in estimating key canopy structure variables than the baseline method. In addition, the reconstructed 3D canopy can be directly used in radiation - transfer modeling software to provide accurate photosynthesis rate predictions. ### Summary The paper solves the problem that existing 3D reconstruction methods cannot handle severe occlusion and complex geometric structures in agricultural scenarios. It proposes a new method that combines neural reconstruction and process - generative models, achieving complete 3D reconstruction of crop plants. This provides important application prospects in fields such as agricultural monitoring, environmental science, and robotics.

CropCraft: Inverse Procedural Modeling for 3D Reconstruction of Crop Plants

Realistic Procedural Plant Modeling from Multiple View Images.

Research on 3D Reconstruction Algorithm of Citrus Seedlings Based on Multi-View Geometry

Realistic procedural plant modeling guided by 3D point cloud

High-fidelity 3D Reconstruction of Plants using Neural Radiance Field

Research on automatic 3D reconstruction of plant phenotype based on Multi-View images

Tomato-Nerf: Advancing Tomato Model Reconstruction with Improved Neural Radiance Fields

Plant growth parameter estimation from sparse 3D reconstruction based on highly-textured feature points

Research Advances and Prospects of Crop 3D Reconstruction Technology

Plant Modeling Based on 3D Reconstruction and Its Application in Digital Museum.

NeRF-based 3D reconstruction pipeline for acquisition and analysis of tomato crop morphology

Plant Modeling and Its Application in Digital Agriculture Museum.

Three-Dimensional Plant Reconstruction with Enhanced Cascade-MVSNet.

Image-based 3D Corn Reconstruction for Retrieval of Geometrical Structural Parameters

Ground truth evaluation of computer vision based 3D reconstruction of synthesized and real plant images

3D Reconstruction of Ground Crops Based on Airborne LiDAR Technology

4D Crop Monitoring: Spatio-Temporal Reconstruction for Agriculture

CropPainter: an Effective and Precise Tool for Trait-to-image Crop Visualization Based on Generative Adversarial Networks.

Research on Crop 3D Model Reconstruction Based on RGB-D Binocular Vision

The Research on Plant's Realistic Modeling Method

Fast Multi-View 3D Reconstruction of Seedlings Based on Automatic Viewpoint Planning