Biomass phenotyping of oilseed rape through UAV multi-view oblique imaging with 3DGS and SAM model

Yutao Shen,Hongyu Zhou,Xin Yang,Xuqi Lu,Ziyue Guo,Lixi Jiang,Yong He,Haiyan Cen
2024-11-13
Abstract:Biomass estimation of oilseed rape is crucial for optimizing crop productivity and breeding strategies. While UAV-based imaging has advanced high-throughput phenotyping, current methods often rely on orthophoto images, which struggle with overlapping leaves and incomplete structural information in complex field environments. This study integrates 3D Gaussian Splatting (3DGS) with the Segment Anything Model (SAM) for precise 3D reconstruction and biomass estimation of oilseed rape. UAV multi-view oblique images from 36 angles were used to perform 3D reconstruction, with the SAM module enhancing point cloud segmentation. The segmented point clouds were then converted into point cloud volumes, which were fitted to ground-measured biomass using linear regression. The results showed that 3DGS (7k and 30k iterations) provided high accuracy, with peak signal-to-noise ratios (PSNR) of 27.43 and 29.53 and training times of 7 and 49 minutes, respectively. This performance exceeded that of structure from motion (SfM) and mipmap Neural Radiance Fields (Mip-NeRF), demonstrating superior efficiency. The SAM module achieved high segmentation accuracy, with a mean intersection over union (mIoU) of 0.961 and an F1-score of 0.980. Additionally, a comparison of biomass extraction models found the point cloud volume model to be the most accurate, with an determination coefficient (R2) of 0.976, root mean square error (RMSE) of 2.92 g/plant, and mean absolute percentage error (MAPE) of 6.81%, outperforming both the plot crop volume and individual crop volume models. This study highlights the potential of combining 3DGS with multi-view UAV imaging for improved biomass phenotyping.
Computer Vision and Pattern Recognition
What problem does this paper attempt to address?
### Problems the Paper Attempts to Solve This paper aims to improve the accuracy of biomass phenotyping analysis of rapeseed crops by combining multi-view oblique imaging technology with drones, 3D Gaussian point clouds (3DGS), and any segmentation model (SAM). Specifically, the paper attempts to address the following issues: 1. **Limitations of Existing Methods**: Most current drone-based biomass estimation methods rely on orthophotos, which often encounter overlapping leaves and incomplete structural information in complex field environments, leading to low prediction accuracy. 2. **Advantages of Multi-View Oblique Imaging**: By using drones to capture oblique images from multiple angles, the 3D structure information of crops can be more comprehensively captured, thereby improving the accuracy of biomass estimation. 3. **Challenges of 3D Reconstruction and Segmentation**: Traditional 3D reconstruction methods such as Structure from Motion (SfM) and Neural Radiance Fields (NeRF) are effective but computationally expensive and time-consuming. 3DGS, as a lightweight and efficient 3D reconstruction method, can significantly improve reconstruction speed and accuracy. However, precise segmentation of 3D point clouds remains a challenge. 4. **Accuracy of Biomass Estimation**: By combining 3DGS with SAM, precise segmentation of the 3D point cloud of rapeseed crops is achieved, and it is converted into point cloud volume. Finally, a linear regression model is used to fit the ground-measured biomass to improve the accuracy of biomass estimation. ### Research Objectives 1. **Propose a Multi-View Oblique Imaging Method**: Utilize drones to capture oblique images of rapeseed crops from multiple angles and verify the performance of 3DGS in field 3D reconstruction. 2. **Integrate 3DGS with SAM**: Achieve precise segmentation of the 3D point cloud of rapeseed crops. 3. **Estimate Biomass Based on Segmented Point Cloud**: Estimate the biomass of rapeseed crops through the point cloud volume model and demonstrate its superiority over traditional crop surface model methods. ### Main Contributions - **Improved Accuracy and Efficiency of 3D Reconstruction**: 3DGS provided peak signal-to-noise ratios (PSNR) of 27.43 and 29.53 at 7k and 30k iterations, respectively, with training times of 7 minutes and 49 minutes, outperforming SfM and Mip-NeRF. - **Achieved High-Precision Point Cloud Segmentation**: The SAM module achieved a mean Intersection over Union (mIoU) of 0.961 and an F1 score of 0.980. - **Improved Accuracy of Biomass Estimation**: The coefficient of determination (R²) of the point cloud volume model was 0.976, the root mean square error (RMSE) was 2.92 grams per plant, and the mean absolute percentage error (MAPE) was 6.81%, outperforming the plot crop volume and single plant crop volume models. Through this research, the paper demonstrates the potential of combining 3DGS and multi-view drone imaging technology to improve the accuracy of biomass phenotyping analysis of rapeseed crops.