PanicleNeRF: low-cost, high-precision in-field phenotypingof rice panicles with smartphone

Xin Yang,Xuqi Lu,Pengyao Xie,Ziyue Guo,Hui Fang,Haowei Fu,Xiaochun Hu,Zhenbiao Sun,Haiyan Cen

2024-08-04

Abstract:The rice panicle traits significantly influence grain yield, making them a primary target for rice phenotyping studies. However, most existing techniques are limited to controlled indoor environments and difficult to capture the rice panicle traits under natural growth conditions. Here, we developed PanicleNeRF, a novel method that enables high-precision and low-cost reconstruction of rice panicle three-dimensional (3D) models in the field using smartphone. The proposed method combined the large model Segment Anything Model (SAM) and the small model You Only Look Once version 8 (YOLOv8) to achieve high-precision segmentation of rice panicle images. The NeRF technique was then employed for 3D reconstruction using the images with 2D segmentation. Finally, the resulting point clouds are processed to successfully extract panicle traits. The results show that PanicleNeRF effectively addressed the 2D image segmentation task, achieving a mean F1 Score of 86.9% and a mean Intersection over Union (IoU) of 79.8%, with nearly double the boundary overlap (BO) performance compared to YOLOv8. As for point cloud quality, PanicleNeRF significantly outperformed traditional SfM-MVS (structure-from-motion and multi-view stereo) methods, such as COLMAP and Metashape. The panicle length was then accurately extracted with the rRMSE of 2.94% for indica and 1.75% for japonica rice. The panicle volume estimated from 3D point clouds strongly correlated with the grain number (R2 = 0.85 for indica and 0.82 for japonica) and grain mass (0.80 for indica and 0.76 for japonica). This method provides a low-cost solution for high-throughput in-field phenotyping of rice panicles, accelerating the efficiency of rice breeding.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The main problem that this paper attempts to solve is to develop a low - cost and high - precision method for phenotyping rice panicles under field conditions. Specifically, the paper proposes a new method named PanicleNeRF. By using multi - view images taken by smart phones and combining the large - scale model Segment Anything Model (SAM) and the small - scale model You Only Look Once version 8 (YOLOv8), it achieves high - precision segmentation of rice panicle images and uses NeRF technology for three - dimensional reconstruction, thereby extracting panicle features.

### Main problems:
1. **Limitations of existing technologies**:
- Most existing technologies are limited to controlled indoor environments and it is difficult to capture the characteristics of rice panicles under natural growth conditions.
- Existing methods usually require harvesting rice panicles from the field and manual processing, which is labor - intensive and inefficient.

2. **Requirements for field phenotyping**:
- High - throughput and low - cost field phenotyping methods are of great significance for improving rice breeding efficiency and monitoring the panicle development process.

### Solutions:
- **PanicleNeRF method**:
- **2D image segmentation**: Combine SAM and YOLOv8 to achieve high - precision segmentation of rice panicle images.
- **3D reconstruction**: Use NeRF technology to perform three - dimensional reconstruction based on the segmented 2D images.
- **Point cloud processing**: Pre - process and calibrate the generated point clouds and extract features such as panicle length and volume.
- **Performance evaluation**: Verify the superiority of the PanicleNeRF method by comparing it with traditional methods (such as COLMAP and Metashape).

### Specific objectives:
1. Accurately segment the target rice panicles and label 2D images.
2. Reconstruct a complete, high - precision, and low - noise 3D model of rice panicles.
3. Pre - process and calibrate the point clouds.
4. Extract panicle length and volume features from the calibrated point cloud model and predict the number of grains and grain quality of the panicles.

### Method innovation points:
- **Multi - model fusion**: Combine the advantages of large - scale and small - scale models to improve the accuracy of 2D image segmentation.
- **NeRF technology**: Use NeRF technology to overcome the limitations of traditional SfM - MVS methods in 3D reconstruction of small targets and achieve high - precision 3D model reconstruction.

Through these innovations, the PanicleNeRF method provides an efficient and low - cost solution, which is suitable for high - throughput phenotyping of rice panicles in the field and helps to accelerate the rice breeding process.