Point Clouds Classification and Segmentation for Nursery Trees Based on Improved PointNet Plus Plus Model
Jie Xu,Hui Liu,Yue Shen,Guanxue Yang,Hao Zhou,Siyuan Wang
DOI: https://doi.org/10.3788/cjl230989
2024-01-01
Chinese Journal of Lasers
Abstract:Objective Nurseries span large areas, so that managing and protecting landscape trees manually requires significant resources. Substituting agricultural robots for human forces can effectively address issues of low productivity and labor shortages. The perception systems are essential elements of agricultural robots. Agricultural robots can obtain essential information for tree statistical analysis, autonomous navigation, and target spray operations by perceiving tree species, trunks, crowns, and others. Therefore, it is of great significance to design a neural network model which can accurately obtain the information of tree species, crowns and trunks. Point clouds are more suitable for outdoor scenes due to their greater robustness to different lighting conditions. While there are many studies related to tree species classification and crown or trunk segmentation, most of them focus on trees located in forestry areas or require prior point cloud registration operations. Compared with trees in forest areas, landscape trees planted in nurseries are generally smaller in size. Moreover, during the process of work, robots are only able to capture partial point clouds of trees. In order to ensure the real-time performances, robots need to obtain relevant information on tree species, trunks, and crowns from partially scanned point clouds. Hence, based on PointNet++, we present an improvement model in this paper. We collect partial point clouds of trees from nurseries to train the model for tree species classification, as well as segmentation tasks of tree crowns, trunks and other parts. Experimental results demonstrate its superior classification and segmentation performances, which can prepare for the subsequent implementation of the application on the hardware platform. Methods The proposed neural network model adopts the hierarchical structure. Multiple set abstraction (SA) layers are used to extract the local features of the point clouds. Each SA layer consists of a sampling layer, a grouping layer, and a PointNet layer. The sampling layer employs the iterative farthest point sampling to select sampling points. Then, taking the sampling points as the center points, spherical regions with a fixed radius d are constructed as the local areas in the grouping layer. Each local area contains K neighboring points. As the number of layers increases, the radius d of the spherical region expands continuously. For larger local areas, the feature distribution adjustment module is employed to convert the relative features of neighboring points from linear to arc tangent, increasing the relative features of those closer to the central point while reducing the influence of distant neighboring points in each local area. Multilayer perceptrons (MLPs) are employed for extracting local features in the PointNet layer. To enhance the model's capacity to capture important information, we integrate the coordinate attention (CA) module with attention pooling for extracting local features. Furthermore, in the final SA layer, the model concatenates the low -dimensional and high -dimensional features of sampled points, and employs fully connected networks to predict the category of the input point clouds. The segmentation branch utilizes the U -Net structure, and employs an interpolation method based on inverse distance weighted average with K nearest neighbors, combining with skip links across levels for feature propagation. Finally, the category of each point is obtained to realize segmentation. Results and Discussions For classification and segmentation experiments, we collect seven kinds of point clouds of common landscape trees in the nursery using the Livox Horizon laser. During the data collection process, the parameters of the laser are shown in Table 1, and more detailed information of the collected point clouds is shown in Table 2. To perform segmentation tasks, each group of collected point clouds is further processed by dividing into 2 - 4 parts represented with different labels (Fig. 9). To demonstrate better classification and segmentation performances, the experimental results of the proposed model are compared with classic PointNet and PointNet+ + using the self-made dataset. Besides, we conduct experiments by replacing the improved attention module in our proposed model with squeeze -and -excitation (SE) attention module, convolutional block attention module (CBAM), and CA attention module. For the classification experiments, the models are first pre -trained using the public dataset ModelNet40 before being trained with the self-made dataset. And the results are shown in Table 4. The overall accuracy (OA) and mean class accuracy (mAcc) of the proposed model are 92.50% and 94.22%, respectively, which are higher than the corresponding metrics of PointNet and PointNet++ . Compared with other kinds of attention modules, the attention model utilized in the proposed model achieves the highest classification accuracy. The same models are also trained for the segmentation experiments, and the testing results are presented in Table 5. The evaluation indicator values including average intersection over union (mIoU), precision, recall and F-1 score of the proposed model are 89.09%, 90.09%, 95.44% and 92.59%, respectively, which are higher than those of PointNet+ +. Furthermore, our proposed model outperforms PointNet+ + in terms of accurately capturing fine details (Fig. 11). Conclusions In this paper, we present an improved neural network model based on point clouds, which is designed to classify tree species in the nursery and segment tree crowns, trunks, and other parts. Different from the classic neural network model PointNet++ , the proposed model combines the high -dimensional and low -dimensional features of the points to improve the performance. And the relative features of the neighboring points are adjusted to the form of arc tangent distribution, which can improve the corresponding values of the neighboring points closer to the central point in the local areas. Furthermore, the CA attention module and attention pooling module are combined to improve the ability of capturing the important features. To enable the practical application of agricultural robots, we collect and process point cloud data of landscape trees in the nursery to train the models. The experimental results demonstrate that our proposed network outperforms PointNet and PointNet++ in terms of both classification and segmentation tasks. The model proposed in this paper provides the theoretical foundation for future applications of agricultural robots in the nursery, such as tree statistical analysis, autonomous navigation, and target spray operations.