Abstract:The 3D reconstruction of plants is challenging due to their complex shape causing many occlusions. Next-Best-View (NBV) methods address this by iteratively selecting new viewpoints to maximize information gain (IG). Deep-learning-based NBV (DL-NBV) methods demonstrate higher computational efficiency over classic voxel-based NBV approaches but current methods require extensive training using ground-truth plant models, making them impractical for real-world plants. These methods, moreover, rely on offline training with pre-collected data, limiting adaptability in changing agricultural environments. This paper proposes a self-supervised learning-based NBV method (SSL-NBV) that uses a deep neural network to predict the IG for candidate viewpoints. The method allows the robot to gather its own training data during task execution by comparing new 3D sensor data to the earlier gathered data and by employing weakly-supervised learning and experience replay for efficient online learning. Comprehensive evaluations were conducted in simulation and real-world environments using cross-validation. The results showed that SSL-NBV required fewer views for plant reconstruction than non-NBV methods and was over 800 times faster than a voxel-based method. SSL-NBV reduced training annotations by over 90% compared to a baseline DL-NBV. Furthermore, SSL-NBV could adapt to novel scenarios through online fine-tuning. Also using real plants, the results showed that the proposed method can learn to effectively plan new viewpoints for 3D plant reconstruction. Most importantly, SSL-NBV automated the entire network training and uses continuous online learning, allowing it to operate in changing agricultural environments.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve the challenges encountered in 3D plant reconstruction, especially the occlusion problem caused by the complex shapes of plants. Specifically, the research proposes a Next - Best - View (NBV) algorithm based on self - supervised learning (SSL - NBV) for efficient 3D plant reconstruction. The following are the main problems that this paper attempts to solve: 1. **Complex plant structures and occlusions**: - The complex shapes and large number of occlusions in plants make it difficult for traditional 3D reconstruction methods to obtain complete information. To this end, the paper proposes an iterative method of selecting new viewpoints to maximize Information Gain (IG), thereby gradually reducing occlusions and improving reconstruction quality. 2. **Limitations of deep - learning methods**: - Current deep - learning - based NBV methods (DL - NBV) are computationally efficient but require a large amount of training data and rely on pre - collected data for offline training, which limits their application in dynamic agricultural environments. The paper proposes an online self - supervised learning method that enables robots to automatically collect and label training data during the execution of tasks, thus adapting to new plants and environments. 3. **Calculation of Information Gain**: - Calculating Information Gain (IG) is the key to NBV methods. Existing methods usually require a complete plant model to calculate the ground - truth IG, which is not feasible in practical applications. The paper proposes an improved IG calculation method based only on the data collected by the robot, without prior knowledge of the complete plant model. 4. **Online learning and sample efficiency**: - Existing DL - NBV methods mainly rely on offline training and cannot be continuously improved during the execution of tasks. In addition, these methods lack adaptability when dealing with new targets and environments. To solve these problems, the paper introduces the experience replay technique, which improves sample efficiency by reusing historical data, and combines weak - supervised learning to reduce the amount of IG labeling required and improve the collection efficiency of training data. 5. **Applications in agricultural environments**: - Agricultural environments are highly dynamic and variable, and existing DL - NBV methods are difficult to adapt to these changes. The SSL - NBV method proposed in the paper can perform online learning and adaptive adjustment in constantly changing agricultural environments, making it more suitable for practical application scenarios. ### Summary The core problem of this paper is to develop an NBV algorithm that can perform 3D plant reconstruction efficiently and adaptively, especially in complex and dynamic agricultural environments. By introducing self - supervised learning, weak - supervised learning, and experience replay techniques, the SSL - NBV method not only improves the quality and efficiency of reconstruction but also enhances the adaptability to new plants and environments.

SSL-NBV: A Self-Supervised-Learning-Based Next-Best-View algorithm for Efficient 3D Plant Reconstruction by a Robot

Gradient-based Local Next-best-view Planning for Improved Perception of Targeted Plant Nodes

A Double Branch Next-Best-View Network and Novel Robot System for Active Object Reconstruction

Attention-driven Next-best-view Planning for Efficient Reconstruction of Plants and Targeted Plant Parts

PC-NBV - A Point Cloud Based Deep Network for Efficient Next Best View Planning.

Plant-part Segmentation Using Deep Learning and Multi-View Vision

Non-destructive Plant Biomass Monitoring With High Spatio-Temporal Resolution via Proximal RGB-D Imagery and End-to-End Deep Learning

Research on automatic 3D reconstruction of plant phenotype based on Multi-View images

NBV/NBC Planning Considering Confidence Obtained From Shape Completion Learning

Pred-NBV: Prediction-guided Next-Best-View for 3D Object Reconstruction

MAP-NBV: Multi-agent Prediction-guided Next-Best-View Planning for Active 3D Object Reconstruction

NBV-SC: Next Best View Planning based on Shape Completion for Fruit Mapping and Reconstruction

GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

An Efficient Projection-Based Next-best-view Planning Framework for Reconstruction of Unknown Objects

NSLF-OL: Online Learning of Neural Surface Light Fields alongside Real-time Incremental 3D Reconstruction

3D-NVS: A 3D Supervision Approach for Next View Selection

A Global Max-Flow-Based Multi-Resolution Next-Best-View Method for Reconstruction of 3D Unknown Objects.

Structured Light-Based 3D Reconstruction System for Plants

High-fidelity 3D Reconstruction of Plants using Neural Radiance Field

SCVP: Learning One-Shot View Planning via Set Covering for Unknown Object Reconstruction

Contrastive 3D Shape Completion and Reconstruction for Agricultural Robots Using RGB-D Frames