3D Camera and Single-Point Laser Sensor Integration for Apple Localization in Spindle-Type Orchard Systems

R. M. Rasika D. Abeyrathna,Victor Massaki Nakaguchi,Zifu Liu,Rizky Mulya Sampurno,Tofael Ahamed
DOI: https://doi.org/10.3390/s24123753
IF: 3.9
2024-06-10
Sensors
Abstract:Accurate localization of apples is the key factor that determines a successful harvesting cycle in the automation of apple harvesting for unmanned operations. In this regard, accurate depth sensing or positional information of apples is required for harvesting apples based on robotic systems, which is challenging in outdoor environments because of uneven light variations when using 3D cameras for the localization of apples. Therefore, this research attempted to overcome the effect of light variations for the 3D cameras during outdoor apple harvesting operations. Thus, integrated single-point laser sensors for the localization of apples using a state-of-the-art model, the EfficientDet object detection algorithm with an mAP@0.5 of 0.775 were used in this study. In the experiments, a RealSense D455f RGB-D camera was integrated with a single-point laser ranging sensor utilized to obtain precise apple localization coordinates for implementation in a harvesting robot. The single-point laser range sensor was attached to two servo motors capable of moving the center position of the detected apples based on the detection ID generated by the DeepSORT (online real-time tracking) algorithm. The experiments were conducted under indoor and outdoor conditions in a spindle-type apple orchard artificial architecture by mounting the combined sensor system behind a four-wheel tractor. The localization coordinates were compared between the RGB-D camera depth values and the combined sensor system under different light conditions. The results show that the root-mean-square error (RMSE) values of the RGB-D camera depth and integrated sensor mechanism varied from 3.91 to 8.36 cm and from 1.62 to 2.13 cm under 476~600 lx to 1023~1100 × 100 lx light conditions, respectively. The integrated sensor system can be used for an apple harvesting robotic manipulator with a positional accuracy of ±2 cm, except for some apples that were occluded due to leaves and branches. Further research will be carried out using changes in the position of the integrated system for recognition of the affected apples for harvesting operations.
engineering, electrical & electronic,chemistry, analytical,instruments & instrumentation
What problem does this paper attempt to address?
This paper mainly discusses how to overcome the influence of outdoor lighting changes on the accuracy of 3D camera positioning in the process of apple picking automation. In the study, the authors proposed a new method combining 3D camera (RealSense D455f) and a single-point laser sensor to improve the positioning accuracy of apples in a spindle-shaped orchard system. They used an advanced object detection algorithm EfficientDet, which achieved a precision of 0.775 in terms of mAP@0.5. In the study, the single-point laser sensor was installed on two servo motors to adjust the laser alignment to the center of the apple based on the detection ID generated by the DeepSORT algorithm. The experiments were conducted under indoor and outdoor conditions, and the results showed that the root mean square error (RMSE) of the 3D camera ranged from 3.91 to 8.36 centimeters under different lighting conditions, while the RMSE of the integrated sensor system ranged from 1.62 to 2.13 centimeters, indicating that the integrated system can provide more accurate positioning coordinates. The paper also discusses the challenges of existing sensor fusion systems in terms of complexity and calibration, as well as their limitations under lighting changes. By combining the 3D camera with a laser rangefinder, the accuracy of apple positioning can be improved, which is suitable for robot picking systems with a positioning accuracy requirement of ±2 centimeters, although it may be affected in some cases due to the obstruction of leaves and branches. Future research plans will explore how to adjust the position of the integrated system to identify and handle obstructed apples, and further optimize the picking operation.