High-Precision Fruit Localization Using Active Laser-Camera Scanning: Robust Laser Line Extraction for 2D-3D Transformation

Pengyu Chu,Zhaojian Li,Kaixiang Zhang,Kyle Lammers,Renfu Lu

DOI: https://doi.org/10.1016/j.atech.2023.100391

2024-01-06

Smart Agricultural Technology

Abstract:Highlights • A novel Active Laser-Camera Scanning system,2 consisting of a red line laser, an RGB-D camera, and a linear motion slide, is designed, and developed for accurate fruit 3D localization. • A Laser Line Extraction (LLE) algorithm is proposed and implemented for robust feature matching to enable stable 2D-3D transformation for ALACS. • System evaluation and validation of the ALACS are performed indoors and outdoors in comparison with a conventional 3D sensing technique, i.e., Intel RealSense D435i RGB-Depth camera. Recent advancements in deep learning-based approaches have led to remarkable progress in fruit detection, enabling robust fruit identification in complex environments. However, much less progress has been made on fruit 3D localization, which is equally crucial for robotic harvesting. Complex fruit shape/orientation, fruit clustering, varying lighting conditions, and occlusions by leaves and branches have greatly restricted existing sensors from achieving accurate fruit localization in the natural orchard environment. In this paper, we report on the design of a novel localization technique, called Active Laser-Camera Scanning (ALACS), to achieve accurate and robust fruit 3D localization. The ALACS hardware setup comprises a red line laser, an RGB color camera, a linear motion slide, and an external RGB-D camera. Leveraging the principles of dynamic-targeting laser-triangulation, ALACS enables precise transformation of the projected 2D laser line from the surface of apples to the 3D positions. To facilitate laser pattern acquisitions, a Laser Line Extraction (LLE) method is proposed for robust and high-precision feature extraction on apples. Comprehensive evaluations of LLE demonstrated its ability to extract precise patterns under variable lighting and occlusion conditions. The ALACS system achieved average apple localization accuracies of 6.9 - 11.2 mm at distances ranging from 1.0 m to 1.6 m, compared to 21.5 mm by a commercial RealSense RGB-D camera, in an indoor experiment. Orchard evaluations demonstrated that ALACS has achieved a 95% fruit detachment rate versus a 71% rate by the RealSense camera. By overcoming the challenges of apple 3D localization, this research contributes to the advancement of robotic fruit harvesting technology.

English Else