Abstract:Intelligent detection and localization of mature citrus fruits is a critical challenge in developing an automatic harvesting robot. Variable illumination conditions and different occlusion states are some of the essential issues that must be addressed for the accurate detection and localization of citrus in the orchard environment. In this paper, a novel method for the detection and localization of mature citrus using improved You Only Look Once (YOLO) v5s with binocular vision is proposed. First, a new loss function (polarity binary cross-entropy with logit loss) for YOLO v5s is designed to calculate the loss value of class probability and objectness score, so that a large penalty for false and missing detection is applied during the training process. Second, to recover the missing depth information caused by randomly overlapping background participants, Cr-Cb chromatic mapping, the Otsu thresholding algorithm, and morphological processing are successively used to extract the complete shape of the citrus, and the kriging method is applied to obtain the best linear unbiased estimator for the missing depth value. Finally, the citrus spatial position and posture information are obtained according to the camera imaging model and the geometric features of the citrus. The experimental results show that the recall rates of citrus detection under non-uniform illumination conditions, weak illumination, and well illumination are 99.55%, 98.47%, and 98.48%, respectively, approximately 2–9% higher than those of the original YOLO v5s network. The average error of the distance between the citrus fruit and the camera is 3.98 mm, and the average errors of the citrus diameters in the 3D direction are less than 2.75 mm. The average detection time per frame is 78.96 ms. The results indicate that our method can detect and localize citrus fruits in the complex environment of orchards with high accuracy and speed. Our dataset and codes are available at https://github.com/AshesBen/citrus-detection-localization.

Safe Leaf Manipulation for Accurate Shape and Pose Estimation of Occluded Fruits

Fruit Mapping with Shape Completion for Autonomous Crop Monitoring

Pose Estimation and Adaptable Grasp Configuration with Point Cloud Registration and Geometry Understanding for Fruit Grasp Planning

Panoptic Mapping with Fruit Completion and Pose Estimation for Horticultural Robots

Multi-vision-based Localization and Pose Estimation of Occluded Apple Fruits for Harvesting Robots

Robotic Harvesting of the Occluded Fruits with a Precise Shape and Position Reconstruction Approach

Learning-Based Slip Detection for Robotic Fruit Grasping and Manipulation under Leaf Interference

Efficient Occlusion Avoidance Based on Active Deep Sensing for Harvesting Robots

Monocular Pose Estimation Method for Automatic Citrus Harvesting Using Semantic Segmentation and Rotating Target Detection

An Integrated Actuation-Perception Framework for Robotic Leaf Retrieval: Detection, Localization, and Cutting

Occluded Apple Fruit Detection and Localization with a Frustum-Based Point-Cloud-Processing Approach for Robotic Harvesting

Citrus Pose Estimation from an RGB Image for Automated Harvesting

Application of amodal segmentation for shape reconstruction and occlusion recovery in occluded tomatoes

Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field

A Dataset and Benchmark for Shape Completion of Fruits for Agricultural Robotics

An improved YOLOv5s model for assessing apple graspability in automated harvesting scene

Fruit Pose Recognition and Directional Orderly Grasping Strategies for Tomato Harvesting Robots

An Obstacle Separation Method for Robotic Picking of Fruits in Clusters

Robotic Lime Picking by Considering Leaves as Permeable Obstacles

Fruit Detection and Pose Estimation for Grape Cluster-Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks

Detection and localization of citrus fruit based on improved You Only Look Once v5s and binocular vision in the orchard