Abstract:Intelligent detection and localization of mature citrus fruits is a critical challenge in developing an automatic harvesting robot. Variable illumination conditions and different occlusion states are some of the essential issues that must be addressed for the accurate detection and localization of citrus in the orchard environment. In this paper, a novel method for the detection and localization of mature citrus using improved You Only Look Once (YOLO) v5s with binocular vision is proposed. First, a new loss function (polarity binary cross-entropy with logit loss) for YOLO v5s is designed to calculate the loss value of class probability and objectness score, so that a large penalty for false and missing detection is applied during the training process. Second, to recover the missing depth information caused by randomly overlapping background participants, Cr-Cb chromatic mapping, the Otsu thresholding algorithm, and morphological processing are successively used to extract the complete shape of the citrus, and the kriging method is applied to obtain the best linear unbiased estimator for the missing depth value. Finally, the citrus spatial position and posture information are obtained according to the camera imaging model and the geometric features of the citrus. The experimental results show that the recall rates of citrus detection under non-uniform illumination conditions, weak illumination, and well illumination are 99.55%, 98.47%, and 98.48%, respectively, approximately 2–9% higher than those of the original YOLO v5s network. The average error of the distance between the citrus fruit and the camera is 3.98 mm, and the average errors of the citrus diameters in the 3D direction are less than 2.75 mm. The average detection time per frame is 78.96 ms. The results indicate that our method can detect and localize citrus fruits in the complex environment of orchards with high accuracy and speed. Our dataset and codes are available at https://github.com/AshesBen/citrus-detection-localization.

Fruit detection in natural environment using partial shape matching and probabilistic Hough transform

Detecting ripe fruits under natural occlusion and illumination conditions

Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field

Method for fruit shape feature acquisition based on multidirectional vision

Immature Green Citrus Detection by Multiple Features Using Machine Vision

Detection Of Young Green Apples In Orchard Environment Using Adaptive Ratio Chromatic Aberration And Hog-Svm

High-Quality Coarse-to-Fine Fruit Detector for Harvesting Robot in Open Environment

Design of Real-time Detection System for Fruits Appearance Quality

Detecting citrus fruits and occlusion recovery under natural illumination conditions

Detection of Young Green Apples for Fruit Robot in Natural Scene

Fruit Detection and Counting in Apple Orchards Based on Improved Yolov7 and Multi-Object Tracking Methods

Fruit Shape Detection by Level Set

Detection and localization of citrus fruit based on improved You Only Look Once v5s and binocular vision in the orchard

Ripe Tomato Detection for Robotic Vision Harvesting Systems in Greenhouses

Immature Green Citrus Fruit Detection and Counting Based on Fast Normalized Cross Correlation (FNCC) Using Natural Outdoor Colour Images

A nearest neighbor approach for fruit recognition in RGB-D images based on detection of convex surfaces

Detection of orchard citrus fruits using a monocular machine vision-based method for automatic fruit picking applications

DeepFruits: A Fruit Detection System Using Deep Neural Networks

Detecting Citrus Fruits with Highlight on Tree Based on Fusion of Multi-Map

Mature pomegranate fruit detection and location combining improved F-PointNet with 3D point cloud clustering in orchard

A Method for Organs Classification and Fruit Counting on Pomegranate Trees Based on Multi-Features Fusion and Support Vector Machine by 3D Point Cloud