Abstract:urate peach detection is a prerequisite for automated agronomic management, e.g., peach mechanical harvesting. However, due to uneven illumination and ubiquitous occlusion, it is challenging to detect the peaches, especially when the peaches are bagged in orchards. To this end, an accurate multi-class peach detection method was proposed by means of improving YOLOv5s and using multi-modal visual data for mechanical harvesting in this paper. RGB-D dataset with multi-class annotations of naked and bagging peach was proposed, including 4127 multimodal images of corresponding pixel-aligned color, depth, and infrared images acquired with consumer-level RGBD camera. Subsequently, an improved lightweight YOLOv5s (small depth) model was put forward by introducing a direction-aware and position-sensitive attention mechanism, which could capture long-range dependencies along one spatial direction and preserve precise positional information along the other spatial direction, helping the networks accurately detect peach targets. Meanwhile, the depthwise separable convolution was employed to reduce the model computation by decomposing the convolution operation into convolution in the depth direction and convolution in the width and height directions, which helped to speed up the training and inference of the network while maintaining accuracy. The comparison experimental results demonstrated that the improved YOLOv5s using multimodal visual data recorded the detection mAP of 98.6% and 88.9% on the naked and bagging peach with 5.05 M model parameters in complex illumination and severe occlusion environment, increasing by 5.3% and 16.5% than only using RGB images, as well as by 2.8% and 6.2% when compared to YOLOv5s. As compared with other networks in detecting bagging peaches, the improved YOLOv5s performed best in terms of mAP, which was 16.3%, 8.1% and 4.5% higher than YOLOX-Nano, PP-YOLO-Tiny, and EfficientDet-D0, respectively. In addition, the proposed improved YOLOv5s model offered better results in different degrees than other methods in detecting Fuji apple and Hayward kiwifruit, verified the effectiveness on different fruit detection tasks. Further investigation revealed the contribution of each imaging modality, as well as the proposed improvement in YOLOv5s, to favorable detection results of both naked and bagging peaches in natural orchards. Additionally, on the popular mobile hardware plat‐form, it was found out that the improved YOLOv5s model could implement 19 times detection per second with the considered five-channel multi-modal images, offering real-time peach detection. These promising results demonstrated the potential of the improved YOLOv5s and multi-modal visual data with multi-class annotations to achieve visual intelligence of automated fruit harvesting systems.

Peaches Detection Using a Deep Learning Technique—A Contribution to Yield Estimation, Resources Management, and Circular Economy

Detection of Strawberry Flowers in Outdoor Field by Deep Neural Network

Accurate Strawberry Plant Detection System Based on Low-altitude Remote Sensing and Deep Learning Technologies

DeepFruits: A Fruit Detection System Using Deep Neural Networks

CRLNet: A Multimodal Peach Detection Network Based on Cooperative Asymptotic Enhancement and the Fusion of Granularity Refinement

Detection of multi-size peach in orchard using RGB-D camera combined with an improved DEtection Transformer model

Fruit Detection and Recognition Based on Deep Learning for Automatic Harvesting: An Overview and Review

Apple Ripeness Identification Using Deep Learning

Tomato Fruit Detection and Counting in Greenhouses Using Deep Learning

Single-Shot Convolution Neural Networks for Real-Time Fruit Detection Within the Tree

Multi-Class on-Tree Peach Detection Using Improved YOLOv5s and Multi-Modal Images

A Novel Greenhouse-Based System for the Detection and Plumpness Assessment of Strawberry Using an Improved Deep Learning Technique

Faster R-CNN for Multi-Class Fruit Detection Using a Robotic Vision System

Integrated Fruit Ripeness Assessment System Based on an Artificial Olfactory Sensor and Deep Learning

A Robust Deep Learning Approach for the Quantitative Characterization and Clustering of Peach Tree Crowns Based on UAV Images

CNN-based object detection and growth estimation of plum fruit (Prunus mume) using RGB and depth imaging techniques

A Method of Robot Picking Citrus Based on 3D Detection

Application of Convolutional Neural Network-Based Detection Methods in Fresh Fruit Production: A Comprehensive Review

Identification and Location Algorithm of Fragrant Pear Fruit and Its Application in Automatic Picking System

Pear Recognition in an Orchard from 3D Stereo Camera Datasets to Develop a Fruit Picking Mechanism Using Mask R-CNN

Real-time Fruit Detection Method Based on RGB-D Image Fusion