Abstract:urate peach detection is a prerequisite for automated agronomic management, e.g., peach mechanical harvesting. However, due to uneven illumination and ubiquitous occlusion, it is challenging to detect the peaches, especially when the peaches are bagged in orchards. To this end, an accurate multi-class peach detection method was proposed by means of improving YOLOv5s and using multi-modal visual data for mechanical harvesting in this paper. RGB-D dataset with multi-class annotations of naked and bagging peach was proposed, including 4127 multimodal images of corresponding pixel-aligned color, depth, and infrared images acquired with consumer-level RGBD camera. Subsequently, an improved lightweight YOLOv5s (small depth) model was put forward by introducing a direction-aware and position-sensitive attention mechanism, which could capture long-range dependencies along one spatial direction and preserve precise positional information along the other spatial direction, helping the networks accurately detect peach targets. Meanwhile, the depthwise separable convolution was employed to reduce the model computation by decomposing the convolution operation into convolution in the depth direction and convolution in the width and height directions, which helped to speed up the training and inference of the network while maintaining accuracy. The comparison experimental results demonstrated that the improved YOLOv5s using multimodal visual data recorded the detection mAP of 98.6% and 88.9% on the naked and bagging peach with 5.05 M model parameters in complex illumination and severe occlusion environment, increasing by 5.3% and 16.5% than only using RGB images, as well as by 2.8% and 6.2% when compared to YOLOv5s. As compared with other networks in detecting bagging peaches, the improved YOLOv5s performed best in terms of mAP, which was 16.3%, 8.1% and 4.5% higher than YOLOX-Nano, PP-YOLO-Tiny, and EfficientDet-D0, respectively. In addition, the proposed improved YOLOv5s model offered better results in different degrees than other methods in detecting Fuji apple and Hayward kiwifruit, verified the effectiveness on different fruit detection tasks. Further investigation revealed the contribution of each imaging modality, as well as the proposed improvement in YOLOv5s, to favorable detection results of both naked and bagging peaches in natural orchards. Additionally, on the popular mobile hardware plat‐form, it was found out that the improved YOLOv5s model could implement 19 times detection per second with the considered five-channel multi-modal images, offering real-time peach detection. These promising results demonstrated the potential of the improved YOLOv5s and multi-modal visual data with multi-class annotations to achieve visual intelligence of automated fruit harvesting systems.

Detection in Complex Scenes Using Rgb and Depth Multimodal Feature Fusion.

MFF-Net: Towards Efficient Monocular Depth Completion With Multi-Modal Feature Fusion

Improving real-time apple fruit detection: Multi-modal data and depth fusion with non-targeted background removal

Real-time Fruit Detection Method Based on RGB-D Image Fusion

Recognition of multi-modal fusion images with irregular interference

MLP-based multimodal tomato detection in complex scenarios: Insights from task-specific analysis of feature fusion architectures

DeepFruits: A Fruit Detection System Using Deep Neural Networks

CRLNet: A Multimodal Peach Detection Network Based on Cooperative Asymptotic Enhancement and the Fusion of Granularity Refinement

Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios

Discriminative feature fusion for RGB-D salient object detection

Detection of multi-size peach in orchard using RGB-D camera combined with an improved DEtection Transformer model

RTMFusion: an Enhanced Dual-Stream Architecture Algorithm Fusing RGB and Depth Features for Instance Segmentation of Tomato Organs

Improved Kiwifruit Detection Using Pre-Trained VGG16 with RGB and NIR Information Fusion

Lightweight and efficient deep learning models for fruit detection in orchards

CMPFFNet: Cross-Modal and Progressive Feature Fusion Network for RGB-D Indoor Scene Semantic Segmentation

Detection of Dense Citrus Fruits by Combining Coordinated Attention and Cross-Scale Connection with Weighted Feature Fusion

A Dataset of Multi-Modal Peach Images for Object Detection

Multi-Modal Fusion Based on Depth Adaptive Mechanism for 3D Object Detection

Multi-Class on-Tree Peach Detection Using Improved YOLOv5s and Multi-Modal Images

A Cross-Modal Feature Fusion Model Based on ConvNeXt for RGB-D Semantic Segmentation

High-Quality Coarse-to-Fine Fruit Detector for Harvesting Robot in Open Environment