Abstract:In orchard fruit picking systems for pears, the challenge is to identify the full shape of the soft fruit to avoid injuries while using robotic or automatic picking systems. Advancements in computer vision have brought the potential to train for different shapes and sizes of fruit using deep learning algorithms. In this research, a fruit recognition method for robotic systems was developed to identify pears in a complex orchard environment using a 3D stereo camera combined with Mask Region-Convolutional Neural Networks (Mask R-CNN) deep learning technology to obtain targets. This experiment used 9054 RGBA original images (3018 original images and 6036 augmented images) to create a dataset divided into a training, validation, and testing sets. Furthermore, we collected the dataset under different lighting conditions at different times which were high-light (9–10 am) and low-light (6–7 pm) conditions at JST, Tokyo Time, August 2021 (summertime) to prepare training, validation, and test datasets at a ratio of 6:3:1. All the images were taken by a 3D stereo camera which included PERFORMANCE, QUALITY, and ULTRA models. We used the PERFORMANCE model to capture images to make the datasets; the camera on the left generated depth images and the camera on the right generated the original images. In this research, we also compared the performance of different types with the R-CNN model (Mask R-CNN and Faster R-CNN); the mean Average Precisions (mAP) of Mask R-CNN and Faster R-CNN were compared in the same datasets with the same ratio. Each epoch in Mask R-CNN was set at 500 steps with total 80 epochs. And Faster R-CNN was set at 40,000 steps for training. For the recognition of pears, the Mask R-CNN, had the mAPs of 95.22% for validation set and 99.45% was observed for the testing set. On the other hand, mAPs were observed 87.9% in the validation set and 87.52% in the testing set using Faster R-CNN. The different models using the same dataset had differences in performance in gathering clustered pears and individual pear situations. Mask R-CNN outperformed Faster R-CNN when the pears are densely clustered at the complex orchard. Therefore, the 3D stereo camera-based dataset combined with the Mask R-CNN vision algorithm had high accuracy in detecting the individual pears from gathered pears in a complex orchard environment.

Fruit position recognition and ripeness determination methods for a fruit automatic harvesting robot

Recognition and Localization Method of Overlapping Apples for Apple Harvesting Robot

Dynamic Recognition of Oscillating Fruit for Harvesting Robot

Fast positioning method of apple harvesting robot for oscillating fruit

Automated harvesting by a dual-arm fruit harvesting robot

Occluded Apple Fruit Detection and Localization with a Frustum-Based Point-Cloud-Processing Approach for Robotic Harvesting

Robotic Harvesting of the Occluded Fruits with a Precise Shape and Position Reconstruction Approach

Detecting ripe fruits under natural occlusion and illumination conditions

Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field

Intelligent robotics harvesting system process for fruits grasping prediction

Fruit Detection and Pose Estimation for Grape Cluster-Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks

A Method of Robot Picking Citrus Based on 3D Detection

Towards vision-based dual arm robotic fruit harvesting

Advance of Target Visual Information Acquisition Technology for Fresh Fruit Robotic Harvesting: A Review

Recognition and Counting of Apples in a Dynamic State Using a 3D Camera and Deep Learning Algorithms for Robotic Harvesting Systems

Real-Time Fruit Recognition and Grasping Estimation for Autonomous Apple Harvesting

Fruit growing direction recognition and nesting grasping strategies for tomato harvesting robots

Fruit location and stem detection method for strawberry harvesting robot

Crop Recognition Method Based On End-effector

Pear Recognition in an Orchard from 3D Stereo Camera Datasets to Develop a Fruit Picking Mechanism Using Mask R-CNN

Recognition and Localization Methods for Vision-Based Fruit Picking Robots: A Review