Abstract:Recognition and localization of fruits are key components to achieve automated fruit picking. However, current neural-network-based fruit recognition algorithms have disadvantages such as high complexity. Traditional stereo matching algorithms also have low accuracy. To solve these problems, this study targeting greenhouse tomatoes proposed an algorithm framework based on YOLO-TomatoSeg, a lightweight tomato instance segmentation model improved from YOLOv5n-seg, and an accurate tomato localization approach using RAFT-Stereo disparity estimation and least squares point cloud fitting. First, binocular tomato images were captured using a binocular camera system. The left image was processed by YOLO-TomatoSeg to segment tomato instances and generate masks. Concurrently, RAFT-Stereo estimated image disparity for computing the original depth point cloud. Then, the point cloud was clipped by tomato masks to isolate tomato point clouds, which were further preprocessed. Finally, a least squares sphere fitting method estimated the 3D centroid co-ordinates and radii of tomatoes by fitting the tomato point clouds to spherical models. The experimental results showed that, in the tomato instance segmentation stage, the YOLO-TomatoSeg model replaced the Backbone network of YOLOv5n-seg with the building blocks of ShuffleNetV2 and incorporated an SE attention module, which reduced model complexity while improving model segmentation accuracy. Ultimately, the YOLO-TomatoSeg model achieved an AP of 99.01% with a size of only 2.52 MB, significantly outperforming mainstream instance segmentation models such as Mask R-CNN (98.30% AP) and YOLACT (96.49% AP). The model size was reduced by 68.3% compared to the original YOLOv5n-seg model. In the tomato localization stage, at the range of 280 mm to 480 mm, the average error of the tomato centroid localization was affected by occlusion and sunlight conditions. The maximum average localization error was ±5.0 mm, meeting the localization accuracy requirements of the tomato-picking robots. This study developed a lightweight tomato instance segmentation model and achieved accurate localization of tomato, which can facilitate research, development, and application of fruit-picking robots.

Multi-vision-based Localization and Pose Estimation of Occluded Apple Fruits for Harvesting Robots

Recognition and Localization Method of Overlapping Apples for Apple Harvesting Robot

Occluded Apple Fruit Detection and Localization with a Frustum-Based Point-Cloud-Processing Approach for Robotic Harvesting

Computer Vision-Based Localisation of Picking Points for Automatic Litchi Harvesting Applications Towards Natural Scenarios

Real-Time Fruit Recognition and Grasping Estimation for Autonomous Apple Harvesting

Fruit Detection and Pose Estimation for Grape Cluster-Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks

Geometry-Aware Fruit Grasping Estimation for Robotic Harvesting in Orchards

The interaction between KSHV RTA and cellular RBP-Jkappa and their subsequent DNA binding are not sufficient for activation of RBP-Jkappa.

Visual Perception and Modelling in Unstructured Orchard for Apple Harvesting Robots

Detection and localization of citrus fruit based on improved You Only Look Once v5s and binocular vision in the orchard

Accurate Fruit Localisation for Robotic Harvesting using High Resolution LiDAR-Camera Fusion

Enhanced 6D Pose Estimation for Robotic Fruit Picking

Semantic Segmentation of Fruits on Multi-sensor Fused Data in Natural Orchards

Recognition and Localization Methods for Vision-Based Fruit Picking Robots: A Review

Deep Learning-Based Real-Time 6D Pose Estimation and Multi-Mode Tracking Algorithms for Citrus-Harvesting Robots

Panoptic Mapping with Fruit Completion and Pose Estimation for Horticultural Robots

Chondrules with Peculiar REE Patterns: Implications for Solar Nebular Condensation at High C/O

On-tree apple fruit size estimation using stereo vision with deep learning-based occlusion handling

Fruit Detection, Segmentation and 3D Visualisation of Environments in Apple Orchards

Fruit Detection and Counting in Apple Orchards Based on Improved Yolov7 and Multi-Object Tracking Methods

Tomato Recognition and Localization Method Based on Improved YOLOv5n-seg Model and Binocular Stereo Vision