Abstract:The proposed project aims to enhance tree fruit identification and targeting for robotic harvesting through the selection of appropriate sensor technology, sensor fusion, and visual servo-control approaches. These technologies will be applicable for apple, orange and grapefruit harvest, although specific sensor wavelengths may vary. The primary challenges are fruit occlusion, light variability, peel color variation with maturity, range to target, and computational requirements of image processing algorithms. There are four major development tasks in original three-year proposed study. First, spectral characteristics in the VIS/NIR (0.4-1.0 micron) will be used in conjunction with thermal data to provide accurate and robust detection of fruit in the tree canopy. Hyper-spectral image pairs will be combined to provide automatic stereo matching for accurate 3D position. Secondly, VIS/NIR/FIR (0.4-15.0 micron) spectral sensor technology will be evaluated for potential in-field on-the-tree grading of surface defect, maturity and size for selective fruit harvest. Thirdly, new adaptive Lyapunov-basedHBVS (homography-based visual servo) methods to compensate for camera uncertainty, distortion effects, and provide range to target from a single camera will be developed, simulated, and implemented on a camera testbed to prove concept. HBVS methods coupled with imagespace navigation will be implemented to provide robust target tracking. And finally, harvesting test will be conducted on the developed technologies using the University of Florida harvesting manipulator test bed. During the course of the project it was determined that the second objective was overly ambitious for the project period and effort was directed toward the other objectives. The results reflect the synergistic efforts of the three principals. The USA team has focused on citrus based approaches while the Israeli counterpart has focused on apples. The USA team has improved visual servo control through the use of a statistical-based range estimate and homography. The results have been promising as long as the target is visible. In addition, the USA team has developed improved fruit detection algorithms that are robust under light variation and can localize fruit centers for partially occluded fruit. Additionally, algorithms have been developed to fuse thermal and visible spectrum image prior to segmentation in order to evaluate the potential improvements in fruit detection. Lastly, the USA team has developed a multispectral detection approach which demonstrated fruit detection levels above 90% of non-occluded fruit. The Israel team has focused on image registration and statistical based fruit detection with post-segmentation fusion. The results of all programs have shown significant progress with increased levels of fruit detection over prior art.

Controlled Lighting and Illumination-Independent Target Detection for Real-Time Cost-Efficient Applications. The Case Study of Sweet Pepper Robotic Harvesting

Dataset and Performance Comparison of Deep Learning Architectures for Plum Detection and Robotic Harvesting

Fast and Accurate Green Pepper Detection in Complex Backgrounds Via an Improved Yolov4-tiny Model

Fruit Quantity and Quality Estimation using a Robotic Vision System

Pepper to fall: a perception method for sweet pepper robotic harvesting

In-Field Peduncle Detection of Sweet Peppers for Robotic Harvesting: a comparative study

Research on Instance Segmentation Algorithm of Greenhouse Sweet Pepper Detection Based on Improved Mask RCNN

Real-Time Recognition and Localization Based on Improved YOLOv5s for Robot’s Picking Clustered Fruits of Chilies

Recognition and Counting of Apples in a Dynamic State Using a 3D Camera and Deep Learning Algorithms for Robotic Harvesting Systems

A Real-Time Zanthoxylum Target Detection Method for an Intelligent Picking Robot under a Complex Background, Based on an Improved YOLOv5s Architecture

High-Speed Detector For Low-Powered Devices In Aerial Grasping

Lessons Learnt from Field Trials of a Robotic Sweet Pepper Harvester

Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field

Faster R–CNN–based Apple Detection in Dense-Foliage Fruiting-Wall Trees Using RGB and Depth Features for Robotic Harvesting

Enhancement of Sensing Technologies for Selective Tree Fruit Identification and Targeting in Robotic Harvesting Systems

Intelligent robotics harvesting system process for fruits grasping prediction

Advance of Target Visual Information Acquisition Technology for Fresh Fruit Robotic Harvesting: A Review

Autonomous Sweet Pepper Harvesting for Protected Cropping Systems

Research on Detection Method of Chaotian Pepper in Complex Field Environments Based on YOLOv8

Towards Real Scenery: A Light-weight Tomato Growth Inspection Algorithm for Leaf Disease Detection and Fruit Counting