Abstract:Accurate positioning of kiwifruit in the field growth environment is crucial for kiwifruit harvesting robots to carry out precise and non-destructive harvesting. To achieve effective detection and accurate positioning of small and dense kiwifruits in complex background environments, this study proposed a spatial coordinate positioning method for kiwifruits based on the stereo imaging principle of the binocular camera + deep learning. The proposed method on the basis of the detected fruit, combined with the acquired fruit depth, and using the stereo imaging principle of the binocular camera, the proposed method can effectively predict the 3D spatial coordinates of the small and dense kiwifruits in the complex background environment. First, by adding the SENet attention mechanism module after the four E-ELAN feature layers of the backbone feature extraction network and the SPPCSPC module in the neck; therefore, the kiwifruit feature information extraction was enhanced, and the detection accuracy was improved. Next, the detected kiwifruit contour was obtained and fitted using the contour detection algorithm to generate the centre of gravity (COG), which was then combined with the internal and external parameters of the ZED Mini camera to determine the distance between the COG of the fruit and the camera (depth information). Finally, using the transformation relationship between the pixel coordinate system and the world coordinate system, the three-dimensional (3D) spatial coordinates of kiwifruits were obtained using the stereoscopic images of the binocular camera. To verify the effectiveness of the proposed method, the effectiveness of Im-YOLOv7 and other four models YOLOv7, YOLOv5s, YOLOv4 and SSD on the detection results was compared and analysed. The results showed that the accuracy of Im-YOLOv7 is 6.87 %, 2.12 %, 8.17 %, and 5.58 % higher than the original YOLOv7, YOLOv5s, YOLOv4, and SSD, respectively. Its detection speed is 19.35 %, 12.55 %, 32.14 %, and 35.16 % faster than the original YOLOv7, YOLOv5s, YOLOv4, and SSD, respectively. Kiwifruit positioning experiments showed that, according to the method proposed in this study, the average positioning errors (Delta x, Delta y, and Delta z) were 3.09, 2.91, and 3.03 mm, respectively, and the positioning error of each axis was less than standard 5 mm. The proposed method is feasible to perform real-time and accurate detection and spatial positioning of kiwifruits. This study can provide a technical reference for the development of intelligent kiwifruit-picking devices.

Research on Spatial Positioning System of Fruits to be Picked in Field Based on Binocular Vision and SSD Model

Object Detection and Spatial Positioning of Kiwifruits in a Wide-Field Complex Environment

High-Quality Coarse-to-Fine Fruit Detector for Harvesting Robot in Open Environment

Pineapple (Ananas comosus) fruit detection and localization in natural environment based on binocular stereo vision and improved YOLOv3 model

Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field

Three-dimensional Perception of Orchard Banana Central Stock Enhanced by Adaptive Multi-Vision Technology.

Real-time Fruit Detection Method Based on RGB-D Image Fusion

Spectral Imaging for Greenhouse Cucumber Fruit Detection Based on Binocular Stereovision

Fruit Detection, Segmentation and 3D Visualisation of Environments in Apple Orchards

Faster R-CNN for Multi-Class Fruit Detection Using a Robotic Vision System

Fruit Detection and Pose Estimation for Grape Cluster-Harvesting Robot Using Binocular Imagery Based on Deep Neural Networks

Recognition and Localization Methods for Vision-Based Fruit Picking Robots: A Review

Integrated detection of citrus fruits and branches using a convolutional neural network

DeepFruits: A Fruit Detection System Using Deep Neural Networks

Analysis Of Deep Learning-Based Visual Perception Technology for Picking Robots

A Real-Time Detection Framework For On-Tree Mango Based On Ssd Network

A Method of Robot Picking Citrus Based on 3D Detection

Robotic Harvesting of the Occluded Fruits with a Precise Shape and Position Reconstruction Approach

An Accurate Detection and Segmentation Model of Obscured Green Fruits

A Refined Apple Binocular Positioning Method with Segmentation-Based Deep Learning for Robotic Picking

An Improved Obstacle Separation Method Using Deep Learning for Object Detection and Tracking in a Hybrid Visual Control Loop for Fruit Picking in Clusters