Kiwifruit recognition method at night based on fruit calyx image
Fu Longsheng,Sun Shipeng,Vázquez-Arellano Manuel,Li Shifeng,Li Rui,Cui Yongjie
DOI: https://doi.org/10.11975/j.issn.1002-6819.2017.02.027
2017-01-01
Abstract:China is the largest country for cultivating kiwifruits, and Shaanxi Province provides the largest production, which accounts for approximately 70% of the nationwide production and 33% of the global production. Harvesting kiwifruits in this region relies mainly on manual picking which is labor-intensive. Therefore, the introduction of robotic harvesting is highly desirable and suitable. Most researches involved so far in kiwifruit harvesting robots suggest the scenario of daytime harvesting for taking advantage of the sunlight. Robot picking at night can overcome the problem of low work efficiency and will help to minimize fruit damage. In addition, artificial lights can be used to ensure constant illumination instead of the variable natural sunlight for image acquisition. The study object of this paper was a kiwifruit recognition system at night using artificial lighting by identifying the fruit calyx. According to kiwifruits’ growth characteristics, which were grown on sturdy support structures, an RGB (red, green, blue) camera was placed underneath the canopy so that kiwifruits clusters could be included in the images. An image processing algorithm was developed to recognize kiwifruits by identifying the fruit’s calyx. Firstly, it subtracted 1.1R-G gray image, and then segmentation was done using the Otsu method for the thresholding. A morphological operation was applied to remove the noise that adhered to the target fruits (such as branches). Afterwards, an area thresholding method was employed to eliminate the remaining noises. This method is based on finding the biggest area of neighboring white pixels in the image and eliminating all areas which are smaller than 1/5 of the biggest area. Using this image as the mask, a fruit image without background was obtained. After that, V (value) component of HSV (hue, saturation, value) color model was calculated for segmenting the fruit’s calyx from the fruit, also using the Otsu method for thresholding. Black areas were then labeled and sorted by their pixels numbers. The first largest black area was the image background and the second largest black areas was a fruit calyx area that used as the reference area. Since the fruit calyx areas varies in a small range in one image, the fruit calyx areas are judged by comparing with the reference area. If a black area in the image was smaller than the reference area and larger than 1/10 of the reference area, it is a fruit calyx; otherwise, it is not. Finally, the nearest edge pixel for each fruit’s calyx was searched and their distance calculated was as radius, and a circle around the fruit calyx was drawn. Finally, the algorithm was also tested for the robustness under 12 different light illuminations (10, 30, 50, 80, 110, 150, 200, 300, 400, 500, 800 and 1 200 lux). The fruits illumination was estimated by averaging the illumination values, which were measured for 3 times at 3 different locations around the target fruit cluster. Results showed that the image processing algorithm based on the calyx could recognize kiwifruit and reached a success rate of 94.3%. Undetected and wrongly detected fruits appeared mostly at the same cluster where one fruit was adjacent to 3 or more fruits. The calyxes of those fruits sometimes were not in the centers of their fruits’ images, thus, causing undetected fruits. Those fruits also formed dark areas among them, which were wrongly recognized as calyx. On the other hand, most clusters were linearly arranged on the branches, which made them suitable for the proposed algorithm. The algorithm was robust to different illuminations although the success rates were slightly decreased under extremely weak or strong illuminations. It only took 0.5 s in average to recognize a fruit, which is a great step toward filed robotic harvesting of kiwifruit.