DAVIS-Ag: A Synthetic Plant Dataset for Prototyping Domain-Inspired Active Vision in Agricultural Robots

Taeyeong Choi,Dario Guevara,Zifei Cheng,Grisha Bandodkar,Chonghan Wang,Brian N. Bailey,Mason Earles,Xin Liu

2024-07-01

Abstract:In agricultural environments, viewpoint planning can be a critical functionality for a robot with visual sensors to obtain informative observations of objects of interest (e.g., fruits) from complex structures of plant with random occlusions. Although recent studies on active vision have shown some potential for agricultural tasks, each model has been designed and validated on a unique environment that would not easily be replicated for benchmarking novel methods being developed later. In this paper, we introduce a dataset, so-called DAVIS-Ag, for promoting more extensive research on Domain-inspired Active VISion in Agriculture. To be specific, we leveraged our open-source "AgML" framework and 3D plant simulator of "Helios" to produce 502K RGB images from 30K densely sampled spatial locations in 632 synthetic orchards. Moreover, plant environments of strawberries, tomatoes, and grapes are considered at two different scales (i.e., Single-Plant and Multi-Plant). Useful labels are also provided for each image, including (1) bounding boxes and (2) instance segmentation masks for all identifiable fruits, and also (3) pointers to other images of the viewpoints that are reachable by an execution of action so as to simulate active viewpoint selections at each time step. Using DAVIS-Ag, we visualize motivating examples where fruit visibility can dramatically change depending on the pose of the camera view primarily due to occlusions by other components, such as leaves. Furthermore, we present several baseline models with experiment results for benchmarking in the task of target visibility maximization. Transferability to real strawberry environments is also investigated to demonstrate the feasibility of using the dataset for prototyping real-world solutions. For future research, our dataset is made publicly available online: <a class="link-external link-https" href="https://github.com/ctyeong/DAVIS-Ag" rel="external noopener nofollow">this https URL</a>.

Robotics

What problem does this paper attempt to address?

The paper attempts to address the problem of how to obtain clear views of targets (such as fruits) in agricultural environments through effective viewpoint planning of robotic vision sensors. Specifically, the paper points out that in agricultural environments, due to the complex structure of plants and the presence of occlusions, target objects (such as fruits, stems, leaves, etc.) are often only partially visible. This leads to difficulties in recognizing crop maturity or health status and inaccuracies in yield estimation.

To solve this problem, the authors propose a synthetic plant dataset named DA VIS-Ag, aimed at promoting extensive research on active vision in the agricultural field. This dataset contains over 502,000 high-resolution RGB images, captured from 30,000 densely sampled locations in 632 synthetic orchards, covering three plant environments: strawberries, tomatoes, and grapes. It provides useful labels for each image, including bounding boxes and instance segmentation masks of target fruits, as well as pointers to other reachable viewpoint images to simulate active viewpoint selection at each time step.

Through this dataset, the authors demonstrate that the visibility of target fruits under different viewpoints significantly changes due to occlusions by other components (such as leaves). They also provide experimental results of several benchmark models to evaluate the performance of the task of maximizing target visibility. Additionally, the authors tested the transferability of this dataset in a real strawberry environment to verify its feasibility and practicality in real-world applications.