FruitNeRF: A Unified Neural Radiance Field based Fruit Counting Framework

Lukas Meyer,Andreas Gilson,Ute Schmid,Marc Stamminger

2024-09-26

Abstract:We introduce FruitNeRF, a unified novel fruit counting framework that leverages state-of-the-art view synthesis methods to count any fruit type directly in 3D. Our framework takes an unordered set of posed images captured by a monocular camera and segments fruit in each image. To make our system independent of the fruit type, we employ a foundation model that generates binary segmentation masks for any fruit. Utilizing both modalities, RGB and semantic, we train a semantic neural radiance field. Through uniform volume sampling of the implicit Fruit Field, we obtain fruit-only point clouds. By applying cascaded clustering on the extracted point cloud, our approach achieves precise fruit <a class="link-external link-http" href="http://count.The" rel="external noopener nofollow">this http URL</a> use of neural radiance fields provides significant advantages over conventional methods such as object tracking or optical flow, as the counting itself is lifted into 3D. Our method prevents double counting fruit and avoids counting irrelevant <a class="link-external link-http" href="http://fruit.We" rel="external noopener nofollow">this http URL</a> evaluate our methodology using both real-world and synthetic datasets. The real-world dataset consists of three apple trees with manually counted ground truths, a benchmark apple dataset with one row and ground truth fruit location, while the synthetic dataset comprises various fruit types including apple, plum, lemon, pear, peach, and <a class="link-external link-http" href="http://mango.Additionally" rel="external noopener nofollow">this http URL</a>, we assess the performance of fruit counting using the foundation model compared to a U-Net.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address the problem of fruit counting in precision agriculture, particularly achieving accurate detection and counting of various types of fruits under different lighting conditions, partial occlusions, and visibility issues. Specifically, the study proposes a new framework called FruitNeRF, which is based on Neural Radiance Fields (NeRF) technology and can directly count any type of fruit in 3D space without relying on specific fruit types or environments.

The main contributions of FruitNeRF include:
1. **Proposing a novel method**: Utilizing semantic NeRF for fruit counting from unordered image sets, which has good generalization ability for different types of fruits.
2. **Releasing a dataset**: Including synthetic data of various fruit trees and a real-world dataset focused on apple trees.
3. **Open-source code**: Allowing other researchers to replicate the experimental results and further improve the algorithm.

By comparing the base model with specialized networks such as U-Net, the effectiveness and robustness of FruitNeRF in different scenarios were validated. Additionally, this method avoids the issue of double counting present in traditional methods and can exclude irrelevant fruit interference in the background, thereby improving counting accuracy.