MetaFood CVPR 2024 Challenge on Physically Informed 3D Food Reconstruction: Methods and Results

Jiangpeng He,Yuhao Chen,Gautham Vinod,Talha Ibn Mahmud,Fengqing Zhu,Edward Delp,Alexander Wong,Pengcheng Xi,Ahmad AlMughrabi,Umair Haroon,Ricardo Marques,Petia Radeva,Jiadong Tang,Dianyi Yang,Yu Gao,Zhaoxiang Liang,Yawei Jueluo,Chengyu Shi,Pengyu Wang

2024-07-12

Abstract:The increasing interest in computer vision applications for nutrition and dietary monitoring has led to the development of advanced 3D reconstruction techniques for food items. However, the scarcity of high-quality data and limited collaboration between industry and academia have constrained progress in this field. Building on recent advancements in 3D reconstruction, we host the MetaFood Workshop and its challenge for Physically Informed 3D Food Reconstruction. This challenge focuses on reconstructing volume-accurate 3D models of food items from 2D images, using a visible checkerboard as a size reference. Participants were tasked with reconstructing 3D models for 20 selected food items of varying difficulty levels: easy, medium, and hard. The easy level provides 200 images, the medium level provides 30 images, and the hard level provides only 1 image for reconstruction. In total, 16 teams submitted results in the final testing phase. The solutions developed in this challenge achieved promising results in 3D food reconstruction, with significant potential for improving portion estimation for dietary assessment and nutritional monitoring. More details about this workshop challenge and access to the dataset can be found at <a class="link-external link-https" href="https://sites.google.com/view/cvpr-metafood-2024" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problems does this paper attempt to solve?

This paper is mainly dedicated to solving the key problems in **3D food reconstruction**, especially in the fields of nutrition and diet monitoring. Specifically, the research aims to reconstruct high - precision 3D food models from 2D images through computer vision technology. The following are the main problems that this research attempts to solve:

1. **Lack of high - quality data**: Current food reconstruction methods are limited due to the lack of high - quality data sets. To overcome this challenge, this research introduced a data set containing 20 food items with different difficulty levels, and provided detailed image and depth information.

2. **Limited cooperation between industry and academia**: Insufficient cooperation between industry and academia has led to slow technological progress. This research promoted cross - domain cooperation and drove technological innovation by organizing the MetaFood Workshop and challenge.

3. **Accurate food portion estimation**: Traditional diet assessment methods (such as 24 - hour recall or food frequency questionnaire) rely on manual input and are prone to errors. In addition, 2D RGB images lack 3D information, making regression - based methods difficult to accurately estimate food portions. Through 3D reconstruction technology, this research provides more accurate estimates of food volume and density, thereby improving the accuracy of food portion estimation.

4. **Adapting to complex food shapes and lighting conditions**: Food shapes, textures, and lighting conditions vary, posing challenges to 3D reconstruction. This research encourages the development of innovative technologies to handle these complexities while meeting the needs in practical applications.

5. **Processing of single - view and multi - view inputs**: This challenge covers different input scenarios from single - view to multi - view, testing the robustness and universality of different methods in various practical application scenarios. For example, the simple level provides about 200 images, the medium - difficulty level provides about 30 images, and the difficult level provides only one image.

6. **Standardization and interpretability**: Although existing deep - learning methods are powerful in performance, they usually lack interpretability and perform poorly when dealing with samples outside the training data distribution. Through the introduction of 3D reconstruction technology, this research provides visual results and a standardized method for food portion estimation.

Overall, this research aims to improve the accuracy of food portion estimation through advanced 3D reconstruction technology, promote the development of nutrition assessment and diet monitoring, and ultimately improve the effectiveness of personal health management and public health activities.