UMDFood: Vision-language models boost food composition compilation

Peihua Ma,Yixin Wu,Ning Yu,Yang Zhang,Michael Backes,Qin Wang,Cheng-I Wei

2023-11-07

Abstract:Nutrition information is crucial in precision nutrition and the food industry. The current food composition compilation paradigm relies on laborious and experience-dependent methods. However, these methods struggle to keep up with the dynamic consumer market, resulting in delayed and incomplete nutrition data. In addition, earlier machine learning methods overlook the information in food ingredient statements or ignore the features of food images. To this end, we propose a novel vision-language model, UMDFood-VL, using front-of-package labeling and product images to accurately estimate food composition profiles. In order to empower model training, we established UMDFood-90k, the most comprehensive multimodal food database to date, containing 89,533 samples, each labeled with image and text-based ingredient descriptions and 11 nutrient annotations. UMDFood-VL achieves the macro-AUCROC up to 0.921 for fat content estimation, which is significantly higher than existing baseline methods and satisfies the practical requirements of food composition compilation. Meanwhile, up to 82.2% of selected products' estimated error between chemical analysis results and model estimation results are less than 10%. This performance sheds light on generalization towards other food and nutrition-related data compilation and catalyzation for the evolution of generative AI-based technology in other food applications that require personalization.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve

The paper aims to address several key issues in the compilation of food nutritional components:

1. **Limitations of Traditional Methods**: Current methods for compiling food components rely on labor-intensive and experience-dependent approaches, which struggle to keep up with the dynamic consumer market, resulting in delayed and incomplete nutritional data.
2. **Insufficient Information Utilization**: Early machine learning methods either ignored the information in food ingredient statements or overlooked the features of food images, failing to fully utilize multimodal data.
3. **Accuracy and Precision Requirements**: Large-scale food component compilation requires high accuracy and precision, but existing methods struggle to meet these requirements due to the high diversity and complexity of the task.

To overcome these issues, the authors propose a new visual-language model (UMDFood-VL), which combines front-label packaging and product images to accurately estimate food ingredient profiles. Additionally, they have established the UMDFood-90k database, the most comprehensive multimodal food database to date, containing 89,533 samples, each annotated with images and text-based ingredient descriptions as well as 11 types of nutritional annotations.

Through these innovations, UMDFood-VL achieved a macro AUCROC of 0.921 in fat content estimation, significantly outperforming existing baseline methods and meeting the practical needs of food component compilation. Moreover, the error between the chemical analysis results and the model estimates for 82.2% of the selected products was less than 10%, indicating that the model has high reliability and accuracy in practical applications.