Abstract:The advancement of artificial intelligence (AI) in food and nutrition research is hindered by a critical bottleneck: the lack of annotated food data. Despite the rise of highly efficient AI models designed for tasks such as food segmentation and classification, their practical application might necessitate proficiency in AI and machine learning principles, which can act as a challenge for non-AI experts in the field of nutritional sciences. Alternatively, it highlights the need to translate AI models into user-friendly tools that are accessible to all. To address this, we present a demo of a semi-automatic food image annotation tool leveraging the Segment Anything Model (SAM). The tool enables prompt-based food segmentation via user interactions, promoting user engagement and allowing them to further categorise food items within meal images and specify weight/volume if necessary. Additionally, we release a fine-tuned version of SAM's mask decoder, dubbed MealSAM, with the ViT-B backbone tailored specifically for food image segmentation. Our objective is not only to contribute to the field by encouraging participation, collaboration, and the gathering of more annotated food data but also to make AI technology available for a broader audience by translating AI into practical tools.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the bottleneck problem of manually annotating food data in the field of food and nutrition research. Specifically, although artificial intelligence (AI) has made significant progress in food segmentation and classification tasks, the practical application of these models often requires users to have professional knowledge of AI and machine learning, which poses a challenge to non - AI experts (such as nutritionists). In addition, although the existing public food image datasets provide valuable resources for food segmentation, classification, and volume estimation, they usually lack detailed annotation information and cannot meet the needs of high - quality training and evaluation. To this end, the authors propose a semi - automatic food image annotation tool based on the Segment Anything Model (SAM), aiming to simplify food annotation tasks, including segmentation, classification, and optional weight/volume annotation. By transforming AI technology into an easy - to - use tool, this tool can not only promote broader user participation, but also improve the quality and efficiency of food data annotation, thereby promoting the development of nutritional science research. ### Main contributions 1. **Open - source tool**: Supports the pre - trained SAM model and its fine - tuned version MealSAM specific to food image segmentation. 2. **User - friendliness**: Performs food segmentation through interactive clicking, making it easy for non - AI experts to use. 3. **Efficient annotation**: Significantly reduces annotation time and improves annotation accuracy compared to fully manual annotation. 4. **Community collaboration**: Encourages more people to participate in food data annotation and AI model improvement through open - source tools. ### Key points of the solution - **Prompt - based semi - automatic segmentation**: Utilizes the SAM model to receive user clicks as prompts to generate segmentation masks. - **MealSAM**: Fine - tunes the mask decoder of SAM for food images to improve segmentation quality. - **Tool design and implementation**: Combines the Python programming language and multiple libraries (such as Tkinter, PIL, OpenCV, PyTorch) to build a complete workflow including segmentation, category selection, and weight/volume annotation functions. In summary, this paper aims to overcome the current application barriers of AI in food and nutrition research by developing a user - friendly semi - automatic food image annotation tool, promote the generation of more high - quality food data, and then support the further development of AI technology in this field.

A SAM based Tool for Semi-Automatic Food Annotation

FoodSAM: Any Food Segmentation

goFOODTM: An Artificial Intelligence System for Dietary Assessment

NutrifyAI: An AI-Powered System for Real-Time Food Detection, Nutritional Analysis, and Personalized Meal Recommendations

FoodMem: Near Real-time and Precise Food Video Segmentation

FoodAI: Food Image Recognition via Deep Learning for Smart Food Logging

AI4Food-NutritionFW: A Novel Framework for the Automatic Synthesis and Analysis of Eating Behaviours

Segment Anything Model for automated image data annotation: empirical studies using text prompts from Grounding DINO

Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications

SAM for Poultry Science

MyFood: A Food Segmentation and Classification System to Aid Nutritional Monitoring

The Nutritional Content of Meal Images in Free-Living Conditions-Automatic Assessment with goFOODTM

Medical Image Segmentation with SAM-generated Annotations

$\mathrm{SAM^{Med}}$: A medical image annotation framework based on large vision model

AI-SAM: Automatic and Interactive Segment Anything Model

A Survey on Segment Anything Model (SAM): Vision Foundation Model Meets Prompt Engineering

A Model for Automated Food Logging Through Food Recognition and Attribute Estimation Using Deep Learning

Towards Granularity-adjusted Pixel-level Semantic Annotation

AGSAM: Agent-Guided Segment Anything Model for Automatic Segmentation in Few-Shot Scenarios

Delving Deep into NutriScan: Automated Nutrition Table Extraction and Ingredient Recognition

From SAM to SAM 2: Exploring Improvements in Meta's Segment Anything Model