A SAM based Tool for Semi-Automatic Food Annotation

Lubnaa Abdur Rahman,Ioannis Papathanail,Lorenzo Brigato,Stavroula Mougiakakou
2024-10-11
Abstract:The advancement of artificial intelligence (AI) in food and nutrition research is hindered by a critical bottleneck: the lack of annotated food data. Despite the rise of highly efficient AI models designed for tasks such as food segmentation and classification, their practical application might necessitate proficiency in AI and machine learning principles, which can act as a challenge for non-AI experts in the field of nutritional sciences. Alternatively, it highlights the need to translate AI models into user-friendly tools that are accessible to all. To address this, we present a demo of a semi-automatic food image annotation tool leveraging the Segment Anything Model (SAM). The tool enables prompt-based food segmentation via user interactions, promoting user engagement and allowing them to further categorise food items within meal images and specify weight/volume if necessary. Additionally, we release a fine-tuned version of SAM's mask decoder, dubbed MealSAM, with the ViT-B backbone tailored specifically for food image segmentation. Our objective is not only to contribute to the field by encouraging participation, collaboration, and the gathering of more annotated food data but also to make AI technology available for a broader audience by translating AI into practical tools.
Computer Vision and Pattern Recognition,Artificial Intelligence
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the bottleneck problem of manually annotating food data in the field of food and nutrition research. Specifically, although artificial intelligence (AI) has made significant progress in food segmentation and classification tasks, the practical application of these models often requires users to have professional knowledge of AI and machine learning, which poses a challenge to non - AI experts (such as nutritionists). In addition, although the existing public food image datasets provide valuable resources for food segmentation, classification, and volume estimation, they usually lack detailed annotation information and cannot meet the needs of high - quality training and evaluation. To this end, the authors propose a semi - automatic food image annotation tool based on the Segment Anything Model (SAM), aiming to simplify food annotation tasks, including segmentation, classification, and optional weight/volume annotation. By transforming AI technology into an easy - to - use tool, this tool can not only promote broader user participation, but also improve the quality and efficiency of food data annotation, thereby promoting the development of nutritional science research. ### Main contributions 1. **Open - source tool**: Supports the pre - trained SAM model and its fine - tuned version MealSAM specific to food image segmentation. 2. **User - friendliness**: Performs food segmentation through interactive clicking, making it easy for non - AI experts to use. 3. **Efficient annotation**: Significantly reduces annotation time and improves annotation accuracy compared to fully manual annotation. 4. **Community collaboration**: Encourages more people to participate in food data annotation and AI model improvement through open - source tools. ### Key points of the solution - **Prompt - based semi - automatic segmentation**: Utilizes the SAM model to receive user clicks as prompts to generate segmentation masks. - **MealSAM**: Fine - tunes the mask decoder of SAM for food images to improve segmentation quality. - **Tool design and implementation**: Combines the Python programming language and multiple libraries (such as Tkinter, PIL, OpenCV, PyTorch) to build a complete workflow including segmentation, category selection, and weight/volume annotation functions. In summary, this paper aims to overcome the current application barriers of AI in food and nutrition research by developing a user - friendly semi - automatic food image annotation tool, promote the generation of more high - quality food data, and then support the further development of AI technology in this field.