Abstract:Conventional approaches to dietary assessment are primarily grounded in self-reporting methods or structured interviews conducted under the supervision of dietitians. These methods, however, are often subjective, potentially inaccurate, and time-intensive. Although artificial intelligence (AI)-based solutions have been devised to automate the dietary assessment process, these prior AI methodologies encounter challenges in their ability to generalize across a diverse range of food types, dietary behaviors, and cultural contexts. This results in AI applications in the dietary field that possess a narrow specialization and limited accuracy. Recently, the emergence of multimodal foundation models such as GPT-4V powering the latest ChatGPT has exhibited transformative potential across a wide range of tasks (e.g., Scene understanding and image captioning) in numerous research domains. These models have demonstrated remarkable generalist intelligence and accuracy, capable of processing various data modalities. In this study, we explore the application of multimodal ChatGPT within the realm of dietary assessment. Our findings reveal that GPT-4V excels in food detection under challenging conditions with accuracy up to 87.5% without any fine-tuning or adaptation using food-specific datasets. By guiding the model with specific language prompts (e.g., African cuisine), it shifts from recognizing common staples like rice and bread to accurately identifying regional dishes like banku and ugali. Another GPT-4V's standout feature is its contextual awareness. GPT-4V can leverage surrounding objects as scale references to deduce the portion sizes of food items, further enhancing its accuracy in translating food weight into nutritional content. This alignment with the USDA National Nutrient Database underscores GPT-4V's potential to advance nutritional science and dietary assessment techniques.

A Visually-Aware Food Analysis System for Diet Management

A Vision-Based Dietary Survey and Assessment System for College Students in China

A review on vision-based analysis for automatic dietary assessment

goFOODTM: An Artificial Intelligence System for Dietary Assessment

DeepFood: Food Image Analysis and Dietary Assessment Via Deep Model.

Visual Food Recognition for Dietary Logging and Health Monitoring

An Intelligent Passive Food Intake Assessment System with Egocentric Cameras

NutrifyAI: An AI-Powered System for Real-Time Food Detection, Nutritional Analysis, and Personalized Meal Recommendations

Visually-aware Collaborative Food Recommendation.

FoodAI: Food Image Recognition via Deep Learning for Smart Food Logging

An End-to-End Food Image Analysis System

A Model for Automated Food Logging Through Food Recognition and Attribute Estimation Using Deep Learning

A SAM based Tool for Semi-Automatic Food Annotation

NUTRIVISION: A System for Automatic Diet Management in Smart Healthcare

Dietary Nutritional Information Autonomous Perception Method Based on Machine Vision in Smart Homes

An Intelligent Vision-Based Nutritional Assessment Method for Handheld Food Items

DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment

AI4Food-NutritionFW: A Novel Framework for the Automatic Synthesis and Analysis of Eating Behaviours

Food recognition and recipe analysis: integrating visual content, context and external knowledge

Smart Diet Diary: Real-Time Mobile Application for Food Recognition

Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis