ARChef: An iOS-Based Augmented Reality Cooking Assistant Powered by Multimodal Gemini LLM

Rithik Vir,Parsa Madinei

2024-12-01

Abstract:Cooking meals can be difficult, causing many to use cookbooks and online recipes, which results in missing ingredients, nutritional hazards, unsatisfactory meals. Using Augmented Reality (AR) can address this issue, however, current AR cooking applications have poor user interfaces and limited accessibility. This paper proposes a prototype of an iOS application that integrates AR and Computer Vision (CV) into the cooking process. We leverage Google's Gemini Large Language Model (LLM) to identify ingredients based on the camera's field of vision, and generate recipe choices with their nutritional information. Additionally, this application uses Apple's ARKit to create an AR user interface compatible with iOS devices. Users can personalize their meal suggestions by inputting their dietary preferences and rating each meal. The application's effectiveness is evaluated through user experience surveys. This application contributes to the field of accessible cooking assistance technologies, aiming to reduce food wastage and improve the meal planning experience.

Human-Computer Interaction,Artificial Intelligence

What problem does this paper attempt to address?

The problems that this paper attempts to solve are the deficiencies in traditional cooking methods and existing cooking aids, specifically including:

1. **Cooking Difficulty and Ingredient Management Problems**: Many people often encounter difficulties when using traditional cooking methods (such as recipe books and online recipes) due to lack of cooking experience, resulting in missed ingredients, unbalanced nutrition or unsatisfactory meals.

2. **Food Waste and Food Safety Problems**: Traditional cooking methods cannot effectively identify the user's existing ingredients, which is likely to cause ingredient waste. In addition, traditional recipes often lack detailed nutritional information (such as potential allergens), which may lead to health problems or allergic reactions.

3. **Limitations of Existing AR Cooking Applications**: Current augmented reality (AR) cooking applications have problems such as poor user interfaces and poor accessibility. For example, some applications rely on preset recipe databases and cannot generate personalized recipes based on the user's existing ingredients; other applications, although they attempt to combine deep - learning models (such as YOLOv5) to identify ingredients, have low accuracy and require expensive hardware devices (such as Magic Leap One AR headsets), making these applications both expensive and difficult to popularize.

To solve the above problems, this paper proposes an augmented reality cooking assistant based on the iOS platform - **ARChef**. By integrating augmented reality (AR), computer vision (CV) and large - language models (LLM), this application aims to provide an easy - to - use, personalized cooking aid to help users more conveniently find recipes suitable for their existing ingredients and provide detailed nutritional information, thereby reducing food waste and improving the cooking experience.

### Main Solutions
- **Combination of AR and CV**: Use Apple's ARKit to create an AR user interface compatible with iOS devices and combine computer vision technology to identify the ingredients in the user's hands.
- **Application of Gemini LLM**: Through Google's Gemini large - language model, generate appropriate recipes based on the identified ingredients and provide detailed nutritional information.
- **Personalization Settings**: Users can enter their dietary preferences (such as allergens, favorite cuisines, etc.) on the settings page to obtain more personalized recipe recommendations.
- **Multilingual Support**: In order to expand the audience of the application, support multiple languages (such as English, Spanish, French, Chinese, Japanese, Arabic, Persian and Hindi).

Through these improvements, ARChef not only improves the user's cooking experience but also reduces food waste and health risks caused by insufficient ingredient management and nutritional information.