MenuAI: Restaurant Food Recommendation System via a Transformer-based Deep Learning Model

Xinwei Ju,Frank Po Wen Lo,Jianing Qiu,Peilun Shi,Jiachuan Peng,Benny Lo

DOI: https://doi.org/10.48550/arXiv.2210.08266

2022-10-15

Abstract:Food recommendation system has proven as an effective technology to provide guidance on dietary choices, and this is especially important for patients suffering from chronic diseases. Unlike other multimedia recommendations, such as books and movies, food recommendation task is highly relied on the context at the moment, since users' food preference can be highly dynamic over time. For example, individuals tend to eat more calories earlier in the day and eat a little less at dinner. However, there are still limited research works trying to incorporate both current context and nutritional knowledge for food recommendation. Thus, a novel restaurant food recommendation system is proposed in this paper to recommend food dishes to users according to their special nutritional needs. Our proposed system utilises Optical Character Recognition (OCR) technology and a transformer-based deep learning model, Learning to Rank (LTR) model, to conduct food recommendation. Given a single RGB image of the menu, the system is then able to rank the food dishes in terms of the input search key (e.g., calorie, protein level). Due to the property of the transformer, our system can also rank unseen food dishes. Comprehensive experiments are conducted to validate our methods on a self-constructed menu dataset, known as MenuRank dataset. The promising results, with accuracy ranging from 77.2% to 99.5%, have demonstrated the great potential of LTR model in addressing food recommendation problems.

Information Retrieval,Machine Learning

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address several key issues in restaurant food recommendation systems:

1. **Dynamic User Preferences**: Unlike multimedia recommendations such as books and movies, food recommendations are highly dependent on the current context, as users' dietary preferences change over time. For example, people typically consume more calories earlier in the day and eat less during dinner. Existing food recommendation systems often assume that user preferences are static, which does not align with real-world scenarios.

2. **Integration of Nutritional Knowledge**: Existing food recommendation systems rarely combine current context with nutritional knowledge for recommendations. This is particularly important for patients with chronic diseases (such as diabetes, cardiovascular diseases, etc.), as they need to choose foods based on specific nutritional requirements.

3. **Recommendation of Unseen Foods**: Due to the lack of associated data, existing food recommendation systems struggle to handle unseen foods (i.e., foods not present in the nutritional database). This limits the system's applicability and flexibility.

To address these issues, the authors propose a Transformer-based deep learning model—Learning to Rank (LTR) model, which combines Optical Character Recognition (OCR) technology and self-attention mechanisms to achieve food recommendations. This system can rank foods on a menu based on the user's specific nutritional needs (such as calorie and protein levels) and can handle unseen foods. The effectiveness of this approach is validated through experiments conducted on a self-built MenuRank dataset.