Abstract:To incorporate the cooking logic into ingredient recognition from food images is beneficial for food cognition. Compared with food categorization, ingredient recognition gives a better understanding on food cognition, by providing crucial information on food compositions. However, there exist situations in which different food are made of different ingredients, thus it is necessary to incorporate cooking logic into ingredient recognition to achieve a better food cognition. Based on this point, our paper proposes a sequential learning method to guide a neural network based (NN-based) model on producing ingredients following the corresponding cooking logic in recipes. Firstly, in order to make a maximum utilization of visual features from images, a double-flow feature fusion module (DFFF) is proposed to obtain features from two image-based, visual tasks (food name proposal and multi-label ingredient proposal). After that, fused features from DFFF, together with original image features, are feed into a bidirectional long short time memory (Bi-LSTM) based ingredient generator to produce sequential ingredients. To guide the sequential ingredient generation process, reinforcement learning is employed by designing a hybrid loss related to both the common and personality traits in ingredients for optimizing the model ability of associating images and sequential ingredients. In addition, sequential ingredients are utilized in a backward flow by reconstructing food images, so that sequential ingredient generation can be further optimized in a complementary manner. In experiments, the results demonstrate the superiority of our method on driving the model to allocate more attention to the correlation between images and sequential ingredients, and produced ingredients are comprehensive and logical.

Deep-based Ingredient Recognition for Cooking Recipe Retrieval

A Study of Multi-Task and Region-Wise Deep Learning for Food Ingredient Recognition.

Deep Understanding Of Cooking Procedure For Cross-Modal Recipe Retrieval

Cross-modal Recipe Retrieval with Rich Food Attributes

Zero-Shot Ingredient Recognition by Multi-Relational Graph Convolutional Network

Cross-Modal Recipe Retrieval: How to Cook This Dish?

Deep Cooking: Predicting Relative Food Ingredient Amounts from Images

Deep Image-to-Recipe Translation

Cross-modal Recipe Retrieval with Stacked Attention Model

Video-based Recipe Retrieval

Sequential learning for ingredient recognition from images

Cross-domain Cross-modal Food Transfer.

Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism

DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment

Recognizing Multiple Ingredients in Food Images Using a Single-Ingredient Classification Model

MCEN: Bridging Cross-Modal Gap Between Cooking Recipes and Dish Images with Latent Variable Model

MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model.

Deep Learning in Food Category Recognition

Food recognition and recipe analysis: integrating visual content, context and external knowledge

Cross-domain Food Image-to-Recipe Retrieval by Weighted Adversarial Learning

Learning From Web Recipe-Image Pairs for Food Recognition: Problem, Baselines and Performance