Abstract:To incorporate the cooking logic into ingredient recognition from food images is beneficial for food cognition. Compared with food categorization, ingredient recognition gives a better understanding on food cognition, by providing crucial information on food compositions. However, there exist situations in which different food are made of different ingredients, thus it is necessary to incorporate cooking logic into ingredient recognition to achieve a better food cognition. Based on this point, our paper proposes a sequential learning method to guide a neural network based (NN-based) model on producing ingredients following the corresponding cooking logic in recipes. Firstly, in order to make a maximum utilization of visual features from images, a double-flow feature fusion module (DFFF) is proposed to obtain features from two image-based, visual tasks (food name proposal and multi-label ingredient proposal). After that, fused features from DFFF, together with original image features, are feed into a bidirectional long short time memory (Bi-LSTM) based ingredient generator to produce sequential ingredients. To guide the sequential ingredient generation process, reinforcement learning is employed by designing a hybrid loss related to both the common and personality traits in ingredients for optimizing the model ability of associating images and sequential ingredients. In addition, sequential ingredients are utilized in a backward flow by reconstructing food images, so that sequential ingredient generation can be further optimized in a complementary manner. In experiments, the results demonstrate the superiority of our method on driving the model to allocate more attention to the correlation between images and sequential ingredients, and produced ingredients are comprehensive and logical.

Mixed Dish Recognition Through Multi-Label Learning.

Mixed Dish Recognition with Contextual Relation and Domain Alignment

A Chinese Dish Detector with Modified YOLO v3

A Study of Multi-Task and Region-Wise Deep Learning for Food Ingredient Recognition.

Chinese Dish Segmentation Based on Local Variation Driven Superpixel Grouping and Region Analysis

Food Ingredients Recognition through Multi-label Learning

Dynamic Mixup for Multi-Label Long-Tailed Food Ingredient Recognition

Multi-Task Learning for Food Identification and Analysis with Deep Convolutional Neural Networks

Deep-Learning-Assisted Multi-Dish Food Recognition Application for Dietary Intake Reporting

Where and What to Eat: Simultaneous Restaurant and Dish Recognition from Food Image

From Market To Dish: Multi-Ingredient Image Recognition For Personalized Recipe Recommendation

Recognizing Multiple Ingredients in Food Images Using a Single-Ingredient Classification Model

A fine-grained recognition technique for identifying Chinese food images

Robust Domain Generalization for Multi-modal Object Recognition

Cross-domain Cross-modal Food Transfer.

Zero-Shot Ingredient Recognition by Multi-Relational Graph Convolutional Network

Mining Discriminative Food Regions for Accurate Food Recognition

Multi-Task Image-Based Dietary Assessment for Food Recognition and Portion Size Estimation

Recognize after early fusion: the Chinese food recognition based on the alignment of image and ingredients

Robust Deep Neural Network for Learning in Noisy Multi-Label Food Images

Sequential learning for ingredient recognition from images