Abstract:Recently, visual food analysis has received more and more attention in the computer vision community due to its wide application scenarios, e.g., diet nutrition management, smart restaurant, and personalized diet recommendation. Considering that food images are unstructured images with complex and unfixed visual patterns, mining food-related semantic-aware regions is crucial. Furthermore, the ingredients contained in food images are semantically related to each other due to the cooking habits and have significant semantic relationships with food categories under the hierarchical food classification ontology. Therefore, modeling the long-range semantic relationships between ingredients and the categories-ingredients semantic interactions is beneficial for ingredient recognition and food analysis. Taking these factors into consideration, we propose a multi-task learning framework for food category and ingredient recognition. This framework mainly consists of a food-orient Transformer named Convolution-Enhanced Bi-Branch Adaptive Transformer (CBiAFormer) and a multi-task category-ingredient recognition network called Structural Learning and Cross-Task Interaction (SLCI). In order to capture the complex and unfixed fine-grained patterns of food images, we propose a query-aware data-adaptive attention mechanism called Bi-Branch Adaptive Attention (BiA-Attention) in CBiAFormer, which consists of a local fine-grained branch and a global coarse-grained branch to mine local and global semantic-aware regions for different input images through an adaptive candidate key/value sets assignment for each query. Additionally, a convolutional patch embedding module is proposed to extract the fine-grained features which are neglected by Transformers. To fully utilize the ingredient information, we propose SLCI, which consists of cross-layer attention to model the semantic relationships between ingredients and two cross-task interaction modules to mine the semantic interactions between categories and ingredients. Extensive experiments show that our method achieves competitive performance on three mainstream food datasets (ETH Food-101, Vireo Food-172, and ISIA Food-200). Visualization analyses of CBiAFormer and SLCI on two tasks prove the effectiveness of our method. Codes will be released upon publication. Code and models are available at https://github.com/Liuyuxinict/CBiAFormer.

DeepFood: Automatic Multi-Class Classification of Food Ingredients Using Deep Learning

Multi-Task Learning for Food Identification and Analysis with Deep Convolutional Neural Networks

Fine-grained food image classification and recipe extraction using a customized deep neural network and NLP

DeepFood: Food Image Analysis and Dietary Assessment Via Deep Model.

Application of Deep Learning in Food: A Review

Deep learning approaches in food recognition

Food Ingredients Recognition through Multi-label Learning

Deep Learning in Food Category Recognition

DeepFood: Deep Learning-Based Food Image Recognition for Computer-Aided Dietary Assessment

A Study of Multi-Task and Region-Wise Deep Learning for Food Ingredient Recognition.

Foodnet: Multi-Scale and Label Dependency Learning-Based Multi-Task Network for Food and Ingredient Recognition

Attention-Based Convolutional Neural Network for Ingredients Identification

Image Augmentation-Based Food Recognition with Convolutional Neural Networks

Recognizing Multiple Ingredients in Food Images Using a Single-Ingredient Classification Model

Food Recognition based on Deep Learning Algorithms

Deep Learning in Food Authenticity: Recent Advances and Future Trends

Deep learning accurately predicts food categories and nutrients based on ingredient statements

Food Image Recognition and Food Safety Detection Method Based on Deep Learning

PyTorch Deep Learning for Food Image Classification with Food Dataset

Food Recipe Recommendation Based on Ingredients Detection Using Deep Learning

Convolution-Enhanced Bi-Branch Adaptive Transformer With Cross-Task Interaction for Food Category and Ingredient Recognition