Abstract:Recently, visual food analysis has received more and more attention in the computer vision community due to its wide application scenarios, e.g., diet nutrition management, smart restaurant, and personalized diet recommendation. Considering that food images are unstructured images with complex and unfixed visual patterns, mining food-related semantic-aware regions is crucial. Furthermore, the ingredients contained in food images are semantically related to each other due to the cooking habits and have significant semantic relationships with food categories under the hierarchical food classification ontology. Therefore, modeling the long-range semantic relationships between ingredients and the categories-ingredients semantic interactions is beneficial for ingredient recognition and food analysis. Taking these factors into consideration, we propose a multi-task learning framework for food category and ingredient recognition. This framework mainly consists of a food-orient Transformer named Convolution-Enhanced Bi-Branch Adaptive Transformer (CBiAFormer) and a multi-task category-ingredient recognition network called Structural Learning and Cross-Task Interaction (SLCI). In order to capture the complex and unfixed fine-grained patterns of food images, we propose a query-aware data-adaptive attention mechanism called Bi-Branch Adaptive Attention (BiA-Attention) in CBiAFormer, which consists of a local fine-grained branch and a global coarse-grained branch to mine local and global semantic-aware regions for different input images through an adaptive candidate key/value sets assignment for each query. Additionally, a convolutional patch embedding module is proposed to extract the fine-grained features which are neglected by Transformers. To fully utilize the ingredient information, we propose SLCI, which consists of cross-layer attention to model the semantic relationships between ingredients and two cross-task interaction modules to mine the semantic interactions between categories and ingredients. Extensive experiments show that our method achieves competitive performance on three mainstream food datasets (ETH Food-101, Vireo Food-172, and ISIA Food-200). Visualization analyses of CBiAFormer and SLCI on two tasks prove the effectiveness of our method. Codes will be released upon publication. Code and models are available at https://github.com/Liuyuxinict/CBiAFormer.

A Noise-robust Locality Transformer for Fine-grained Food Image Retrieval.

LoFTR: Detector-Free Local Feature Matching with Transformers

LCTR: On Awakening the Local Continuity of Transformer for Weakly Supervised Object Localization

LGFCTR: Local and Global Feature Convolutional Transformer for Image Matching

Food recognition via an efficient neural network with transformer grouping

MenuAI: Restaurant Food Recommendation System via a Transformer-based Deep Learning Model

Re-Attention Transformer for Weakly Supervised Object Localization

Convolution-Enhanced Bi-Branch Adaptive Transformer With Cross-Task Interaction for Food Category and Ingredient Recognition

Transformer Decoders with MultiModal Regularization for Cross-Modal Food Retrieval

Learning Using Privileged Information for Food Recognition

An improved transformer model based on global and local features for few-shot learning with noise samples

Locating Target Regions for Image Retrieval in an Unsupervised Manner

LGIT: local–global interaction transformer for low-light image denoising

Reperceive Global Vision of Transformer for Remote Sensing Images Weakly Supervised Object Localization

Exploring latent weight factors and global information for food-oriented cross-modal retrieval

Low Light RAW Image Enhancement Using Paired Fast Fourier Convolution and Transformer

A DeNoising FPN With Transformer R-CNN for Tiny Object Detection

TransLO: A Window-Based Masked Point Transformer Framework for Large-Scale LiDAR Odometry

Food Recognition with Visual Transformers

A Lightweight Hybrid Model with Location-Preserving ViT for Efficient Food Recognition