Abstract:Image-to-recipe retrieval is a challenging vision-to-language task of significant practical value. The main challenge of the task lies in the ultra-high redundancy in the long recipe and the large variation reflected in both food item combination and food item appearance. A de-facto idea to address this task is to learn a shared feature embedding space in which a food image is aligned better to its paired recipe than other recipes. However, such supervised global matching is prone to supervision collapse, i.e., only partial information that is necessary for distinguishing training pairs can be identified, while other information that is potentially useful in generalization could be lost. To mitigate such a problem, we propose a mask-augmentation-based local matching network (MALM), where an image-text matching module and a masked self-distillation module benefit each other mutually to learn generalizable cross-modality representations. On one hand, we perform local matching between the tokenized representations of image and text to locate fine-grained cross-modality correspondence explicitly. We involve representations of masked image patches in this process to alleviate overfitting resulting from local matching especially when some food items are underrepresented. On the other hand, predicting the hidden representations of the masked patches through self-distillation helps to learn general-purpose image representations that are expected to generalize better. And the multi-task nature of the model enables the representations of masked patches to be text-aware and thus facilitates the lost information reconstruction. Experimental results on Recipe1M dataset show our method can clearly outperform state-of-the-art (SOTA) methods. Our code will be available at <a class="link-external link-https" href="https://github.com/MyFoodChoice/MALM_Mask_Augmentation_based_Local_Matching-_for-_Food_Recipe_Retrieval" rel="external noopener nofollow">this https URL</a>

Learning From Web Recipe-Image Pairs for Food Recognition: Problem, Baselines and Performance

Cross-domain Cross-modal Food Transfer.

Cross-Modal Recipe Retrieval: How to Cook This Dish?

Deep Understanding Of Cooking Procedure For Cross-Modal Recipe Retrieval

Cross-modal Recipe Retrieval with Rich Food Attributes

MCEN: Bridging Cross-Modal Gap Between Cooking Recipes and Dish Images with Latent Variable Model

Cross-domain Food Image-to-Recipe Retrieval by Weighted Adversarial Learning

MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model.

Cross-modal Recipe Retrieval with Stacked Attention Model

CREAMY: Cross-Modal Recipe Retrieval By Avoiding Matching Imperfectly

Cross-Modal Food Retrieval: Learning a Joint Embedding of Food Images and Recipes with Semantic Consistency and Attention Mechanism

Deep-based Ingredient Recognition for Cooking Recipe Retrieval

Revamping Image-Recipe Cross-Modal Retrieval with Dual Cross Attention Encoders

Deep Image-to-Recipe Translation

Exploring latent weight factors and global information for food-oriented cross-modal retrieval

Learning Using Privileged Information for Food Recognition

Cross-Modal Recipe Retrieval with Self-Attention Mechanism

MALM: Mask Augmentation based Local Matching for Food-Recipe Retrieval

Self-Attention and Ingredient-Attention Based Model for Recipe Retrieval from Image Queries

Cross-lingual Adaptation for Recipe Retrieval with Mixup

Video-based Recipe Retrieval