Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learning

Ivona Najdenkoska,Xiantong Zhen,Marcel Worring

DOI: https://doi.org/10.48550/arXiv.2302.14794

2023-03-01

Abstract:Multimodal few-shot learning is challenging due to the large domain gap between vision and language modalities. Existing methods are trying to communicate visual concepts as prompts to frozen language models, but rely on hand-engineered task induction to reduce the hypothesis space. To make the whole process learnable, we introduce a multimodal meta-learning approach. Specifically, our approach decomposes the training of the model into a set of related multimodal few-shot tasks. We define a meta-mapper network, acting as a meta-learner, to efficiently bridge frozen large-scale vision and language models and leverage their already learned capacity. By updating the learnable parameters only of the meta-mapper, it learns to accrue shared meta-knowledge among these tasks. Thus, it can rapidly adapt to newly presented samples with only a few gradient updates. Importantly, it induces the task in a completely data-driven manner, with no need for a hand-engineered task induction. We evaluate our approach on recently proposed multimodal few-shot benchmarks, measuring how rapidly the model can bind novel visual concepts to words and answer visual questions by observing only a limited set of labeled examples. The experimental results show that our meta-learning approach outperforms the baseline across multiple datasets and various training settings while being computationally more efficient.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the issue of a significant domain gap between visual and language modalities in multimodal few-shot learning, which makes it difficult for existing methods to effectively communicate visual concepts with text representations in language models. Specifically, existing methods typically rely on manually designed task guidance to reduce the hypothesis space, which becomes an obstacle when dealing with complex tasks. Therefore, this paper proposes a new multimodal meta-learning method that aims to automatically induce tasks from the data itself through a learnable process, without the need for manually designed task guidance. This method not only quickly adapts to new tasks but also demonstrates superior performance compared to baseline methods across multiple datasets and different training settings, with higher computational efficiency.

In short, the core issue of the paper is to solve the communication problem between visual and language modalities in multimodal few-shot learning by introducing a meta-learning mechanism, achieving rapid task adaptation without human intervention.