VL-Meta: Vision-Language Models for Multimodal Meta-Learning

Han Ma,Baoyu Fan,Benjamin K. Ng,Chan-Tong Lam

DOI: https://doi.org/10.3390/math12020286

IF: 2.4

2024-01-16

Mathematics

Abstract:Multimodal learning is a promising area in artificial intelligence (AI) that can make the model understand different kinds of data. Existing works are trying to re-train a new model based on pre-trained models that requires much data, computation power, and time. However, it is difficult to achieve in low-resource or small-sample situations. Therefore, we propose VL-Meta, Vision Language Models for Multimodal Meta Learning. It (1) presents the vision-language mapper and multimodal fusion mapper, which are light model structures, to use the existing pre-trained models to make models understand images to language feature space and save training data, computation power, and time; (2) constructs the meta-task pool that can only use a small amount of data to construct enough training data and improve the generalization of the model to learn the data knowledge and task knowledge; (3) proposes the token-level training that can align inputs with the outputs during training to improve the model performance; and (4) adopts the multi-task fusion loss to learn the different abilities for the models. It achieves a good performance on the Visual Question Answering (VQA) task, which shows the feasibility and effectiveness of the model. This solution can help blind or visually impaired individuals obtain visual information.

mathematics

What problem does this paper attempt to address?

The paper attempts to address the problem of how to effectively train multimodal models to perform Visual Question Answering (VQA) tasks in resource-limited or small-sample scenarios. Specifically, existing multimodal learning methods typically require a large amount of data, computational power, and time to retrain pre-trained models, which is difficult to achieve in low-resource or small-sample situations. Therefore, the authors propose the VL-Meta model, aiming to solve these issues through the following innovations:

1. **Lightweight Visual-Language Mappers and Multimodal Fusion Mappers**: These structures can utilize existing pre-trained models to map image features to the language feature space, thereby reducing the need for training data, computational power, and time.

2. **Meta-Task Pool**: By constructing a meta-task pool, a small amount of data is used to generate sufficient training data, improving the model's generalization ability, enabling it to learn both data knowledge and task knowledge.

3. **Token-Level Training**: During the training process, inputs and outputs are aligned one by one to improve model performance.

4. **Multi-Task Fusion Loss**: A multi-task fusion loss function is employed to help the model learn different capabilities from different perspectives.

Through these methods, the VL-Meta model demonstrates good performance in visual question answering tasks, showcasing its feasibility and effectiveness. Additionally, this model can also assist visually impaired individuals in obtaining visual information.