Mixture of Rationale: Multi-Modal Reasoning Mixture for Visual Question Answering

Tao Li,Linjun Shou,Xuejun Liu

2024-06-03

Abstract:Zero-shot visual question answering (VQA) is a challenging task that requires reasoning across modalities. While some existing methods rely on a single rationale within the Chain of Thoughts (CoT) framework, they may fall short of capturing the complexity of the VQA problem. On the other hand, some other methods that use multiple rationales may still suffer from low diversity, poor modality alignment, and inefficient retrieval and fusion. In response to these challenges, we propose \emph{Mixture of Rationales (MoR)}, a novel multi-modal reasoning method that mixes multiple rationales for VQA. MoR uses a single frozen Vision-and-Language Pre-trained Models (VLPM) model to {dynamically generate, retrieve and fuse multi-modal thoughts}. We evaluate MoR on two challenging VQA datasets, i.e. NLVR2 and OKVQA, with two representative backbones OFA and VL-T5. MoR achieves a 12.43\% accuracy improvement on NLVR2, and a 2.45\% accuracy improvement on OKVQA-S( the science and technology category of OKVQA).

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The paper aims to address the multimodal reasoning problem in the Zero-shot Visual Question Answering (VQA) task. Existing methods have the following limitations when dealing with such tasks:

1. The single reason generated by the Chain of Thoughts (CoT) framework may be insufficient to capture the complexity of VQA questions.
2. Methods using multiple reasons may encounter issues such as lack of diversity, poor modality alignment, and inefficiency in retrieval and fusion.

To address these issues, the authors propose the Mixture of Rationales (MoR) method, a novel multimodal reasoning approach that improves zero-shot VQA performance by mixing multiple reasons. MoR leverages a frozen Vision-and-Language Pre-trained Model (VLPM) to dynamically generate, retrieve, and fuse multimodal reasoning, thereby achieving more comprehensive cross-modal reasoning. Experimental results show that MoR significantly enhances zero-shot VQA performance on two challenging datasets, NLVR2 and OKVQA-S.