MC-CoT: A Modular Collaborative CoT Framework for Zero-shot Medical-VQA with LLM and MLLM Integration

Lai Wei,Wenkai Wang,Xiaoyu Shen,Yu Xie,Zhihao Fan,Xiaojin Zhang,Zhongyu Wei,Wei Chen

2024-10-06

Abstract:In recent advancements, multimodal large language models (MLLMs) have been fine-tuned on specific medical image datasets to address medical visual question answering (Med-VQA) tasks. However, this common approach of task-specific fine-tuning is costly and necessitates separate models for each downstream task, limiting the exploration of zero-shot capabilities. In this paper, we introduce MC-CoT, a modular cross-modal collaboration Chain-of-Thought (CoT) framework designed to enhance the zero-shot performance of MLLMs in Med-VQA by leveraging large language models (LLMs). MC-CoT improves reasoning and information extraction by integrating medical knowledge and task-specific guidance, where LLM provides various complex medical reasoning chains and MLLM provides various observations of medical images based on instructions of the LLM. Our experiments on datasets such as SLAKE, VQA-RAD, and PATH-VQA show that MC-CoT surpasses standalone MLLMs and various multimodality CoT frameworks in recall rate and accuracy. These findings highlight the importance of incorporating background information and detailed guidance in addressing complex zero-shot Med-VQA tasks.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to improve the performance of multimodal large language models (MLLMs) in the zero - sample learning scenario in the medical visual question answering (Med - VQA) task. Specifically, current methods usually need to fine - tune the model for specific tasks, which not only consumes a large amount of resources but also limits the generalization ability of the model among different tasks. In addition, these methods often lack sufficient medical knowledge and reasoning ability when dealing with complex medical images.

To solve these problems, the paper proposes a modular collaborative chain - of - thought framework named MC - CoT. MC - CoT enhances the performance of MLLMs in the Med - VQA task by integrating the knowledge and reasoning ability of large language models (LLMs). Specifically, MC - CoT contains three pre - designed image feature extraction modules: a pathology module, a radiology module, and an anatomy module. Each module focuses on different aspects of the image and generates corresponding responses according to the task requirements. The entire framework activates the appropriate module and generates the final answer with the background knowledge and guidance provided by the LLM.

The main contributions of the paper include:
1. Proposing the MC - CoT framework, which enhances the zero - sample learning ability of MLLMs in the Med - VQA task.
2. Conducting comprehensive experiments on multiple Med - VQA datasets, verifying the effectiveness and wide applicability of MC - CoT, and providing an impact analysis of key processes and modules, which provides directions for future research optimization.