Abstract:Medical Visual Question Answering (Med-VQA) is a very important task in healthcare industry, which answers a natural language question with a medical image. Existing VQA techniques in information systems can be directly applied to solving the task. However, they often suffer from (i) the data insufficient problem, which makes it difficult to train the state of the arts (SOTAs) for the domain-specific task, and (ii) the reproducibility problem, that many existing models have not been thoroughly evaluated in a unified experimental setup. To address these issues, this paper develops a Benchmark Evaluation SysTem for Medical Visual Question Answering, denoted by BESTMVQA. Given self-collected clinical data, our system provides a useful tool for users to automatically build Med-VQA datasets, which helps overcoming the data insufficient problem. Users also can conveniently select a wide spectrum of SOTA models from our model library to perform a comprehensive empirical study. With simple configurations, our system automatically trains and evaluates the selected models over a benchmark dataset, and reports the comprehensive results for users to develop new techniques or perform medical practice. Limitations of existing work are overcome (i) by the data generation tool, which automatically constructs new datasets from unstructured clinical data, and (ii) by evaluating SOTAs on benchmark datasets in a unified experimental setup. The demonstration video of our system can be found at <a class="link-external link-https" href="https://youtu.be/QkEeFlu1x4A" rel="external noopener nofollow">this https URL</a>. Our code and data will be available soon.

AVA: an Automatic eValuation Approach to Question Answering Systems

SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References

Unsupervised Evaluation for Question Answering with Transformers

QAVidCap: Enhancing Video Captioning Through Question Answering Techniques

'Just because you are right, doesn't mean I am wrong': Overcoming a Bottleneck in the Development and Evaluation of Open-Ended Visual Question Answering (VQA) Tasks

Evaluation of Question Answering Systems: Complexity of judging a natural language

Improving Automatic VQA Evaluation Using Large Language Models

CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert Judgments For Open-Domain Question Answering

AutoEval-Video: An Automatic Benchmark for Assessing Large Vision Language Models in Open-Ended Video Question Answering

EfficientEQA: An Efficient Approach for Open Vocabulary Embodied Question Answering

QUADRo: Dataset and Models for QUestion-Answer Database Retrieval

AI-VQA

Towards Flexible Evaluation for Generative Visual Question Answering

BESTMVQA: A Benchmark Evaluation System for Medical Visual Question Answering

ASQ: Automatically Generating Question-Answer Pairs using AMRs

Evaluating Open-QA Evaluation

Learning Answer Generation using Supervision from Automatic Question Answering Evaluators

XAIQA: Explainer-Based Data Augmentation for Extractive Question Answering

AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs

Towards Automatic Generation of Questions from Long Answers

ISAAQ -- Mastering Textbook Questions with Pre-trained Transformers and Bottom-Up and Top-Down Attention