Abstract:Knowledge-based Visual Question Answering (KVQA) requires both image and world knowledge to answer questions. Current methods first retrieve knowledge from the image and external knowledge base with the original complex question, then generate answers with Large Language Models (LLMs). However, since the original question contains complex elements that require knowledge from different sources, acquiring different kinds of knowledge in a coupled manner may confuse models and hinder them from retrieving precise knowledge. Furthermore, the ``forward-only'' answering process fails to explicitly capture the knowledge needs of LLMs, which can further hurt answering quality. To cope with the above limitations, we propose DKA: Disentangled Knowledge Acquisition from LLM feedback, a training-free framework that disentangles knowledge acquisition to avoid confusion and uses LLM's feedback to specify the required knowledge. Specifically, DKA requires LLMs to specify what knowledge they need to answer the question and decompose the original complex question into two simple sub-questions: Image-based sub-question and Knowledge-based sub-question. Then we use the two sub-questions to retrieve knowledge from the image and knowledge base, respectively. In this way, two knowledge acquisition models can focus on the content that corresponds to them and avoid disturbance of irrelevant elements in the original complex question, which can help to provide more precise knowledge and better align the knowledge needs of LLMs to yield correct answers. Experiments on benchmark datasets show that DKA significantly outperforms SOTA models. To facilitate future research, our data and code are available at \url{<a class="link-external link-https" href="https://github.com/Lackel/DKA" rel="external noopener nofollow">this https URL</a>}.

Offline Definition Extraction Using Machine Learning for Knowledge-Oriented Question Answering

Automatic Question-Answering Based on Wikipedia Data Extraction

Answering Definition Questions Using Web Knowledge Bases

Coarse-to-Careful: Seeking Semantic-related Knowledge for Open-domain Commonsense Question Answering

A Deep Learning Approach for Question Answering over Knowledge Base

Knowledge-Aided Open-Domain Question Answering

Learning To Rank Answers For Definitional Question Answering

Hybrid Question Answering over Knowledge Base and Free Text.

A Chinese Intelligent Question Answering System Based on Domain Ontology and Sentence Templates

Answering Definitional Question By Dependency-Based Knowledge

Open Domain Question Answering Via Semantic Enrichment

Multi-document Question Answering Powered by External Knowledge.

Knowledge-Oriented Answer Extraction in Chinese Question Answering System for E-Learning

Long-Tailed Question Answering in an Open World.

Target Detection and Knowledge Learning for Domain Restricted Question Answering

Deep Learning in Question Answering

PullNet: Open Domain Question Answering with Iterative Retrieval on Knowledge Bases and Text

Knowledge Acquisition Disentanglement for Knowledge-based Visual Question Answering with Large Language Models

Domain Adaptive Question Answering over Knowledge Base.

OpenQA: Hybrid QA System Relying on Structured Knowledge Base As Well As Non-structured Data

Knowledge-Enhanced Retrieval: A Scheme for Question Answering