Abstract:Knowledge-based Visual Question Answering (KVQA) requires external knowledge beyond the visible content to answer questions about an image. This ability is challenging but indispensable to achieve general VQA. One limitation of existing KVQA solutions is that they jointly embed all kinds of information without fine-grained selection, which introduces unexpected noises for reasoning the correct answer. How to capture the question-oriented and information-complementary evidence remains a key challenge to solve the problem. Inspired by the human cognition theory, in this paper, we depict an image by multiple knowledge graphs from the visual, semantic and factual views. Thereinto, the visual graph and semantic graph are regarded as image-conditioned instantiation of the factual graph. On top of these new representations, we re-formulate Knowledge-based Visual Question Answering as a recurrent reasoning process for obtaining complementary evidence from multimodal information. To this end, we decompose the model into a series of memory-based reasoning steps, each performed by a G raph-based R ead, U pdate, and C ontrol ( GRUC ) module that conducts parallel reasoning over both visual and semantic information. By stacking the modules multiple times, our model performs transitive reasoning and obtains question-oriented concept representations under the constrain of different modalities. Finally, we perform graph neural networks to infer the global-optimal answer by jointly considering all the concepts. We achieve a new state-of-the-art performance on three popular benchmark datasets, including FVQA, Visual7W-KB and OK-VQA, and demonstrate the effectiveness and interpretability of our model with extensive experiments.

The Visual Query Language for an Object-Oriented Knowledge-Based System

KGVQL: A knowledge graph visual query language with bidirectional transformations

VisKoP: Visual Knowledge oriented Programming for Interactive Knowledge Base Question Answering

GVQA: Learning to Answer Questions about Graphs with Visualizations Via Knowledge Base.

VOGUE: Towards A Visual Interaction-aware Graph Query Processing Framework.

Knowledge-based Visual Question Answering:A Survey

Quda: Natural Language Queries for Visual Data Analytics

OKQA:Ontology based software development Knowledge Question Answering tool

An Interactive System for Knowledge Graph Search

A Formal Foundation for Implementing Object Query Language

Visual ontology modeling tool and ontology based querying of semantic web resources

KVQA: Knowledge-Aware Visual Question Answering

Knowledge-Based Visual Question Generation

SVQL: A SQL Extended Query Language for Video Databases

A visual graph-based query interface for video databases

Slq: A User-Friendly Graph Querying System

Can Open Domain Question Answering Systems Answer Visual Knowledge Questions?

A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge

LinkQ: An LLM-Assisted Visual Interface for Knowledge Graph Question-Answering

Implementation of Object-Oriented Graphic Database Query Technology

Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering