Abstract:Knowledge-based Visual Question Answering (KVQA) requires external knowledge beyond the visible content to answer questions about an image. This ability is challenging but indispensable to achieve general VQA. One limitation of existing KVQA solutions is that they jointly embed all kinds of information without fine-grained selection, which introduces unexpected noises for reasoning the correct answer. How to capture the question-oriented and information-complementary evidence remains a key challenge to solve the problem. Inspired by the human cognition theory, in this paper, we depict an image by multiple knowledge graphs from the visual, semantic and factual views. Thereinto, the visual graph and semantic graph are regarded as image-conditioned instantiation of the factual graph. On top of these new representations, we re-formulate Knowledge-based Visual Question Answering as a recurrent reasoning process for obtaining complementary evidence from multimodal information. To this end, we decompose the model into a series of memory-based reasoning steps, each performed by a G raph-based R ead, U pdate, and C ontrol ( GRUC ) module that conducts parallel reasoning over both visual and semantic information. By stacking the modules multiple times, our model performs transitive reasoning and obtains question-oriented concept representations under the constrain of different modalities. Finally, we perform graph neural networks to infer the global-optimal answer by jointly considering all the concepts. We achieve a new state-of-the-art performance on three popular benchmark datasets, including FVQA, Visual7W-KB and OK-VQA, and demonstrate the effectiveness and interpretability of our model with extensive experiments.

On Visual Knowledge

Visual Knowledge in the Big Model Era: Retrospect and Prospect

SKANet - Structured Knowledge-Aware Network for Visual Dialog.

Computational knowledge vision: paradigmatic knowledge based prescriptive learning and reasoning for perception and vision

Visual Knowledge Discovery with Artificial Intelligence: Challenges and Future Directions

Knowledge Visualization: An Effective Way of Improving Learning

You should know more: Learning external knowledge for visual dialog

Knowledge-Embedded Mutual Guidance for Visual Reasoning

Knowledge Representation by Naive Mental Image

Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering

Explicit Knowledge Incorporation for Visual Reasoning

Explicit Knowledge-based Reasoning for Visual Question Answering

Multi-Level Knowledge Injecting for Visual Commonsense Reasoning

Visual knowledge: an attempt to explore machine creativity

Learning Visual Knowledge Memory Networks for Visual Question Answering

Perceptual Visual Reasoning with Knowledge Propagation

A Theoretical Approach for Structuring and Analysing Knowledge Provenance for Visual Analytics

Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks

Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting

KM 4 : Visual reasoning via Knowledge Embedding Memory Model with Mutual Modulation

The Core of Smart Cities: Knowledge Representation and Descriptive Framework Construction in Knowledge-Based Visual Question Answering