VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering

Yanan Wang,Michihiro Yasunaga,Hongyu Ren,Shinya Wada,Jure Leskovec

2023-09-15

Abstract:Visual question answering (VQA) requires systems to perform concept-level reasoning by unifying unstructured (e.g., the context in question and answer; "QA context") and structured (e.g., knowledge graph for the QA context and scene; "concept graph") multimodal knowledge. Existing works typically combine a scene graph and a concept graph of the scene by connecting corresponding visual nodes and concept nodes, then incorporate the QA context representation to perform question answering. However, these methods only perform a unidirectional fusion from unstructured knowledge to structured knowledge, limiting their potential to capture joint reasoning over the heterogeneous modalities of knowledge. To perform more expressive reasoning, we propose VQA-GNN, a new VQA model that performs bidirectional fusion between unstructured and structured multimodal knowledge to obtain unified knowledge representations. Specifically, we inter-connect the scene graph and the concept graph through a super node that represents the QA context, and introduce a new multimodal GNN technique to perform inter-modal message passing for reasoning that mitigates representational gaps between modalities. On two challenging VQA tasks (VCR and GQA), our method outperforms strong baseline VQA methods by 3.2% on VCR (Q-AR) and 4.6% on GQA, suggesting its strength in performing concept-level reasoning. Ablation studies further demonstrate the efficacy of the bidirectional fusion and multimodal GNN method in unifying unstructured and structured multimodal knowledge.

Computer Vision and Pattern Recognition,Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to perform concept - level reasoning more effectively in the Visual Question Answering (VQA) task. Specifically, it aims to improve the joint reasoning ability of multimodal knowledge by unifying unstructured knowledge (e.g., the context in questions and answers, "QA context") and structured knowledge (e.g., the knowledge graph of QA context and scenes, "concept graph"). Existing methods usually combine the scene graph and the concept graph by connecting the corresponding visual nodes and concept nodes, and then incorporate the QA context representation to perform question answering. However, these methods only perform one - way fusion from unstructured knowledge to structured knowledge, limiting their potential for joint reasoning across heterogeneous modal knowledge.

To achieve more effective reasoning, the authors propose VQA - GNN, a new VQA model that obtains a unified knowledge representation by bidirectionally fusing unstructured and structured multimodal knowledge. Specifically, the authors connect the scene graph and the concept graph through a super node (representing the QA context) and introduce a new multimodal graph neural network technique to perform cross - modal message passing, thereby reducing the representational gap between modalities and promoting deeper concept - level reasoning.

On two challenging VQA tasks (VCR and GQA), this method improves by 3.2% (the Q - AR metric of VCR) and 4.6% (GQA) respectively, indicating its advantage in performing concept - level reasoning. Ablation studies further demonstrate the effectiveness of the bidirectional fusion and multimodal GNN methods in unifying unstructured and structured multimodal knowledge.