Abstract:Different from Visual Question Answering task that requires to answer only one question about an image, Visual Dialogue task involves multiple rounds of dialogues which cover a broad range of visual content that could be related to any objects, relationships or high-level semantics. Thus one of the key challenges in Visual Dialogue task is to learn a more comprehensive and semantic-rich image representation that can adaptively attend to the visual content referred by variant questions. In this paper, we first propose a novel scheme to depict an image from both visual and semantic views. Specifically, the visual view aims to capture the appearance-level information in an image, including objects and their visual relationships, while the semantic view enables the agent to understand high-level visual semantics from the whole image to the local regions. Furthermore, on top of such dual-view image representations, we propose a Dual Encoding Visual Dialogue (DualVD) module, which is able to adaptively select question-relevant information from the visual and semantic views in a hierarchical mode. To demonstrate the effectiveness of DualVD, we propose two novel visual dialogue models by applying it to the Late Fusion framework and Memory Network framework. The proposed models achieve state-of-the-art results on three benchmark datasets. A critical advantage of the DualVD module lies in its interpretability. We can analyze which modality (visual or semantic) has more contribution in answering the current question by explicitly visualizing the gate values. It gives us insights in understanding of information selection mode in the Visual Dialogue task. The code is available at https://github.com/JXZe/Learning_DualVD.

Context Gating with Multi-Level Ranking Learning for Visual Dialog

Exploring Contextual-Aware Representation and Linguistic-Diverse Expression for Visual Dialog.

SKANet - Structured Knowledge-Aware Network for Visual Dialog.

New Datasets and Models for Contextual Reasoning in Visual Dialog.

Multi-Modal Fusion with Multi-Level Attention for Visual Dialog.

Multi-Granularity Semantic Collaborative Reasoning Network for Visual Dialog

Learning to Ground Visual Objects for Visual Dialog

Context-Aware Graph Inference with Knowledge Distillation for Visual Dialog

Iterative Context-Aware Graph Inference for Visual Dialog

Aligning Vision-Language for Graph Inference in Visual Dialog.

Reciprocal Question Representation Learning Network for Visual Dialog

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Multi-Modal Dialogue State Tracking for Playing GuessWhich Game

Integrating Historical States and Co-attention Mechanism for Visual Dialog

HVLM: Exploring Human-Like Visual Cognition and Language-Memory Network for Visual Dialog

Small Changes Make Big Differences: Improving Multi-turn Response Selection in Dialogue Systems Via Fine-Grained Contrastive Learning

Modality-Balanced Models for Visual Dialogue

Learning Dual Encoding Model for Adaptive Visual Understanding in Visual Dialogue

Visual Question Answering with Gated Relation-Aware Auxiliary

Recurrent Attention Network with Reinforced Generator for Visual Dialog

Multi-modal Contextual Graph Neural Network for Text Visual Question Answering.