Abstract:With the rapid advancement of image captioning and visual question answering at single-round level, the question of how to generate multi-round dialogue about visual content has not yet been well <a class="link-external link-http" href="http://explored.Existing" rel="external noopener nofollow">this http URL</a> visual dialogue methods encode the image into a fixed feature vector directly, concatenated with the question and history embeddings to predict the <a class="link-external link-http" href="http://response.Some" rel="external noopener nofollow">this http URL</a> recent methods tackle the co-reference resolution problem using co-attention mechanism to cross-refer relevant elements from the image, history, and the target <a class="link-external link-http" href="http://question.However" rel="external noopener nofollow">this http URL</a>, it remains challenging to reason visual relationships, since the fine-grained object-level information is omitted before co-attentive reasoning. In this paper, we propose an object relationship discovery (ORD) framework to preserve the object interactions for visual dialogue generation. Specifically, a hierarchical graph convolutional network (HierGCN) is proposed to retain the object nodes and neighbour relationships locally, and then refines the object-object connections globally to obtain the final graph embeddings. A graph attention is further incorporated to dynamically attend to this graph-structured representation at the response reasoning stage. Extensive experiments have proved that the proposed method can significantly improve the quality of dialogue by utilising the contextual information of visual relationships. The model achieves superior performance over the state-of-the-art methods on the Visual Dialog dataset, increasing MRR from 0.6222 to 0.6447, and recall@1 from 48.48% to 51.22%.

VisDG: Towards Spontaneous Visual Dialogue Generation

OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts

ORD: Object Relationship Discovery for Visual Dialogue Generation

SInViG: A Self-Evolving Interactive Visual Agent for Human-Robot Interaction

Multimodal Incremental Transformer with Visual Grounding for Visual Dialogue Generation

The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training

OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts

GoG: Relation-aware Graph-over-Graph Network for Visual Dialog.

DAM: Deliberation, Abandon and Memory Networks for Generating Detailed and Non-repetitive Responses in Visual Dialogue

From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models

KBGN: Knowledge-Bridge Graph Network for Adaptive Vision-Text Reasoning in Visual Dialogue

Hybrid Graph Reasoning with Dynamic Interaction for Visual Dialog

Visualizing Dialogues: Enhancing Image Selection through Dialogue Understanding with Large Language Models

Open Domain Dialogue Generation with Latent Images

Multi-Modal Dialogue State Tracking for Playing GuessWhich Game

Exploring Contextual-Aware Representation and Linguistic-Diverse Expression for Visual Dialog.

Recurrent Attention Network with Reinforced Generator for Visual Dialog

Aligning Vision-Language for Graph Inference in Visual Dialog.

Decoupled Spatial Temporal Graphs for Generic Visual Grounding

Guide and Interact: Scene-Graph Based Generation and Control of Video Captions

Learning Dual Encoding Model for Adaptive Visual Understanding in Visual Dialogue