Abstract:By offering a natural way for information seeking, multimodal dialogue systems are attracting increasing attention in several domains such as retail, travel etc. However, most existing dialogue systems are limited to textual modality, which cannot be easily extended to capture the rich semantics in visual modality such as product images. For example, in fashion domain, the visual appearance of clothes and matching styles play a crucial role in understanding the user's intention. Without considering these, the dialogue agent may fail to generate desirable responses for users. In this paper, we present a Knowledge-aware Multimodal Dialogue (KMD) model to address the limitation of text-based dialogue systems. It gives special consideration to the semantics and domain knowledge revealed in visual content, and is featured with three key components. First, we build a taxonomy-based learning module to capture the fine-grained semantics in images the category and attributes of a product). Second, we propose an end-to-end neural conversational model to generate responses based on the conversation history, visual semantics, and domain knowledge. Lastly, to avoid inconsistent dialogues, we adopt a deep reinforcement learning method which accounts for future rewards to optimize the neural conversational model. We perform extensive evaluation on a multi-turn task-oriented dialogue dataset in fashion domain. Experiment results show that our method significantly outperforms state-of-the-art methods, demonstrating the efficacy of modeling visual modality and domain knowledge for dialogue systems.

Read Key Points: Dialogue-Grounded Knowledge Points Generation with Multi-Level Salience-Aware Mixture

Open-domain Dialogue Generation Grounded with Dynamic Multi-form Knowledge Fusion

Topic-level knowledge sub-graphs for multi-turn dialogue generation

Knowledge-graph based Proactive Dialogue Generation with Improved Meta-learning

Generative Knowledge Selection for Knowledge-Grounded Dialogues

A Multi-Hop Reasoning Knowledge Selection Module for Dialogue Generation

Section-Aware Commonsense Knowledge-Grounded Dialogue Generation with Pre-trained Language Model.

Dual Semantic Knowledge Composed Multimodal Dialog Systems

More is Better: Enhancing Open-Domain Dialogue Generation via Multi-Source Heterogeneous Knowledge

Improving the Applicability of Knowledge-Enhanced Dialogue Generation Systems by Using Heterogeneous Knowledge from Multiple Sources

Knowledge Augmented Dialogue Generation with Divergent Facts Selection

Improving Open-Domain Dialogue Response Generation with Multi-Source Multilingual Commonsense Knowledge

Infusing internalized knowledge of language models into hybrid prompts for knowledgeable dialogue generation

Diverse and Informative Dialogue Generation with Context-Specific Commonsense Knowledge Awareness

Policy-driven Knowledge Selection and Response Generation for Document-grounded Dialogue

Knowledge-aware Dialogue Generation with Hybrid Attention (Student Abstract)

Difference-aware Knowledge Selection for Knowledge-grounded Conversation Generation

PluDG: enhancing task-oriented dialogue system with knowledge graph plug-in module

Dynamically Retrieving Knowledge via Query Generation for Informative Dialogue Generation

Know Deeper: Knowledge-Conversation Cyclic Utilization Mechanism for Open-domain Dialogue Generation

Knowledge-aware Multimodal Dialogue Systems.