Abstract:As an intelligent way to interact with computers, the dialog system has been catching more and more attention. However, most research efforts only focus on text-based dialog systems, completely ignoring the rich semantics conveyed by the visual cues. Indeed, the desire for multimodal task-oriented dialog systems is growing with the rapid expansion of many domains, such as the online retailing and travel. Besides, few work considers the hierarchical product taxonomy and the users' attention to products explicitly. The fact is that users tend to express their attention to the semantic attributes of products such as color and style as the dialog goes on. Towards this end, in this work, we present a hierarchical User attention-guided Multimodal Dialog system, named UMD for short. UMD leverages a bidirectional Recurrent Neural Network to model the ongoing dialog between users and chatbots at a high level; As to the low level, the multimodal encoder and decoder are capable of encoding multimodal utterances and generating multimodal responses, respectively. The multimodal encoder learns the visual presentation of images with the help of a taxonomy-attribute combined tree, and then the visual features interact with textual features through an attention mechanism; whereas the multimodal decoder selects the required visual images and generates textual responses according to the dialog history. To evaluate our proposed model, we conduct extensive experiments on a public multimodal dialog dataset in the retailing domain. Experimental results demonstrate that our model outperforms the existing state-of-the-art methods by integrating the multimodal utterances and encoding the visual features based on the users' attribute-level attention.

Dynamic Multi-level Attention Models for Dialogue Response Generation.

Deep Reinforcement Learning for Dialogue Generation

Multi-turn Dialogue Model Based on the Improved Hierarchical Recurrent Attention Network

A Static and Dynamic Attention Framework for Multi Turn Dialogue Generation

Multi-Turn Response Selection for Chatbots with Deep Attention Matching Network.

A non-hierarchical attention network with modality dropout for textual response generation in multimodal dialogue systems

Neural Dialogue Model with Retrieval Attention for Personalized Response Generation

Towards Robust Online Dialogue Response Generation

Augmenting Dialogue Response Generation with Unstructured Textual Knowledge

Survey of Research on End-to-End Emotional Dialogue Generation

Hierarchical Interactive Matching Network for Multi-turn Response Selection in Retrieval-Based Chatbots

Improve the Response Diversity of Multi-turn Dialogue System by Combining Knowledge

Turn-Level Recurrence Self-attention for Joint Dialogue Action Prediction and Response Generation.

Get The Point of My Utterance! Learning Towards Effective Responses with Multi-Head Attention Mechanism

Topic-Aware Dialogue Generation with Two-Hop Based Graph Attention.

User Attention-guided Multimodal Dialog Systems

Multi-convolutional Attention Networks for Dialogue Q &A Extraction

Deep Hybrid Networks Based Response Selection for Multi-turn Dialogue Systems

Multi-turn Dialogue Generation Using Self-attention and Nonnegative Matrix Factorization

Deep context modeling for multi-turn response selection in dialogue systems

Improving Contextual Language Models for Response Retrieval in Multi-Turn Conversation