Abstract:As an intelligent way to interact with computers, the dialog system has been catching more and more attention. However, most research efforts only focus on text-based dialog systems, completely ignoring the rich semantics conveyed by the visual cues. Indeed, the desire for multimodal task-oriented dialog systems is growing with the rapid expansion of many domains, such as the online retailing and travel. Besides, few work considers the hierarchical product taxonomy and the users' attention to products explicitly. The fact is that users tend to express their attention to the semantic attributes of products such as color and style as the dialog goes on. Towards this end, in this work, we present a hierarchical User attention-guided Multimodal Dialog system, named UMD for short. UMD leverages a bidirectional Recurrent Neural Network to model the ongoing dialog between users and chatbots at a high level; As to the low level, the multimodal encoder and decoder are capable of encoding multimodal utterances and generating multimodal responses, respectively. The multimodal encoder learns the visual presentation of images with the help of a taxonomy-attribute combined tree, and then the visual features interact with textual features through an attention mechanism; whereas the multimodal decoder selects the required visual images and generates textual responses according to the dialog history. To evaluate our proposed model, we conduct extensive experiments on a public multimodal dialog dataset in the retailing domain. Experimental results demonstrate that our model outperforms the existing state-of-the-art methods by integrating the multimodal utterances and encoding the visual features based on the users' attribute-level attention.

User Attention-guided Multimodal Dialog Systems

Knowledge-aware Multimodal Dialogue Systems.

Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic Elements

Multimodal Dialogue Systems via Capturing Context-aware Dependencies of Semantic Elements

A multimodal dialogue system for improving user satisfaction via knowledge-enriched response and image recommendation

Multimodal Dialog System: Relational Graph-based Context-aware Question Understanding

Multimodal Dialog System: Generating Responses via Adaptive Decoders

Multi-Modal Fusion with Multi-Level Attention for Visual Dialog.

A non-hierarchical attention network with modality dropout for textual response generation in multimodal dialogue systems

A Survey on Multimodal Dialogue Systems: Recent Advances and New Frontiers

Unified Multimodal Model with Unlikelihood Training for Visual Dialog.

Enhancing Product Representation with Multi-form Interactions for Multimodal Conversational Recommendation

Towards Building Large Scale Multimodal Domain-Aware Conversation Systems

HAIN: Multi-label Classification with Hierarchical Attention-based Interaction Network for Multi-turn Dialogue Texts

Neural Multimodal Belief Tracker with Adaptive Attention for Dialogue Systems

Multi-modal multi-hop interaction network for dialogue response generation

Multimodal Dialogue Generation Based on Transformer and Collaborative Attention

Dynamic Multi-level Attention Models for Dialogue Response Generation.

S3: A Simple Strong Sample-effective Multimodal Dialog System

Dual Semantic Knowledge Composed Multimodal Dialog Systems

Multi-View Attention Network for Visual Dialog