Abstract:Multimodal dialogue system, due to its many-fold applications, has gained much attention to the researchers and developers in recent times. With the release of large-scale multimodal dialog dataset Saha et al. 2018 on the fashion domain, it has been possible to investigate the dialogue systems having both textual and visual modalities. Response generation is an essential aspect of every dialogue system, and making the responses diverse is an important problem. For any goal-oriented conversational agent, the system's responses must be informative, diverse and polite, that may lead to better user experiences. In this paper, we propose an end-to-end neural framework for generating varied responses in a multimodal dialogue setup capturing information from both the text and image. Multimodal encoder with co-attention between the text and image is used for focusing on the different modalities to obtain better contextual information. For effective information sharing across the modalities, we combine the information of text and images using the BLOCK fusion technique that helps in learning an improved multimodal representation. We employ stochastic beam search with Gumble Top K-tricks to achieve diversified responses while preserving the content and politeness in the responses. Experimental results show that our proposed approach performs significantly better compared to the existing and baseline methods in terms of distinct metrics, and thereby generates more diverse responses that are informative, interesting and polite without any loss of information. Empirical evaluation also reveals that images, while used along with the text, improve the efficiency of the model in generating diversified responses.

Learning to balance the coherence and diversity of response generation in generation-based chatbots

Deep Reinforcement Learning for Dialogue Generation

Generating Emotional Coherence and Diverse Responses in a Multimodal Dialogue System

Towards Robust Online Dialogue Response Generation

Improving the Responses Diversity of Persona-based Neural Conversation System: A Conditional Variational Autoencoders-based Approach

PVGRU: Generating Diverse and Relevant Dialogue Responses Via Pseudo-Variational Mechanism

Exploiting Emotion for Diverse Response Generation in Multi-turn Dialogues

A Novel Two-Stage Generation Framework for Promoting the Persona-Consistency and Diversity of Responses in Neural Dialog Systems

Towards Multimodal Response Generation with Exemplar Augmentation and Curriculum Optimization

Improve the Response Diversity of Multi-turn Dialogue System by Combining Knowledge

DiverseDialogue: A Methodology for Designing Chatbots with Human-Like Diversity

Group Linguistic Bias Aware Neural Response Generation.

Generating Multiple Diverse Responses for Short-Text Conversation

Towards Robustness and Diversity: Continual Learning in Dialog Generation with Text-Mixup and Batch Nuclear-Norm Maximization

Answer-guided and Semantic Coherent Question Generation in Open-domain Conversation.

Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders

Knowledge dialogue generation with multi-head attention mechanism

Generate, Evaluate, and Select: A Dialogue System with a Response Evaluator for Diversity-Aware Response Generation

Emotional Dialogue Generation Based on Conditional Variational Autoencoder and Dual Emotion Framework

A Multi-view Meta-learning Approach for Multi-modal Response Generation

More to diverse: Generating diversified responses in a task oriented multimodal dialog system