Abstract:Due to the natural multi-modal occurrence format (text, audio, vision) of the dialogues, textual response generation in dialogues should rely on the multi-modal contexts beyond text only. However, most existing studies normally ignore the rich information of other modalities, such as audio. To investigate the importance of the acoustic contexts, we explore the multi-modal dialogue scenario with aligned text and audio temporal sequences for textual response generation of an assumed system, namely RGMD task. To this end, we construct a new multi-modal dataset for this task based on TV shows, which contains 84.9K utterances. Considering the response diversity limited by the context and modality interactions for RGMD, we attempt the split pre-generation (SPG) strategy and the cross-modal contrastive learning (CCL) strategy in multi-modal pre-training for better response generation. On the one hand, with SPG, we can obtain many diverse responses without the restrictions of too many historical mixed multi-modal contexts. On the other hand, with CCL, we can capture the interactions between text and audio. Extensive experiments demonstrate that our approach based on BART can consistently perform better than the state-of-the-art textual approach DP by 4.17%, 8.96%, 2.43%, 1.04% and 7.54% regarding metrics of BLEU, DIST, ROUGE, METEOR and NIST, respectively. Moreover, our approach based on GPT can outperform the state-of-the-art multi-modal approach RLM by 6.79%, 9.25%, 7.49%, 9.31% and 13.75% regarding metrics of BLEU, DIST, ROUGE, METEOR and NIST, respectively. Besides, we conduct much in-depth analysis, showing the necessity of audio for response generation and further verifying the effectiveness of our approach.

Improving Open-Domain Dialogue Response Generation with Multi-Source Multilingual Commonsense Knowledge

Exploring the Effectiveness of Multi-Lingual Commonsense Knowledge-Aware Open-Domain Dialogue Response Generation.

Read Key Points: Dialogue-Grounded Knowledge Points Generation with Multi-Level Salience-Aware Mixture

Conversations Powered by Cross-Lingual Knowledge

More is Better: Enhancing Open-Domain Dialogue Generation via Multi-Source Heterogeneous Knowledge

Improving the Applicability of Knowledge-Enhanced Dialogue Generation Systems by Using Heterogeneous Knowledge from Multiple Sources

Select, Extend, and Generate: Generative Knowledge Selection for Open-Domain Dialogue Response Generation.

There Is No Standard Answer: Knowledge-Grounded Dialogue Generation with Adversarial Activated Multi-Reference Learning

Improving Knowledge-Aware Dialogue Response Generation by Using Human-Written Prototype Dialogues.

Knowledge Augmented Dialogue Generation with Divergent Facts Selection

A Knowledge Plug-and-Play Test Bed for Open-domain Dialogue Generation

Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model.

XDailyDialog: A Multilingual Parallel Dialogue Corpus

Open-domain Dialogue Generation Grounded with Dynamic Multi-form Knowledge Fusion

Insert Commonsense Knowledge Through Semantics for Dialogue Generation

Improving Empathetic Dialogue Generation by Dynamically Infusing Commonsense Knowledge.

A Multi-Hop Reasoning Knowledge Selection Module for Dialogue Generation

Reason first, then respond: Modular Generation for Knowledge-infused Dialogue

Read Then Respond: Multi-granularity Grounding Prediction for Knowledge-Grounded Dialogue Generation.

Large Language Models as Source Planner for Personalized Knowledge-grounded Dialogue

Response Generation in Multi-Modal Dialogues with Split Pre-Generation and Cross-Modal Contrasting