Abstract:The popularity of multimodal dialogue has stimulated the need for a new generation of dialogue agents with multimodal interactivity.When users communicate with customer service, they may express their requirements by means of text, images, or even videos. Visual information usually acts as discriminators for product models, or indicators of product failures, which play an important role in the E-commerce scenario.On the other hand, detailed information provided by the images is limited, and typically, customer service systems cannot understand the intent of users without the input text.Thus, bridging the gap between the image and text is crucial for communicating with customers.In this paper, we construct JDDC 2.1, a large-scale multimodal multi-turn dialogue dataset collected from a mainstream Chinese E-commerce platform, containing about 246K dialogue sessions, 3M utterances, and 507K images, along with product knowledge bases and image category annotations. Over our dataset, we jointly define four tasks: the multimodal dialogue response generation task,the multimodal query rewriting task, the multimodal dialogue discourse parsing task, and the multimodal dialogue summarization task.JDDC 2.1 is the first corpus with annotations for all the above tasks over the same dialogue sessions, which facilitates the comprehensive research around the dialogue.In addition, we present several text-only and multimodal baselines and show the importance of visual information for these tasks. Our dataset and implements will be publicly available.

Overview of the NLPCC 2022 Shared Task: Multi-modal Dialogue Understanding and Generation

Overview of the NLPCC 2023 Shared Task 10: Learn to Watch TV: Multimodal Dialogue Understanding and Response Generation.

Overview of the NLPCC 2018 Shared Task: Multi-turn Human-Computer Conversations.

OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts

Multimodal Dialogue Understanding via Holistic Modeling and Sequence Labeling.

Overview of the NLPCC 2022 Shared Task: Dialogue Text Analysis (DTA)

JDDC 2.1: A Multimodal Chinese Dialogue Dataset with Joint Tasks of Query Rewriting, Response Generation, Discourse Parsing, and Summarization

MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation.

DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset

MultiWOZ 2.3: A Multi-Domain Task-Oriented Dialogue Dataset Enhanced with Annotation Corrections and Co-Reference Annotation

XDailyDialog: A Multilingual Parallel Dialogue Corpus

Overview of NLPCC Shared Task 2: Multi-perspective Scientific Machine Reading Comprehension

A Survey on Multimodal Dialogue Systems: Recent Advances and New Frontiers

Overview of the NLPCC 2018 Shared Task: Social Media User Modeling

MultiModal-GPT: A Vision and Language Model for Dialogue with Humans

Multi-Domain Dialogue Acts and Response Co-Generation

SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents

Multi-modal multi-hop interaction network for dialogue response generation

MULTI3NLU++: A Multilingual, Multi-Intent, Multi-Domain Dataset for Natural Language Understanding in Task-Oriented Dialogue

ChatterBox: Multi-round Multimodal Referring and Grounding

Multi-User MultiWOZ: Task-Oriented Dialogues among Multiple Users