Abstract:Abstract Multilingual task-oriented dialogue (ToD) facilitates access to services and information for many (communities of) speakers. Nevertheless, its potential is not fully realized, as current multilingual ToD datasets—both for modular and end-to-end modeling—suffer from severe limitations. 1) When created from scratch, they are usually small in scale and fail to cover many possible dialogue flows. 2) Translation-based ToD datasets might lack naturalness and cultural specificity in the target language. In this work, to tackle these limitations we propose a novel outline-based annotation process for multilingual ToD datasets, where domain-specific abstract schemata of dialogue are mapped into natural language outlines. These in turn guide the target language annotators in writing dialogues by providing instructions about each turn’s intents and slots. Through this process we annotate a new large-scale dataset for evaluation of multilingual and cross-lingual ToD systems. Our Cross-lingual Outline-based Dialogue dataset (cod) enables natural language understanding, dialogue state tracking, and end-to-end dialogue evaluation in 4 diverse languages: Arabic, Indonesian, Russian, and Kiswahili. Qualitative and quantitative analyses of cod versus an equivalent translation-based dataset demonstrate improvements in data quality, unlocked by the outline-based approach. Finally, we benchmark a series of state-of-the-art systems for cross-lingual ToD, setting reference scores for future work and demonstrating that cod prevents over-inflated performance, typically met with prior translation-based ToD datasets.

TED Multilingual Discourse Bank (TED-MDB): a parallel corpus annotated in the PDTB style

GDTB: Genre Diverse Data for English Shallow Discourse Parsing across Modalities, Text Types, and Domains

An Arabic-Hebrew parallel corpus of TED talks

Shallow Discourse Annotation for Chinese TED Talks

Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation

SciDTB: Discourse Dependency TreeBank for Scientific Abstracts

BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling

Improving Discourse Relation Projection to Build Discourse Annotated Corpora

Automatic Mapping of French Discourse Connectives to PDTB Discourse Relations

A description of Turkish Discourse Bank 1.2 and an examination of common dependencies in Turkish discourse

Discourse Centric Evaluation of Machine Translation with a Densely Annotated Parallel Corpus

Multi3WOZ: A Multilingual, Multi-Domain, Multi-Parallel Dataset for Training and Evaluating Culturally Adapted Task-Oriented Dialog Systems

Bianet: A Parallel News Corpus in Turkish, Kurdish and English

Extending the MPC Corpus to Chinese and Urdu - A Multiparty Multi-Lingual Chat Corpus for Modeling Social Phenomena in Language.

IndoToD: A Multi-Domain Indonesian Benchmark For End-to-End Task-Oriented Dialogue Systems

Many uses, many annotations for large speech corpora: Switchboard and TDT as case studies

Universal Discourse Representation Structure Parsing

TDMSci: A Specialized Corpus for Scientific Literature Entity Tagging of Tasks Datasets and Metrics

PTVD: A Large-Scale Plot-Oriented Multimodal Dataset Based on Television Dramas

Automatic Construction of Discourse Corpora for Dialogue Translation