Learning WHO Saying WHAT to WHOM in Multi-Party Conversations

Jia-Chen Gu,Zhuosheng Zhang,Zhen-Hua Ling

DOI: https://doi.org/10.18653/v1/2023.ijcnlp-tutorials.5

2023-01-01

Abstract:Jia-Chen Gu, Zhuosheng Zhang, Zhen-Hua Ling. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: Tutorial Abstract. 2023.

What problem does this paper attempt to address?

Who is Speaking to Whom? Learning to Identify Utterance Addressee in Multi-Party Conversations.

Ran Le,Wenpeng Hu,Mingyue Shang,Zhenjun You,Lidong Bing,Dongyan Zhao,Rui Yan

DOI: https://doi.org/10.18653/v1/d19-1199

2019-01-01

Abstract:Previous research on dialogue systems generally focuses on the conversation between two participants, yet multi-party conversations which involve more than two participants within one session bring up a more complicated but realistic scenario. In real multi- party conversations, we can observe who is speaking, but the addressee information is not always explicit. In this paper, we aim to tackle the challenge of identifying all the miss- ing addressees in a conversation session. To this end, we introduce a novel who-to-whom (W2W) model which models users and utterances in the session jointly in an interactive way. We conduct experiments on the benchmark Ubuntu Multi-Party Conversation Corpus and the experimental results demonstrate that our model outperforms baselines with consistent improvements.
On-line Dialogue Policy Learning with Companion Teaching.

Lu Chen,Runzhe Yang,Cheng Chang,Zihao Ye,Xiang Zhou,Kai Yu

DOI: https://doi.org/10.18653/v1/e17-2032

2017-01-01

Abstract:Lu Chen, Runzhe Yang, Cheng Chang, Zihao Ye, Xiang Zhou, Kai Yu. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers. 2017.
Who Says What to Whom: A Survey of Multi-Party Conversations

Jia-Chen Gu,Chongyang Tao,Zhen-Hua Ling

DOI: https://doi.org/10.24963/ijcai.2022/768

2022-01-01

Abstract:Multi-party conversations (MPCs) are a more practical and challenging scenario involving more than two interlocutors. This research topic has drawn significant attention from both academia and industry, and it is nowadays counted as one of the most promising research areas in the field of dialogue systems. In general, MPC algorithms aim at addressing the issues of Who says What to Whom, specifically, who speaks, say what, and address whom. The complicated interactions between interlocutors, between utterances, and between interlocutors and utterances develop many variant tasks of MPCs worth investigation. In this paper, we present a comprehensive survey of recent advances in text-based MPCs. In particular, we first summarize recent advances on the research of MPC context modeling including dialogue discourse parsing, dialogue flow modeling and self-supervised training for MPCs. Then we review the state-of-the-art models categorized by Who says What to Whom in MPCs. Finally, we highlight the challenges which are not yet well addressed in MPCs and present future research directions.
When Less is More: Using Less Context Information to Generate Better Utterances in Group Conversations.

Haisong Zhang,Zhangming Chan,Yan Song,Dongyan Zhao,Rui Yan

DOI: https://doi.org/10.1007/978-3-319-99495-6_7

2018-01-01

Abstract:Previous research on dialogue systems generally focuses on the conversation between two participants. Yet, group conversations which involve more than two participants within one session bring up a more complicated situation. The scenario is real such as meetings or online chatting rooms. Learning to converse in groups is challenging due to different interaction patterns among users when they exchange messages with each other. Group conversations are structure-aware while the structure results from different interactions among different users. In this paper, we have an interesting observation that fewer contexts can lead to better performance by tackling the structure of group conversations. We conduct experiments on the public Ubuntu Multi-Party Conversation Corpus and the experiment results demonstrate that our model outperforms baselines.
Speaker-Aware Discourse Parsing on Multi-Party Dialogues.

Nan Yu,Guohong Fu,Min Zhang

2022-01-01

Abstract:Discourse parsing on multi-party dialogues is an important but difficult task in dialogue systems and conversational analysis. It is believed that speaker interactions are helpful for this task. However, most previous research ignores speaker interactions between different speakers. To this end, we present a speaker-aware model for this task. Concretely, we propose a speaker-context interaction joint encoding (SCIJE) approach, using the interaction features between different speakers. In addition, we propose a second-stage pre-training task, same speaker prediction (SSP), enhancing the conversational context representations by predicting whether two utterances are from the same speaker. Experiments on two standard benchmark datasets show that the proposed model achieves the best-reported performance in the literature. We will release the codes of this paper to facilitate future research.
Multilingual and multimodal mediation in online intercultural conversations: a translingual perspective

Mei-Ya Liang

DOI: https://doi.org/10.1080/09658416.2021.1941069

2021-06-30

Language Awareness

Abstract:<span>This study explored peer groups' mediation processes in the digital space. Building upon translingual research and multimodal discourse analysis, the researcher analysed video transcript excerpts of oral conversations and on-screen interactions among university students of different linguistic and cultural backgrounds. This article focuses on multilingual and multimodal mediation practices, which include polyaccented exemplification, bilingual interpretation, intercultural comparison, transnational definition, polyphonic performance and multimodal instruction (<i>N</i> = 24). The results show that the students attended to dynamic uses of English, Chinese and other languages as multilingual franca while communicating affect and stance through embodied expressions and shared screens in stylised multimodal discourses. The translingual analyses emphasise the students' collaborative transformations of English as a lingua franca (ELF) communication into playful interactions within digitally mediated networks of relationships. This study suggests that online peer mediation provides useful methods for developing university students' translingual awareness and capability for participating in intercultural conversations.</span>

linguistics
Towards Neural Speaker Modeling in Multi-Party Conversation: the Task, Dataset, and Models.

Zhao Meng,Lili Mou,Zhi Jin

DOI: https://doi.org/10.1609/aaai.v32i1.12140

2018-01-01

Proceedings of the AAAI Conference on Artificial Intelligence

Abstract:In this paper, we address the problem of speaker classification in multi-party conversation, and collect massive data to facilitate research in this direction. We further investigate temporal-based and content-based models of speakers, and propose several hybrids of them. Experiments show that speaker classification is feasible, and that hybrid models outperform each single component.
Joint Learning for Addressee Selection and Response Generation in Multi-Party Conversation

Qi Song,Sheng Li,Ping Wei,Ge Luo,Xinpeng Zhang,Zhenxing Qian

DOI: https://doi.org/10.1109/icassp43922.2022.9746498

2022-01-01

Abstract:A large number of multi-party conversation scenarios exist in social networks, which have been seldom studied in the field of human-machine conversation. In this paper, we study a novel task of joint learning for addressee selection and response generation in multi-party conversations. Systems are expected to select whom they address and generate the corresponding response. To solve it, we propose an end-to-end addressee selection and response generation (ASRG) model, containing an addressee selection module and a response generation module. In the selection module, we develop an addressee prediction attention scheme to obtain a unique context vector for each candidate, thereby calculating the probability of the candidate more accurately. In the generation module, we propose a Focus Transformer to generate responses. These two modules are jointly learnt to fully explore the correlations between addressee and response. Experimental results show ASRG remarkably outperforms baselines and generates relevant content for different addressees.
Learning to Disentangle Interleaved Conversational Threads with a Siamese Hierarchical Network and Similarity Ranking

Jyun-Yu Jiang,Francine Chen,Yan-Ying Chen,Wei Wang

DOI: https://doi.org/10.18653/v1/n18-1164

2018-01-01

Abstract:Jyun-Yu Jiang, Francine Chen, Yan-Ying Chen, Wei Wang. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.
Multi-Party Chat: Conversational Agents in Group Settings with Humans and Models

Jimmy Wei,Kurt Shuster,Arthur Szlam,Jason Weston,Jack Urbanek,Mojtaba Komeili

2023-06-09

Abstract:Current dialogue research primarily studies pairwise (two-party) conversations, and does not address the everyday setting where more than two speakers converse together. In this work, we both collect and evaluate multi-party conversations to study this more general case. We use the LIGHT environment to construct grounded conversations, where each participant has an assigned character to role-play. We thus evaluate the ability of language models to act as one or more characters in such conversations. Models require two skills that pairwise-trained models appear to lack: (1) being able to decide when to talk; (2) producing coherent utterances grounded on multiple characters. We compare models trained on our new dataset to existing pairwise-trained dialogue models, as well as large language models with few-shot prompting. We find that our new dataset, MultiLIGHT, which we will publicly release, can help bring significant improvements in the group setting.

Computation and Language,Machine Learning
Listen, Interact and Talk: Learning to Speak via Interaction

Haichao Zhang,Haonan Yu,Wei Xu

DOI: https://doi.org/10.48550/arXiv.1705.09906

2017-05-28

Computation and Language

Abstract:One of the long-term goals of artificial intelligence is to build an agent that can communicate intelligently with human in natural language. Most existing work on natural language learning relies heavily on training over a pre-collected dataset with annotated labels, leading to an agent that essentially captures the statistics of the fixed external training data. As the training data is essentially a static snapshot representation of the knowledge from the annotator, the agent trained this way is limited in adaptiveness and generalization of its behavior. Moreover, this is very different from the language learning process of humans, where language is acquired during communication by taking speaking action and learning from the consequences of speaking action in an interactive manner. This paper presents an interactive setting for grounded natural language learning, where an agent learns natural language by interacting with a teacher and learning from feedback, thus learning and improving language skills while taking part in the conversation. To achieve this goal, we propose a model which incorporates both imitation and reinforcement by leveraging jointly sentence and reward feedbacks from the teacher. Experiments are conducted to validate the effectiveness of the proposed approach.
Spoken Style Learning with Multi-modal Hierarchical Context Encoding for Conversational Text-to-Speech Synthesis.

Jingbei Li,Yuan Meng,Chenyi Li,Zhiyong Wu,Helen Meng,Chao Weng,Dan Su

2021-01-01

Abstract:For conversational text-to-speech (TTS) systems, it is vital that the systems can adjust the spoken styles of synthesized speech according to different content and spoken styles in historical conversations. However, the study about learning spoken styles from historical conversations is still in its infancy. Only the transcripts of the historical conversations are considered, which neglects the spoken styles in historical speeches. Moreover, only the interactions of the global aspect between speakers are modeled, missing the party aspect self interactions inside each speaker. In this paper, to achieve better spoken style learning for conversational TTS, we propose a spoken style learning approach with multi-modal hierarchical context encoding. The textual information and spoken styles in the historical conversations are processed through multiple hierarchical recurrent neural networks to learn the spoken style related features in global and party aspects. The attention mechanism is further employed to summarize these features into a conversational context encoding. Experimental results demonstrate the effectiveness of our proposed approach, which outperform a baseline method using context encoding learnt only from the transcripts in global aspects, with MOS score on the naturalness of synthesized speech increasing from 3.138 to 3.408 and ABX preference rate exceeding the baseline method by 36.45%.
A Self-Attention Joint Model for Spoken Language Understanding in Situational Dialog Applications

Mengyang Chen,Jin Zeng,Jie Lou

DOI: https://doi.org/10.48550/arXiv.1905.11393

2019-05-27

Computation and Language

Abstract:Spoken language understanding (SLU) acts as a critical component in goal-oriented dialog systems. It typically involves identifying the speakers intent and extracting semantic slots from user utterances, which are known as intent detection (ID) and slot filling (SF). SLU problem has been intensively investigated in recent years. However, these methods just constrain SF results grammatically, solve ID and SF independently, or do not fully utilize the mutual impact of the two tasks. This paper proposes a multi-head self-attention joint model with a conditional random field (CRF) layer and a prior mask. The experiments show the effectiveness of our model, as compared with state-of-the-art models. Meanwhile, online education in China has made great progress in the last few years. But there are few intelligent educational dialog applications for students to learn foreign languages. Hence, we design an intelligent dialog robot equipped with different scenario settings to help students learn communication skills.
Affective-NLI: Towards Accurate and Interpretable Personality Recognition in Conversation

Zhiyuan Wen,Jiannong Cao,Yu Yang,Ruosong Yang,Shuaiqi Liu

2024-04-03

Abstract:Personality Recognition in Conversation (PRC) aims to identify the personality traits of speakers through textual dialogue content. It is essential for providing personalized services in various applications of Human-Computer Interaction (HCI), such as AI-based mental therapy and companion robots for the elderly. Most recent studies analyze the dialog content for personality classification yet overlook two major concerns that hinder their performance. First, crucial implicit factors contained in conversation, such as emotions that reflect the speakers' personalities are ignored. Second, only focusing on the input dialog content disregards the semantic understanding of personality itself, which reduces the interpretability of the results. In this paper, we propose Affective Natural Language Inference (Affective-NLI) for accurate and interpretable PRC. To utilize affectivity within dialog content for accurate personality recognition, we fine-tuned a pre-trained language model specifically for emotion recognition in conversations, facilitating real-time affective annotations for utterances. For interpretability of recognition results, we formulate personality recognition as an NLI problem by determining whether the textual description of personality labels is entailed by the dialog content. Extensive experiments on two daily conversation datasets suggest that Affective-NLI significantly outperforms (by 6%-7%) state-of-the-art approaches. Additionally, our Flow experiment demonstrates that Affective-NLI can accurately recognize the speaker's personality in the early stages of conversations by surpassing state-of-the-art methods with 22%-34%.

Computation and Language,Artificial Intelligence
Extending the MPC Corpus to Chinese and Urdu - A Multiparty Multi-Lingual Chat Corpus for Modeling Social Phenomena in Language.

Ting Liu,Samira Shaikh,Tomek Strzalkowski,Aaron Broadwell,Jennifer Stromer-Galley,Sarah Taylor,Umit Boz,Xiaoai Ren,Jingsi Wu

2012-01-01

Language Resources and Evaluation

Abstract:In this paper, we report our efforts in building a multi-lingual multi-party online chat corpus (MMPC) in order to develop a firm understanding in a set of social constructs such as agenda control, influence, and leadership as well as to computationally model such constructs in online interactions. These automated models will help capture the dialogue dynamics that are essential for developing, among others, realistic human-machine dialogue systems, including autonomous virtual chat agents. In this paper, we first introduce our experiment design and data collection method in Chinese and Urdu, and then report on the current stage of our data collection. We annotated the collected corpus on four levels: communication links, dialogue acts, local topics, and meso-topics. Results from the analyses of annotated data on different languages indicate some interesting phenomena, which are reported in this paper.
Who Responded to Whom: The Joint Effects of Latent Topics and Discourse in Conversation Structure

Lu Ji,Jing Li,Zhongyu Wei,Qi Zhang,Xuanjing Huang

DOI: https://doi.org/10.48550/arXiv.2104.08601

2021-04-18

Abstract:Numerous online conversations are produced on a daily basis, resulting in a pressing need to conversation understanding. As a basis to structure a discussion, we identify the responding relations in the conversation discourse, which link response utterances to their initiations. To figure out who responded to whom, here we explore how the consistency of topic contents and dependency of discourse roles indicate such interactions, whereas most prior work ignore the effects of latent factors underlying word occurrences. We propose a model to learn latent topics and discourse in word distributions, and predict pairwise initiation-response links via exploiting topic consistency and discourse dependency. Experimental results on both English and Chinese conversations show that our model significantly outperforms the previous state of the arts, such as 79 vs. 73 MRR on Chinese customer service dialogues. We further probe into our outputs and shed light on how topics and discourse indicate conversational user interactions.

Computation and Language
Multimodal Dialogue Understanding via Holistic Modeling and Sequence Labeling.

Chenran Cai,Qin Zhao,Ruifeng Xu,Bing Qin

DOI: https://doi.org/10.1007/978-3-031-44699-3_36

2023-01-01

Abstract:This paper introduces the experimental schemes of Team HLT-base for the NLPCC-2023-Shared-Task-10 Learn to Watch TV: Multimodal Dialogue Understanding and Response Prediction (MDUG) competition. In this paper, we focus on two subtasks of multimodal dialogue understanding: the dialogue scene identification task and the dialogue session identification task. To solve these subtasks, we propose a simple and efficient multimodal framework, where two points are taken into account: i.e., modeling the interaction of different utterances and effectively fusing the information of different modalities. For the former, we concatenate all utterances into a single sentence and feed it into the pre-trained model; for the latter, we use a transformer layer to fuse the multimodal features. Extensive experiments show that our proposed framework achieves state-of-the-art (SOTA) performance compared with other competitive methods, and ranks 1st in both subtasks (i.e., track1: dialogue scene identification and track2: dialogue session identification) in the MDUG competition.
COSPLAY: Concept Set Guided Personalized Dialogue Generation Across Both Party Personas

Chen Xu,Piji Li,Wei Wang,Haoran Yang,Siyun Wang,Chuangbai Xiao

DOI: https://doi.org/10.48550/arXiv.2205.00872

2022-05-02

Computation and Language

Abstract:Maintaining a consistent persona is essential for building a human-like conversational model. However, the lack of attention to the partner makes the model more egocentric: they tend to show their persona by all means such as twisting the topic stiffly, pulling the conversation to their own interests regardless, and rambling their persona with little curiosity to the partner. In this work, we propose COSPLAY(COncept Set guided PersonaLized dialogue generation Across both partY personas) that considers both parties as a "team": expressing self-persona while keeping curiosity toward the partner, leading responses around mutual personas, and finding the common ground. Specifically, we first represent self-persona, partner persona and mutual dialogue all in the concept sets. Then, we propose the Concept Set framework with a suite of knowledge-enhanced operations to process them such as set algebras, set expansion, and set distance. Based on these operations as medium, we train the model by utilizing 1) concepts of both party personas, 2) concept relationship between them, and 3) their relationship to the future dialogue. Extensive experiments on a large public dataset, Persona-Chat, demonstrate that our model outperforms state-of-the-art baselines for generating less egocentric, more human-like, and higher quality responses in both automatic and human evaluations.
Learning to communicate about shared procedural abstractions

William P. McCarthy,Robert D. Hawkins,Haoliang Wang,Cameron Holdaway,Judith E. Fan

DOI: https://doi.org/10.48550/arXiv.2107.00077

2021-06-30

Computation and Language

Abstract:Many real-world tasks require agents to coordinate their behavior to achieve shared goals. Successful collaboration requires not only adopting the same communicative conventions, but also grounding these conventions in the same task-appropriate conceptual abstractions. We investigate how humans use natural language to collaboratively solve physical assembly problems more effectively over time. Human participants were paired up in an online environment to reconstruct scenes containing two block towers. One participant could see the target towers, and sent assembly instructions for the other participant to reconstruct. Participants provided increasingly concise instructions across repeated attempts on each pair of towers, using higher-level referring expressions that captured each scene's hierarchical structure. To explain these findings, we extend recent probabilistic models of ad-hoc convention formation with an explicit perceptual learning mechanism. These results shed light on the inductive biases that enable intelligent agents to coordinate upon shared procedural abstractions.
Corpus and Computational Methods for Usage-Based Chinese Language Learning: Toward a Professional Multilingualism

Zhuo Jing-Schmidt

DOI: https://doi.org/10.1007/978-981-13-3570-9_2

2019-01-01

Abstract:Language is a complex functional adaptive system (Beckner et al. in Lang Learn 59:1–26, 2009; Ellis in Usage-based perspectives on second language learning, De Gruyter, Berlin/Boston, pp 49–73, 2015). Language learning is learning to use a complex system Larsen-Freeman (Alternative approaches to second language acquisition, Routledge, London, pp 47–72, 2011). It is a multidimensional task involving social cognitive processes that interact both in time and space MacWhinney (Usage based perspectives on second language learning, De Gruyter, Berlin, pp 19–48, 2015). Language learning in the twenty-first century is “enmeshed in globalization, technologization, and mobility” and hence “emergent, dynamic, unpredictable, open ended, and intersubjectively negotiated” (The Douglas Fir Group in Mod Lang J 100:19–47, 2016: 19). Accordingly, the field of language teaching is more interdisciplinary than ever. It requires not only knowledge of language as a functional system and knowledge of language learning as a human socio-cognitive endeavor but also expanded communication across domains traditionally separated by differences in the methodology of knowledge construction. This chapter focuses on the usage-based model of language and language learning within a broader cognitive theory of linguistic knowledge and its development. The goal is to establish a theoretical relevancy and methodological necessity of corpus and computational methods to the knowledge base for language pedagogy as a practical field. In doing so, this chapter serves to ground the application of such methods as part of a professional multilingualism that informs the learning, teaching, and assessment of Chinese as a second language.

Learning WHO Saying WHAT to WHOM in Multi-Party Conversations

Who is Speaking to Whom? Learning to Identify Utterance Addressee in Multi-Party Conversations.

On-line Dialogue Policy Learning with Companion Teaching.

Who Says What to Whom: A Survey of Multi-Party Conversations

When Less is More: Using Less Context Information to Generate Better Utterances in Group Conversations.

Speaker-Aware Discourse Parsing on Multi-Party Dialogues.

Multilingual and multimodal mediation in online intercultural conversations: a translingual perspective

Towards Neural Speaker Modeling in Multi-Party Conversation: the Task, Dataset, and Models.

Joint Learning for Addressee Selection and Response Generation in Multi-Party Conversation

Learning to Disentangle Interleaved Conversational Threads with a Siamese Hierarchical Network and Similarity Ranking

Multi-Party Chat: Conversational Agents in Group Settings with Humans and Models

Listen, Interact and Talk: Learning to Speak via Interaction

Spoken Style Learning with Multi-modal Hierarchical Context Encoding for Conversational Text-to-Speech Synthesis.

A Self-Attention Joint Model for Spoken Language Understanding in Situational Dialog Applications

Affective-NLI: Towards Accurate and Interpretable Personality Recognition in Conversation

Extending the MPC Corpus to Chinese and Urdu - A Multiparty Multi-Lingual Chat Corpus for Modeling Social Phenomena in Language.

Who Responded to Whom: The Joint Effects of Latent Topics and Discourse in Conversation Structure

Multimodal Dialogue Understanding via Holistic Modeling and Sequence Labeling.

COSPLAY: Concept Set Guided Personalized Dialogue Generation Across Both Party Personas

Learning to communicate about shared procedural abstractions

Corpus and Computational Methods for Usage-Based Chinese Language Learning: Toward a Professional Multilingualism