Abstract:The objective of the Multiple Appropriate Facial Reaction Generation (MAFRG) task is to produce contextually appropriate and diverse listener facial behavioural responses based on the multimodal behavioural data of the conversational partner (i.e., the speaker). Current methodologies typically assume continuous availability of speech and facial modality data, neglecting real-world scenarios where these data may be intermittently unavailable, which often results in model failures. Furthermore, despite utilising advanced deep learning models to extract information from the speaker's multimodal inputs, these models fail to adequately leverage the speaker's emotional context, which is vital for eliciting appropriate facial reactions from human listeners. To address these limitations, we propose an Emotion-aware Modality Compensatory (EMC) framework. This versatile solution can be seamlessly integrated into existing models, thereby preserving their advantages while significantly enhancing performance and robustness in scenarios with missing modalities. Our framework ensures resilience when faced with missing modality data through the Compensatory Modality Alignment (CMA) module. It also generates more appropriate emotion-aware reactions via the Emotion-aware Attention (EA) module, which incorporates the speaker's emotional information throughout the entire encoding and decoding process. Experimental results demonstrate that our framework improves the appropriateness metric FRCorr by an average of 57.2\% compared to the original model structure. In scenarios where speech modality data is missing, the performance of appropriate generation shows an improvement, and when facial data is missing, it only exhibits minimal degradation.

Emotional Conversation Generation Orientated Syntactically Constrained Bidirectional-Asynchronous Framework

Emotional Conversation Generation Based on a Bayesian Deep Neural Network

SCBG: Semantic-Constrained Bidirectional Generation for Emotional Support Conversation

Neural Conversation Generation with Auxiliary Emotional Supervised Models

Generating More Interesting Responses in Neural Conversation Models with Distributional Constraints

Emotional Editing Constraint Conversation Content Generation Based on Reinforcement Learning.

An Emotional Dialogue System Using Conditional Generative Adversarial Networks with a Sequence-to-Sequence Transformer Encoder

Emotional Human-Machine Conversation Generation Based on Long Short-Term Memory

Affective Neural Response Generation

Emotional Neural Language Generation Grounded in Situational Contexts

Neural Response Generation with Relevant Emotions for Short Text Conversation

Emotional conversation generation with heterogeneous graph neural network

Chinese Emotional Dialogue Response Generation via Reinforcement Learning

Emotion-and-knowledge grounded response generation in an open-domain dialogue setting

Emotion-aware Chat Machine: Automatic Emotional Response Generation for Human-like Emotional Interaction

DLCEncDec : A Fully Character-Level Encoder-Decoder Model for Neural Responding Conversation.

CAB: Empathetic Dialogue Generation with Cognition, Affection and Behavior

Generating Responses with a Specific Emotion in Dialog.

Robust Facial Reactions Generation: An Emotion-Aware Framework with Modality Compensation

EmpCRL: Controllable Empathetic Response Generation via In-Context Commonsense Reasoning and Reinforcement Learning

Exploiting Emotion-Semantic Correlations for Empathetic Response Generation