Abstract:The integration of conversational artificial intelligence (AI) into mental health care promises a new horizon for therapist-client interactions, aiming to closely emulate the depth and nuance of human conversations. Despite the potential, the current landscape of conversational AI is markedly limited by its reliance on single-modal data, constraining the systems' ability to empathize and provide effective emotional support. This limitation stems from a paucity of resources that encapsulate the multimodal nature of human communication essential for therapeutic counseling. To address this gap, we introduce the Multimodal Emotional Support Conversation (MESC) dataset, a first-of-its-kind resource enriched with comprehensive annotations across text, audio, and video modalities. This dataset captures the intricate interplay of user emotions, system strategies, system emotion, and system responses, setting a new precedent in the field. Leveraging the MESC dataset, we propose a general Sequential Multimodal Emotional Support framework (SMES) grounded in Therapeutic Skills Theory. Tailored for multimodal dialogue systems, the SMES framework incorporates an LLM-based reasoning model that sequentially generates user emotion recognition, system strategy prediction, system emotion prediction, and response generation. Our rigorous evaluations demonstrate that this framework significantly enhances the capability of AI systems to mimic therapist behaviors with heightened empathy and strategic responsiveness. By integrating multimodal data in this innovative manner, we bridge the critical gap between emotion recognition and emotional support, marking a significant advancement in conversational AI for mental health support.

Towards Emotional Support Dialog Systems

Towards Multimodal Emotional Support Conversation Systems

Improving Multi-turn Emotional Support Dialogue Generation with Lookahead Strategy Planning

ESCoT: Towards Interpretable Emotional Support Dialogue Systems

Facilitating Multi-turn Emotional Support Conversation with Positive Emotion Elicitation: A Reinforcement Learning Approach

Knowledge-enhanced Mixed-initiative Dialogue System for Emotional Support Conversations

AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation

Dynamic Demonstration Retrieval and Cognitive Understanding for Emotional Support Conversation

CauESC: A Causal Aware Model for Emotional Support Conversation

SCBG: Semantic-Constrained Bidirectional Generation for Emotional Support Conversation

Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversation

FEEL: A Framework for Evaluating Emotional Support Capability with Large Language Models

Towards Multi-Turn Empathetic Dialogs with Positive Emotion Elicitation

SweetieChat: A Strategy-Enhanced Role-playing Framework for Diverse Scenarios Handling Emotional Support Agent

DiagESC: Dialogue Synthesis for Integrating Depression Diagnosis into Emotional Support Conversation

Enhancing the conversational agent with an emotional support system for mental health digital therapeutics

Producing Considerate Responses: Progressive Staged Training for Emotional Support Conversation

AdMISC: Advanced Multi-Task Learning and Feature-Fusion for Emotional Support Conversation

Building Emotional Support Chatbots in the Era of LLMs

TransESC: Smoothing Emotional Support Conversation via Turn-Level State Transition

Generating Emotional Social Chatbot Responses with a Consistent Speaking Style