Abstract:The integration of conversational artificial intelligence (AI) into mental health care promises a new horizon for therapist-client interactions, aiming to closely emulate the depth and nuance of human conversations. Despite the potential, the current landscape of conversational AI is markedly limited by its reliance on single-modal data, constraining the systems' ability to empathize and provide effective emotional support. This limitation stems from a paucity of resources that encapsulate the multimodal nature of human communication essential for therapeutic counseling. To address this gap, we introduce the Multimodal Emotional Support Conversation (MESC) dataset, a first-of-its-kind resource enriched with comprehensive annotations across text, audio, and video modalities. This dataset captures the intricate interplay of user emotions, system strategies, system emotion, and system responses, setting a new precedent in the field. Leveraging the MESC dataset, we propose a general Sequential Multimodal Emotional Support framework (SMES) grounded in Therapeutic Skills Theory. Tailored for multimodal dialogue systems, the SMES framework incorporates an LLM-based reasoning model that sequentially generates user emotion recognition, system strategy prediction, system emotion prediction, and response generation. Our rigorous evaluations demonstrate that this framework significantly enhances the capability of AI systems to mimic therapist behaviors with heightened empathy and strategic responsiveness. By integrating multimodal data in this innovative manner, we bridge the critical gap between emotion recognition and emotional support, marking a significant advancement in conversational AI for mental health support.

SIMMC: Situated Interactive Multi-Modal Conversational Data Collection And Evaluation Platform

MMAC-Copilot: Multi-modal Agent Collaboration Operating System Copilot

MM-Conv: A Multi-modal Conversational Dataset for Virtual Humans

CNAMD Corpus: A Chinese Natural Audiovisual Multimodal Database of Conversations for Social Interactive Agents

CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society

DialSim: A Real-Time Simulator for Evaluating Long-Term Multi-Party Dialogue Understanding of Conversational Agents

Multi-User Chat Assistant (MUCA): a Framework Using LLMs to Facilitate Group Conversations

BIM Sim/3D: Multi-Agent Human Activity Simulation in Indoor Spaces

Accelerating the Development of Multimodal, Integrative-AI Systems with Platform for Situated Intelligence

Multi-Party Chat: Conversational Agents in Group Settings with Humans and Models

MMConv: An Environment for Multimodal Conversational Search across Multiple Domains

Exploring Smart Agents for the Interaction with Multimodal Mediated Environments

OpenOmni: A Collaborative Open Source Tool for Building Future-Ready Multimodal Conversational Agents

LMAgent: A Large-scale Multimodal Agents Society for Multi-user Simulation

Social Conjuring: Multi-User Runtime Collaboration with AI in Building Virtual 3D Worlds

cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers

SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems

Towards Multimodal Emotional Support Conversation Systems

SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Research

SocialMind: LLM-based Proactive AR Social Assistive System with Human-like Perception for In-situ Live Interactions

Med-PMC: Medical Personalized Multi-modal Consultation with a Proactive Ask-First-Observe-Next Paradigm