Abstract:Affective computing is an important branch in the field of artificial intelligence(AI).It aims to build a computa-tional system that can automatically perceive,recognize,understand,and provide feedback on human emotions.It involves the intersection of multiple disciplines such as computer science,neuroscience,psychology,and social science.Deep emotional understanding and interaction can enable computers to better understand and respond to human emotional needs.It can also provide personalized interactions and feedback based on emotional states,which enhances the human-computer interaction experience.It has various applications in areas such as intelligent assistants,virtual reality,and smart healthcare.Relying solely on single-modal information,such as speech signal or video,does not align with the way humans perceive emotions.The accuracy of recognition rapidly decreases when faced with interference.Multimodal emo-tion understanding and interaction technologies aim to fully model multidimensional information from audio,video,and physiological signals to achieve more accurate emotion understanding.This technology is fundamental and an important prerequisite for achieving natural,human-like,and personalized human-computer interaction.It holds significant value for ushering in the era of intelligence and digitalization.Multimodal fusion for sentiment recognition receives increasing atten-tion from researchers in fully exploiting the complementary nature of different modalities.This study introduces the current research status of multimodal sentiment computation from three dimensions:an overview of multimodal sentiment recogni-tion,multimodal sentiment understanding,and detection and assessment of emotional disorders such as depression.The overview of emotion recognition is elaborated from the aspects of academic definition,mainstream datasets,and interna-tional competitions.In recent years,large language models(LLMs)have demonstrated excellent modeling capabilities and achieved great success in the field of natural language processing with their outstanding language understanding and reason-ing abilities.LLMs have garnered widespread attention because of their ability to handle various complex tasks by under-standing prompts with minimal or zero-shot learning.Through methods such as self-supervised learning or contrastive learn-ing,LLMs can learn more expressive multimodal representations,which can capture the correlations between different modalities and emotional information.Multimodal sentiment recognition and understanding are discussed in terms of emo-tion feature extraction,multimodal fusion,and the representation and models involved in sentiment recognition under the background of pre-trained large models.With the rapid development of society,people are facing increasing pressure,which can lead to feelings of depression,anxiety,and other negative emotions.Those who are in a prolonged state of depression and anxiety are more likely to develop mental illnesses.Depression is a common and serious condition,with symptoms including low mood,poor sleep quality,loss of appetite,fatigue,and difficulty concentrating.Depression not only harms individuals and families but also causes significant economic losses to society.The detection of emotional disor-ders starts from specific applications,which selects depression as the most common emotional disorder.We analyze its lat-est developments and trends from the perspectives of assessment and intervention.In addition,this study provides a detailed comparison of the research status of affective computation domestically,and prospects for future development trends are offered.We believe that scalable emotion feature design and large-scale model transfer learning based methods will be the future directions of development.The main challenge in multimodal emotion recognition lies in data scarcity,which means that data available to build and explore complex models are insufficient.This insufficiency causes difficulty in creating robust models based on deep neural network methods.The above mentioned issues can be addressed by construct-ing large-scale multimodal emotion databases and exploring transfer learning methods based on large models.By transfer-ring knowledge learned from unsupervised tasks or other tasks to emotion recognition tasks,the problem of limited data resources can be alleviated.The use of explicit discrete and dimensional labels to represent ambiguous emotional states has limitations due to the inherent fuzziness of emotions.Enhancing the interpretability of prediction results to improve the reli-ability of recognition results is also an important research direction for the future.The role of multimodal emotion comput-ing in addressing emotional disorders such as depression and anxiety is increasingly prominent.Future research can be con-ducted in the following three areas.First,research and construction of multimodal emotion disorder datasets can provide a solid foundation for the automatic recognition of emotional disorders.However,this field still needs to address challenges such as data privacy and ethics.In addition,considerations such as designing targeted interview questions,ensuring patient safety during data collection,and sample augmentation through algorithms are still worth exploring.Second,more effective algorithms should be developed.Emotional disorders fall within the psychological domain,and they can also affect the physiological features of patients,such as voice and body movements.This psychological-physiological correla-tion is worthy of comprehensive exploration.Therefore,improving the accuracy of algorithms for multimodal emotion disor-der recognition is a pressing research issue.Finally,intelligent psychological intervention systems should be designed and implemented.The following issues can be further studied:effectively simulating the counseling process of a psychologist,promptly receiving user emotional feedback,and generating empathetic conversations.

Multimodal Emotion Recognition in the Metaverse Era: New Needs and Transformation in Mental Health Work

The Metaverse: Exploring the Convergence of Technologies and Social Dynamics

Mental health in the virtual world: Challenges and opportunities in the metaverse era

The Empathic Metaverse: An Assistive Bioresponsive Platform For Emotional Experience Sharing

The promise of the metaverse in mental health: the new era of MEDverse

The Best Predictor of the Future—the Metaverse, Mental Health, and Lessons Learned From Current Technologies

A Comprehensive Review of Multimodal XR Applications, Risks, and Ethical Challenges in the Metaverse

Therapeutic effect of the metaverse on mental health

Understanding Emotional Hijacking in Metaverse

Development of Multimodal Sentiment Recognition and Understanding

A Review of Key Technologies for Emotion Analysis Using Multimodal Information

Are you caught in the dilemma of metaverse avatars? The impact of individuals' congruity perceptions on paradoxical emotions and actual behaviors

Socio-Digital Challenges for Social Work in the Metaverse

Taking modern psychiatry into the metaverse: Integrating augmented, virtual, and mixed reality technologies into psychiatric care

Metaverse beyond the hype: Multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy

Metaverse in Mental Health: The Beginning of a Long History

Exploring contactless techniques in multimodal emotion recognition: insights into diverse applications, challenges, solutions, and prospects

Mixed Reality for a collective and adaptive mental health metaverse

A new technology perspective of the Metaverse: Its essence, framework and challenges

The Metaverse: A new digital frontier for consumer behavior

Metaverse Servicescape: Emotional Response to Virtual Retail Design