Affect is Computable——A Survey on Affective Computing

Lin Xue-yin

2008-01-01

Computer Science

Abstract:Affective computing is crucial for the implementation of natural human computer interaction.A survey is given on the background,related research areas and key techniques of affective computing.Some future research direc- tions are discussed.

What problem does this paper attempt to address?

A Comprehensive Survey on Affective Computing; Challenges, Trends, Applications, and Future Directions

Sitara Afzal,Haseeb Ali Khan,Imran Ullah Khan,Md. Jalil Piran,Jong Weon Lee

DOI: https://doi.org/10.48550/arXiv.2305.07665

2023-05-08

Abstract:As the name suggests, affective computing aims to recognize human emotions, sentiments, and feelings. There is a wide range of fields that study affective computing, including languages, sociology, psychology, computer science, and physiology. However, no research has ever been done to determine how machine learning (ML) and mixed reality (XR) interact together. This paper discusses the significance of affective computing, as well as its ideas, conceptions, methods, and outcomes. By using approaches of ML and XR, we survey and discuss recent methodologies in affective computing. We survey the state-of-the-art approaches along with current affective data resources. Further, we discuss various applications where affective computing has a significant impact, which will aid future scholars in gaining a better understanding of its significance and practical relevance.

Artificial Intelligence
A systematic review on affective computing: emotion models, databases, and recent advances

Yan Wang,Wei Song,Wei Tao,Antonio Liotta,Dawei Yang,Xinlei Li,Shuyong Gao,Yixuan Sun,Weifeng Ge,Wei Zhang,Wenqiang Zhang

DOI: https://doi.org/10.1016/j.inffus.2022.03.009

IF: 18.6

2022-07-01

Information Fusion

Abstract:Affective computing conjoins the research topics of emotion recognition and sentiment analysis, and can be realized with unimodal or multimodal data, consisting primarily of physical information (e.g., text, audio, and visual) and physiological signals (e.g., EEG and ECG). Physical-based affect recognition caters to more researchers due to the availability of multiple public databases, but it is challenging to reveal one's inner emotion hidden purposefully from facial expressions, audio tones, body gestures, etc. Physiological signals can generate more precise and reliable emotional results; yet, the difficulty in acquiring these signals hinders their practical application. Besides, by fusing physical information and physiological signals, useful features of emotional states can be obtained to enhance the performance of affective computing models. While existing reviews focus on one specific aspect of affective computing, we provide a systematical survey of important components: emotion models, databases, and recent advances. Firstly, we introduce two typical emotion models followed by five kinds of commonly used databases for affective computing. Next, we survey and taxonomize state-of-the-art unimodal affect recognition and multimodal affective analysis in terms of their detailed architectures and performances. Finally, we discuss some critical aspects of affective computing and its applications and conclude this review by pointing out some of the most promising future directions, such as the establishment of benchmark database and fusion strategies. The overarching goal of this systematic review is to help academic and industrial researchers understand the recent advances as well as new developments in this fast-paced, high-impact domain.

computer science, artificial intelligence, theory & methods
A Survey on Personalized Affective Computing in Human-Machine Interaction

Jialin Li,Alia Waleed,Hanan Salam

2023-04-02

Abstract:In computing, the aim of personalization is to train a model that caters to a specific individual or group of people by optimizing one or more performance metrics and adhering to specific constraints. In this paper, we discuss the need for personalization in affective and personality computing (hereinafter referred to as affective computing). We present a survey of state-of-the-art approaches for personalization in affective computing. Our review spans training techniques and objectives towards the personalization of affective computing models. We group existing approaches into seven categories: (1) Target-specific Models, (2) Group-specific Models, (3) Weighting-based Approaches, (4) Fine-tuning Approaches, (5) Multitask Learning, (6) Generative-based Models, and (7) Feature Augmentation. Additionally, we provide a statistical meta-analysis of the surveyed literature, analyzing the prevalence of different affective computing tasks, interaction modes, interaction contexts, and the level of personalization among the surveyed works. Based on that, we provide a road-map for those who are interested in exploring this direction.

Human-Computer Interaction,Artificial Intelligence,Computer Vision and Pattern Recognition,Machine Learning
Investigation on Experiment Design and Affect Measure in Affective Computing

Lin Xue-yin

2009-01-01

Journal of Image and Graphics

Abstract:The purpose of affective computing is to enable computers to perceive,understand and express emotions.Affect data acquisition and affect measure are essential to the research of affective computing.To explore the problem of affect coding,experiments were elaborately performed in the scenario of FIFA football electronic game.Firstly natural affect of game players were elicited in the game environment.Then a scheme of affect coding based on emotion prototypes was proposed,and both categorical emotions and levels within the affective space were assessed by game players as well as third-party observers.Finally an analysis was performed on the inter-observer agreement.The experiment data showed that valid and consistent affect coding results can be attained with the proposed method.
Affective Computing for Healthcare: Recent Trends, Applications, Challenges, and Beyond

Yuanyuan Liu,Ke Wang,Lin Wei,Jingying Chen,Yibing Zhan,Dapeng Tao,Zhe Chen

DOI: https://doi.org/10.48550/arXiv.2402.13589

2024-02-21

Abstract:Affective computing, which aims to recognize, interpret, and understand human emotions, provides benefits in healthcare, such as improving patient care and enhancing doctor-patient communication. However, there is a noticeable absence of a comprehensive summary of recent advancements in affective computing for healthcare, which could pose difficulties for researchers entering this field. To address this, our paper aims to provide an extensive literature review of related studies published in the last five years. We begin by analyzing trends, benefits, and limitations of recent datasets and affective computing methods devised for healthcare. Subsequently, we highlight several healthcare application hotspots of current technologies that could be promising for real-world deployment. Through our analysis, we identify and discuss some ongoing challenges in the field as evidenced by the literature. Concluding with a thorough review, we further offer potential future research directions and hope our findings and insights could guide related researchers to make better contributions to the evolution of affective computing in healthcare.

Human-Computer Interaction
Affective Computing for Large-Scale Heterogeneous Multimedia Data: A Survey

Sicheng Zhao,Shangfei Wang,Mohammad Soleymani,Dhiraj Joshi,Qiang Ji

DOI: https://doi.org/10.1145/3363560

2019-10-04

Abstract:The wide popularity of digital photography and social networks has generated a rapidly growing volume of multimedia data (i.e., image, music, and video), resulting in a great demand for managing, retrieving, and understanding these data. Affective computing (AC) of these data can help to understand human behaviors and enable wide applications. In this article, we survey the state-of-the-art AC technologies comprehensively for large-scale heterogeneous multimedia data. We begin this survey by introducing the typical emotion representation models from psychology that are widely employed in AC. We briefly describe the available datasets for evaluating AC algorithms. We then summarize and compare the representative methods on AC of different multimedia types, i.e., images, music, videos, and multimodal data, with the focus on both handcrafted features-based methods and deep learning methods. Finally, we discuss some challenges and future directions for multimedia affective computing.

Multimedia,Computer Vision and Pattern Recognition
A Comprehensive Survey on Computational Aesthetic Evaluation of Visual Art Images: Metrics and Challenges

Jiajing Zhang,Yongwei Miao,Jinhui Yu

DOI: https://doi.org/10.1109/ACCESS.2021.3083075

IF: 3.9

2021-01-01

IEEE Access

Abstract:Computational image aesthetic evaluation is a computable human aesthetic perception and judgment realized by machines, which has a significant impact on a variety of applications such as image advanced search and promotional exhibition of painting arts. Various approaches have been proposed in copious literature trying to solve this challenging problem. However, there have been few attempts in reviewing works from different types of visual arts, due to their significant differences in visual features and aesthetic principles. In this survey, we present a comprehensive listing of the reviewed works on aesthetic assessment of photographs and paintings, mainly highlighting the contributions and innovations of the existing approaches. We firstly introduce aesthetic assessment benchmark datasets in different categories. Then, conventional aesthetic evaluation approaches based on handcrafted features are reviewed. Besides, we systematically evaluate recent deep learning techniques that are useful for developing robust models for aesthetic prediction tasks in scoring, distribution, attribute, and description. Moreover, the possibility of aesthetic-aware color enhancement, recomposition of photo images, and automatic generation of aesthetic-guided art paintings through computational approaches are summarized. Finally, challenges and potential future directions for this field are discussed. We hope that our survey could serve as a comprehensive reference source for future research on computational aesthetics in visual media.
Ten Challenges for EEG-based Affective Computing

Xin Hu,Jingjing Chen,Fei Wang,Dan Zhang

DOI: https://doi.org/10.1177/2096595819896200

2019-01-01

Brain Science Advances

Abstract:The emerging field of affective computing focuses on enhancing computers’ ability to understand and appropriately respond to people’s affective states in human-computer interactions, and has revealed significant potential for a wide spectrum of applications.Recently, the electroencephalography（EEG） based affective computing has gained increasing interest for its good balance between mechanistic exploration and real-world practical application. The present work reviewed ten theoretical and operational challenges for the existing affective computing researches from an interdisciplinary perspective of information technology, psychology, and neuroscience.On the theoretical side, we suggest that researchers should be well aware of the limitations of the commonly used emotion models, and be cautious about the widely accepted assumptions on EEG-emotion relationships as well as the transferability of findings based on different research paradigms. On the practical side, we propose several operational recommendations for the challenges about data collection, feature extraction, model implementation, online system design, as well as the potential ethical issues. The present review is expected to contribute to an improved understanding of EEG-based affective computing and promote further applications.
Affective Game Computing: A Survey

Georgios N. Yannakakis,David Melhart

2023-09-25

Abstract:This paper surveys the current state of the art in affective computing principles, methods and tools as applied to games. We review this emerging field, namely affective game computing, through the lens of the four core phases of the affective loop: game affect elicitation, game affect sensing, game affect detection and game affect adaptation. In addition, we provide a taxonomy of terms, methods and approaches used across the four phases of the affective game loop and situate the field within this taxonomy. We continue with a comprehensive review of available affect data collection methods with regards to gaming interfaces, sensors, annotation protocols, and available corpora. The paper concludes with a discussion on the current limitations of affective game computing and our vision for the most promising future research directions in the field.

Human-Computer Interaction,Machine Learning,Multimedia
Advances of Research in Affective Processes in Communication and Collaboration

Jie Xu,Enid Montague,Jonathan Gratch,Peter Hancock,Myounghoon Jeon,Mark S. Pfaff

DOI: https://doi.org/10.1177/1541931215591061

2015-01-01

Proceedings of the Human Factors and Ergonomics Society Annual Meeting

Abstract:Affective processes have been an important research area for human factors and ergonomics. Although there is an obvious connection between affect and communication and collaboration, little research has been conducted in the human factors community until recently. In this panel, the panelists will discuss recent advances in affective research in communication and collaboration systems. Theoretical perspectives in human computer interaction, human agent interaction, and teamwork that take affective process into account will be discussed. Methodological issues will also be addressed, such as the measurements of affect, research design, and data analysis methods. Finally the applications of the theories and methods in different systems, such as human robot interaction, healthcare, and multi-tasking teams, will be discussed.
Affective Information Processing Introduction

Jianhua Tao,Tieniu Tan

DOI: https://doi.org/10.1007/978-1-84800-306-4_1

2009-01-01

Abstract:Affective information processing is intended to give computers the human-like capabilities of observation, interpretation and generation of affect features. It is essential for natural and effective human-computer interaction and has become a very hot research topic. This state-of-the-art volume discusses the latest developments in affective information processing, and summarises the key technologies researched, such as facial expression recognition, face animation, emotional speech synthesis, intelligent agents, and virtual reality. The detailed and accessible coverage includes a wide range of topics, including areas which look to challenge and improve current research. The book is organised into the following themed parts to aid the readers in their understanding of the field: Cognitive Emotion Model, Affect in Speech, Affect in Face and Affect in Multimodal Interaction. Topics and Features: Reports on the state of the art research in affective information processing Summarises key technologies researched during the last few years, such as affective speech processing, affective facial expressions, affective body gesture and movement, affective multi-modal systems, etc. Offers a foundational introductory chapter that grounds readers in core concepts and principles Discusses applications of this technology, such as in education against bullying Provides summarising conclusions throughout Describes affect in multimodal information Looks at cognitive emotion modelling in natural language communication Explores emotion perception and recognition from speech, and based on multimodal information Includes face animation based on large audio-visual database Presents physiological sensing for affective computing Examines evolutionary expression of emotions in virtual humans using lights and pixels Written to provide an opportunity for scientists, engineers and graduate students to explore the difficulties, solutions and technologies in this exciting new area, this ground-breaking book provides a comprehensive overview and significant insight into the field and will prove a valuable reference tool and resource.
A Survey of Affective Brain-Computer Interface

LU Bao-Liang,ZHANG Yaqian,ZHENG Wei-Long

DOI: https://doi.org/10.11959/j.issn.2096−6652.202104

2021-01-01

Abstract:An important research goal in emotion artificial intelligence is to make machines understand and recognize human emotions in real-time and facilitate human-computer interaction in a more natural and friendly way.Affective brain-computer interface (aBCI) is a type of BCI that can recognize and/or modulate human emotion.Thus, aBCI plays a critical role in promoting emotion artificial intelligence.The basic concepts and recent research development of aBCI were summarized, and the applications of aBCI in a wide range of domains were outlined.The roles that the aBCI can play in the development of artificial general intelligence and the challenges faced by the aBCI research community were discussed.
Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective

Guimin Hu,Yi Xin,Weimin Lyu,Haojian Huang,Chang Sun,Zhihong Zhu,Lin Gui,Ruichu Cai,Erik Cambria,Hasti Seifi

2024-10-30

Abstract:Multimodal affective computing (MAC) has garnered increasing attention due to its broad applications in analyzing human behaviors and intentions, especially in text-dominated multimodal affective computing field. This survey presents the recent trends of multimodal affective computing from NLP perspective through four hot tasks: multimodal sentiment analysis, multimodal emotion recognition in conversation, multimodal aspect-based sentiment analysis and multimodal multi-label emotion recognition. The goal of this survey is to explore the current landscape of multimodal affective research, identify development trends, and highlight the similarities and differences across various tasks, offering a comprehensive report on the recent progress in multimodal affective computing from an NLP perspective. This survey covers the formalization of tasks, provides an overview of relevant works, describes benchmark datasets, and details the evaluation metrics for each task. Additionally, it briefly discusses research in multimodal affective computing involving facial expressions, acoustic signals, physiological signals, and emotion causes. Additionally, we discuss the technical approaches, challenges, and future directions in multimodal affective computing. To support further research, we released a repository that compiles related works in multimodal affective computing, providing detailed resources and references for the community.

Computation and Language
A survey on mobile affective computing

Eugenia Politou,Efthimios Alepis,Constantinos Patsakis

DOI: https://doi.org/10.1016/j.cosrev.2017.07.002

IF: 8.757

2017-08-01

Computer Science Review

Abstract:The spontaneous recognition of emotional states and personality traits of individuals has been puzzling researchers for years whereas pertinent studies demonstrating the progress in the field, despite their diversity, are still encouraging. This work surveys the most well-known research studies and the state-of-the-art on affect recognition domain based on smartphone acquired data, namely smartphone embedded sensors and smartphone usage. Inevitably, supplementary modalities employed in many eminent studies are also reported here for the sake of completeness. Nevertheless, the intention of the survey is threefold; firstly to document all the to-date relevant literature on affect recognition through smartphone modalities, secondly to argue for the full potential of smartphone use in the inference of affect, and thirdly to demonstrate the current research trends towards mobile affective computing.

computer science, information systems, theory & methods, software engineering
A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges

Christian Mühl,Brendan Allison,Anton Nijholt,Guillaume Chanel

DOI: https://doi.org/10.1080/2326263x.2014.912881

2014-04-03

Abstract:Affective states, moods and emotions, are an integral part of human nature: they shape our thoughts, govern the behavior of the individual, and influence our interpersonal relationships. The last decades have seen a growing interest in the automatic detection of such states from voice, facial expression, and physiological signals, primarily with the goal of enhancing human-computer interaction with an affective component. With the advent of brain-computer interface research, the idea of affective brain-computer interfaces (aBCI), enabling affect detection from brain signals, arose. In this article, we set out to survey the field of neurophysiology-based affect detection. We outline possible applications of aBCI in a general taxonomy of brain-computer interface approaches and introduce the core concepts of affect and their neurophysiological fundamentals. We show that there is a growing body of literature that evidences the capabilities, but also the limitations and challenges of affect detection from neurophysiological activity.

English Else
Ubiquitous Affective Computing: A Review

Rawin Assabumrungrat,Soravitt Sangnark,Thananya Charoenpattarawut,Wipamas Polpakdee,Thapanun Sudhawiyangkul,Ekkarat Boonchieng,Theerawit Wilaiprasitporn

DOI: https://doi.org/10.1109/jsen.2021.3138269

IF: 4.3

2022-02-01

IEEE Sensors Journal

Abstract:This review investigated research works on affective computing by using electrocardiogram (ECG) and electrodermal activity (EDA). The 27 related research papers, including 23 from IEEE Journals and 4 from other Q1 Journals in the last five years, were studied. The main goals have been to summarize common trends in this field in recent years as well as provide discussions and a big picture of how future research should be done. The content of this review covers the fundamental pipeline of affective computing research consisting of stimuli, intelligent affective computing, and sensor design and data processing. Additionally, we discuss future research directions and potential applications, including stress monitoring, music cognition, and robotics, where ECG and EDA will play a significant role with wearable devices. Therefore, this review serves as an information summary for everyone who is interested in affective computing and improving AI’s understanding of human emotion.

engineering, electrical & electronic,instruments & instrumentation,physics, applied
Multimodal Information Processing For Affective Computing

Jianhua Tao

DOI: https://doi.org/10.1007/978-0-387-73819-2_9

2010-01-01

Abstract: Affective computing is interaction that relates to, arises from or deliberately influences emotions [1]; it tries to assign computers the human-like capabilities of observation, interpretation and generation of affect features. It is an important topic in human–computer interaction (HCI), because it helps increase the quality of human to computer communications.
Intelligent Signal Processing for Affective Computing [from the Guest Editors]

Bjoern W. Schuller,Rosalind Picard,Elisabeth Andre,Jonathan Gratch,Jianhua Tao

DOI: https://doi.org/10.1109/msp.2021.3096415

IF: 15.204

2021-01-01

IEEE Signal Processing Magazine

Abstract:Affective computing is computing that relates to, arises from, or deliberately influences emotion or other affective phenomena. Human emotion and affect in general are fundamental to human experience, influencing cognition, perception, and everyday tasks such as learning and communication, but are also fundamental to human health and well-being. The field of affective computing has matured over its roughly two-and-a-half decades coming closer than ever to the point of usage at scale. Affective computing is facing a plethora of different signal types-audio, video, and physiological signals, to name but the most dominant.
Affective computing scholarship and the rise of China: a view from 25 years of bibliometric data

Manh-Tung Ho,Peter Mantello,Hong-Kong T. Nguyen,Quan-Hoang Vuong

DOI: https://doi.org/10.1057/s41599-021-00959-8

2021-11-18

Humanities and Social Sciences Communications

Abstract:Abstract Affective computing, also known as emotional artificial intelligence (AI), is an emerging and cutting-edge field of AI research. It draws on computer science, engineering, psychology, physiology, and neuroscience to computationally model, track, and classify human emotions and affective states. While the US once dominated the field in terms of research and citation from 1995–2015, China is now emerging as a global contender in research output, claiming second place for the most cited country from 2016–2020. This article maps the rhizomatic growth and development of scientific publications devoted to emotion-sensing AI technologies. It employs a bibliometric analysis that identifies major national contributors and international alliances in the field over the past 25 years. Contrary to the ongoing political rhetoric of a new Cold War, we argue that there are in fact vibrant AI research alliances and ongoing collaborations between the West and China, especially with the US, despite competing interests and ethical concerns. Our observations of historical data indicate two major collaborative networks: the “US/Asia-Pacific cluster” consisting of the US, China, Singapore, Japan and the “European” cluster of Germany, the UK, and the Netherlands. Our analysis also uncovers a major shift in the focus of affective computing research away from diagnosis and detection of mental illnesses to more commercially viable applications in smart city design. The discussion notes the state-of-the-art techniques such as the ensemble method of symbolic and sub-symbolic AI as well as the absence of Russia in the list of top countries for scientific output.
A survey of speech emotion recognition in human computer interaction

ZHANG Shi-qing,LI Le-min,ZHAO Zhi-jin

2013-01-01

Abstract:Speech emotion recognition is a current active research topic in the fields of signal processing,pattern recognition,artificial intelligence,human computer interaction,etc.The ultimate purpose of such research is to endow computers with emotion ability and make human computer interaction be genuinely harmonic and natural.This paper reviews the recent advance of several key problems involved in speech emotion recognition,including emotional description theory,emotional speech databases,emotional acoustic analysis as well as emotion recognition methods.In addition,the existing research problems and the future direction are presented.

Affect is Computable——A Survey on Affective Computing

A Comprehensive Survey on Affective Computing; Challenges, Trends, Applications, and Future Directions

A systematic review on affective computing: emotion models, databases, and recent advances

A Survey on Personalized Affective Computing in Human-Machine Interaction

Investigation on Experiment Design and Affect Measure in Affective Computing

Affective Computing for Healthcare: Recent Trends, Applications, Challenges, and Beyond

Affective Computing for Large-Scale Heterogeneous Multimedia Data: A Survey

A Comprehensive Survey on Computational Aesthetic Evaluation of Visual Art Images: Metrics and Challenges

Ten Challenges for EEG-based Affective Computing

Affective Game Computing: A Survey

Advances of Research in Affective Processes in Communication and Collaboration

Affective Information Processing Introduction

A Survey of Affective Brain-Computer Interface

Recent Trends of Multimodal Affective Computing: A Survey from NLP Perspective

A survey on mobile affective computing

A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges

Ubiquitous Affective Computing: A Review

Multimodal Information Processing For Affective Computing

Intelligent Signal Processing for Affective Computing [from the Guest Editors]

Affective computing scholarship and the rise of China: a view from 25 years of bibliometric data

A survey of speech emotion recognition in human computer interaction