Research Progress of Emotion Processing Based on Speech Signal

HAN Ji-qing,SHAO Yan-qiu

DOI: https://doi.org/10.3969/j.issn.1002-8684.2006.05.014

2006-01-01

Abstract:The research progress of emotion processing based on speech signal is firstly introduced in this paper. Then the related methods, some techniques and applications in the field are reviewed. Finally, the unsolved problems and the prospect are pointed out.

What problem does this paper attempt to address?

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Ziping Zhao,Yu Zheng,Zixing Zhang,Haishuai Wang,Yiqin Zhao,Chao Li

DOI: https://doi.org/10.21437/interspeech.2018-1477

2018-01-01

Abstract:Automatic emotion recognition from speech, which is an important and challenging task in the field of affective computing, heavily relies on the effectiveness of the speech features for classification. Previous approaches to emotion recognition have mostly focused on the extraction of carefully hand-crafted features. How to model spatio-temporal dynamics for speech emotion recognition effectively is still under active investigation. In this paper, we propose a method to tackle the problem of emotional relevant feature extraction from speech by leveraging Attention-based Bidirectional Long Short-Term Memory Recurrent Neural Networks with fully convolutional networks in order to automatically learn the best spatio-temporal representations of speech signals. The learned high-level features are then fed into a deep neural network (DNN) to predict the final emotion. The experimental results on the Chinese Natural Audio-Visual Emotion Database (CHEAVD) and the Interactive Emotional Dyadic Motion Capture (IEMOCAP) corpora show that our method provides more accurate predictions compared with other existing emotion recognition algorithms.
Deep Spectrum Feature Representations for Speech Emotion Recognition

Ziping Zhao,Yiqin Zhao,Zhongtian Bao,Haishuai Wang,Zixing Zhang,Chao Li

DOI: https://doi.org/10.1145/3267935.3267948

2018-01-01

Abstract:Automatically detecting emotional state in human speech, which plays an effective role in areas of human machine interactions, has been a difficult task for machine learning algorithms. Previous work for emotion recognition have mostly focused on the extraction of carefully hand-crafted and tailored features. Recently, spectrogram representations of emotion speech have achieved competitive performance for automatic speech emotion recognition. In this work we propose a method to tackle the problem of deep features, herein denoted as deep spectrum features, extraction from the spectrogram by leveraging Attention-based Bidirectional Long Short-Term Memory Recurrent Neural Networks with fully convolutional networks. The learned deep spectrum features are then fed into a deep neural network (DNN) to predict the final emotion. The proposed model is then evaluated on the Interactive Emotional Dyadic Motion Capture (IEMOCAP) dataset to validate its effectiveness. Promising results indicate that our deep spectrum representations extracted from the proposed model perform the best, 65.2% for weighted accuracy and 68.0% for unweighted accuracy when compared to other existing methods. We then compare the performance of our deep spectrum features with two standard acoustic feature representations for speech-based emotion recognition. When combined with a support vector classifier, the performance of the deep feature representations extracted are comparable with the conventional features. Moreover, we also investigate the impact of different frequency resolutions of the input spectrogram on the performance of the system.
Emotion recognition from sound stimuli based on back-propagation neural networks and electroencephalograms

guoqing di,sixia wu

DOI: https://doi.org/10.1121/1.4927693

2015-01-01

Abstract:This research aims to explore the feasibility of using back-propagation (BP) neural networks and electroencephalograms (EEGs) to recognize the emotional reactions induced by sound stimuli in the dimensions of pleasure and arousal, as well as compare the recognition performance of each method on these two dimensions. It could provide an aided design on choosing proper sounds to induce or regulate individuals' emotional states under specific situations for potential users at the design stage. Emotional reactions to different sound stimuli are investigated by Self-Assessment Manikin. The results of BP neural network indicate that the arousal predictions are more satisfactory than the pleasure predictions, and the recognition rates can be improved by optimizing input parameters. EEG signals induced by sound stimuli are recorded. The results show that when induced by each pleasant sound, the Average Power of Electroencephalogram of the a wave in the left frontal pole electrode is significantly lower than that in the right frontal pole electrode, while when induced by each unpleasant sound, the former is significantly higher than the latter. This finding indicates that pleasant and unpleasant sounds can be identified based on the asymmetry of the a wave between the left and right frontal pole electrodes. (C) 2015 Acoustical Society of America.
Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions

Jiaqi Sun

DOI: https://doi.org/10.54097/agvjvq19

2024-03-13

Abstract:In this study, the focus is primarily on the diverse methods for recognizing human emotions through language, text, and facial expressions via computational technology. Emphasizing the real-world applicability of these techniques, the paper underscores the significance of multimodal emotion recognition in areas such as human-computer interaction, psychology, and emotion analytics. Multimodal methods, which combine data from various sources like voice tone, facial cues, and textual context, offer a robust approach for discerning nuanced emotional states. Compared to single-mode analysis, these multimodal techniques tend to produce more accurate and comprehensive results, bridging the gaps left by any one mode in isolation. As technology increasingly integrates with daily human activity, the importance of nuanced, reliable emotion recognition is becoming paramount for fostering more natural and empathic machine-human interactions. Moreover, in the realm of psychology, these methods offer groundbreaking possibilities for diagnosis and treatment. By discussing the future applications and methodologies of multimodal emotion recognition, this paper aims to provide a comprehensive roadmap for both academic research and practical applications in the evolving landscape of emotion-aware computing.
Artificial Emotion and Its Recognition, Modeling and Applications: an Overview

Li Xu,Zhenrong Zhuge,Jie Yang

DOI: https://doi.org/10.1109/wcica.2004.1342020

2004-01-01

Abstract:Artificial emotion is a new and attractive area to the intelligent control community. In this paper, the major medical psychological view-points on emotion, i.e., the characteristics of emotion and the relationship among emotion, cognition and intelligence, are introduced. The signals based on which emotion recognition is carried out, as well as the main approaches to the recognition and the modeling of emotion are overviewed. Applications of artificial emotion to those areas such as intelligent robotics, human-machine interaction, and intelligent transportation are also introduced, and the future works are further discussed.
Binding of mammalian ribosomes to ms2 phage rna reveals an overlapping gene encoding a lysis function

J. F. Atkins,J. Steitz,C. Anderson,P. Model

DOI: https://doi.org/10.1016/0092-8674(79)90044-8

IF: 64.5

1979-10-01

Cell

Abstract:
The Research Progress of Speech Information Processing

CAI Lianhong,JIA Jia,ZHENG Fang

DOI: https://doi.org/10.3969/j.issn.1003-0077.2011.06.017

2011-01-01

Abstract:This paper introduces the progress of speech information processing,especially the researches on Chinese speech processing.Speech information processing includes speech recognition,speaker recognition,speech synthesis and computational speech perception.Researches on speech recognition with accent and personal style support the systems of language learning and evaluation,while speaker recognition focuses on how to improve the performance in different conditions.Researches on speech synthesis pay more attention on cross-language,emotional and audio-visual speech synthesis.Fomputational speech perception focuses on the implementation on speech testing and rehabilitation,denoising,and speech enhancement.Through these researches,especially the combination of speech information processing,linguistics and web technology,we can build more harmonious human-computer speech interaction system.
Speech Emotion Recognition Based on Formant Characteristics Feature Extraction and Phoneme Type Convergence.

Zhen-Tao Liu,Abdul Rehman,Min Wu,Wei-Hua Cao,Man Hao

DOI: https://doi.org/10.1016/j.ins.2021.02.016

IF: 8.1

2021-01-01

Information Sciences

Abstract:Speech Emotion Recognition (SER) has numerous applications including human-robot interaction, online gaming, and health care assistance. While deep learning-based approaches achieve considerable precision, they often come with high computational and time costs. Indeed, feature learning strategies must search for important features in a large amount of speech data. In order to reduce these time and computational costs, we propose pre-processing step in which speech segments with similar formant characteristics are clustered together and labeled as the same phoneme. The phoneme occurrence rates in emotional utterances are then used as the input features for classifiers. Using six databases (EmoDB, RAVDESS, IEMOCAP, ShEMO, DEMoS and MSP-Improv) for evaluation, the level of accuracy is comparable to that of current state-of-the-art methods and the required training time was significantly reduced from hours to minutes. (c) 2021 Elsevier Inc. All rights reserved.
Research on Speech Emotion Recognition Technology based on Deep and Shallow Neural Network

Jian Wang,Zhiyan Han

DOI: https://doi.org/10.23919/chicc.2019.8866568

2019-07-01

Abstract:Speech emotion recognition has become a hot topic in the field of human-computer interaction. In order to improve the accuracy of emotion recognition, this paper proposes a new speech emotion recognition technology based on the combination of deep and shallow neural networks. First, the speech signal is preprocessed, then the parallel training sample set is established, and the Deep Belief Network (DBN) is used to automatically extract and recognize the speech emotion features. Finally, the shallow neural network is used to obtain the final recognition results. In order to evaluate the quality of the new method, we compared three systems to identify five emotions, and the experimental results show that the proposed method can effectively improve the accuracy of emotion recognition.
Speech Emotion Recognition: A Comprehensive Survey

Ebrahimi-Moghadam, Abbas

DOI: https://doi.org/10.1007/s11277-023-10244-3

IF: 2.017

2023-03-09

Wireless Personal Communications

Abstract:Speech emotion recognition could be considered a new topic in speech processing where he plays that plays an essential role in human interaction. Emotions are a king of speech that recognizes the three significant aspects of designing the speech emotion recognition system. This article reviews the work on speech emotion recognition and is helpful for further research. Firstly, speech emotion recognition databases are described for evaluating system performance. Secondly, the choice of feature is presented in the speech representation. And third is the design of a suitable class. While the section fourth explains the multiple classifier system and its impact on system. In the fifth part of the article, we review the most important challenges in the system speech emotion recognition. The final results obtained from the system function and its constraints are discussed, and we provide directions to improve speech emotion recognition systems.

telecommunications
Speech emotion recognition methods: A literature review

Babak Basharirad,Mohammadreza Moradhaseli

DOI: https://doi.org/10.1063/1.5005438

2017-01-01

AIP Conference Proceedings

Abstract:Recently, attention of the emotional speech signals research has been boosted in human machine interfaces due to availability of high computation capability. There are many systems proposed in the literature to identify the emotional state through speech. Selection of suitable feature sets, design of a proper classifications methods and prepare an appropriate dataset are the main key issues of speech emotion recognition systems. This paper critically analyzed the current available approaches of speech emotion recognition methods based on the three evaluating parameters (feature set, classification of features, accurately usage). In addition, this paper also evaluates the performance and limitations of available methods. Furthermore, it highlights the current promising direction for improvement of speech emotion recognition systems.
Research Progress of EEG-Based Emotion Recognition: A Survey

Yiming Wang,Bin Zhang,Lamei Di

DOI: https://doi.org/10.1145/3666002

IF: 16.6

2024-07-10

ACM Computing Surveys

Abstract:Emotion recognition based on electroencephalography (EEG) signals has emerged as a prominent research field, facilitating objective evaluation of diseases like depression and motion detection for heathy people. Starting from the basic concepts of temporal-frequency-spatial features in EEG and the methods for cross-domain feature fusion, this survey then extends the overfitting challenge of EEG single-modal to the problem of heterogeneous modality modeling in multimodal conditions. It explores issues such as feature selection, sample scarcity, cross-subject emotional transfer, physiological knowledge discovery, multimodal fusion methods, and modality missing. These findings provide clues for researchers to further investigate emotion recognition based on EEG signals.

computer science, theory & methods
A Comparison of Methods of Facial Expression Recognition

Yingjie Chen,Han Wu,Tao Wang,Yizhou Wang

DOI: https://doi.org/10.1109/wrc-sara.2018.8584202

2018-01-01

Abstract:Emotional recognition based on facial expressions is a very active research topic in the field of human-computer interaction. There has been a great body of work with in-depth of study in this area. In this paper, we analyze and compare the state-of-the-art facial expression recognition methods, propose some evaluation dimensions and discuss possible directions for future research.
Research Progresses in Music Emotion Recognition

Xiao'ou CHEN,Deshun YANG

DOI: https://doi.org/10.15943/j.cnki.fdxb-jns.2017.02.001

2017-01-01

Abstract:This paper first gives an introduction to the definition and representation of music emotion,the computational model and the datasets proposed for the research on music emotion recognition.Then,the research work and the progresses on audio music emotion recognition are reviewed.Finally,some crucial issues are raised and the possible solutions for them are proposed.
Recognition of Infant's Emotions and Needs from Speech Signals.

Xuan Zhou,Jian Wang,Hongzhi Hu,Weihui Dai,Lina Wei,Huajuan Mao

DOI: https://doi.org/10.1109/smc.2016.7844960

2016-01-01

Abstract:Speech is not only a way for infants under one year of age to communicate with the outside world, but also the important information source to reflect their emotions and needs, as well as health status and mental level. In order to explore the intelligent machine technology for understanding infant's emotions and needs from speech signals, and therefore help parents in child rearing, this paper studied the signal processing and feature extraction of the above speeches. It seems that a high accuracy and reliable result couldn't be reached from the infant's speech signals only in dealing with multi kinds of emotions and needs. So an effective recognition approach considering the combined features of acoustic characteristics and rearing behaviors was proposed based on the self-adapting algorithm. Experiment results showed that most common emotions and needs of infants, such as happy, hungry, and sleepy states which reflect their typically physiological and psychological status in daily life, can be recognized correctly at a relatively high accuracy.
A Review of Key Technologies for Emotion Analysis Using Multimodal Information

Xianxun Zhu,Chaopeng Guo,Heyang Feng,Yao Huang,Yichen Feng,Xiangyang Wang,Rui Wang

DOI: https://doi.org/10.1007/s12559-024-10287-z

IF: 4.89

2024-06-02

Cognitive Computation

Abstract:Emotion analysis, an integral aspect of human–machine interactions, has witnessed significant advancements in recent years. With the rise of multimodal data sources such as speech, text, and images, there is a profound need for a comprehensive review of pivotal elements within this domain. Our paper delves deep into the realm of emotion analysis, examining multimodal data sources encompassing speech, text, images, and physiological signals. We provide a curated overview of relevant literature, academic forums, and competitions. Emphasis is laid on dissecting unimodal processing methods, including preprocessing, feature extraction, and tools across speech, text, images, and physiological signals. We further discuss the nuances of multimodal data fusion techniques, spotlighting early, late, model, and hybrid fusion strategies. Key findings indicate the essentiality of analyzing emotions across multiple modalities. Detailed discussions on emotion elicitation, expression, and representation models are presented. Moreover, we uncover challenges such as dataset creation, modality synchronization, model efficiency, limited data scenarios, cross-domain applicability, and the handling of missing modalities. Practical solutions and suggestions are provided to address these challenges. The realm of multimodal emotion analysis is vast, with numerous applications ranging from driver sentiment detection to medical evaluations. Our comprehensive review serves as a valuable resource for both scholars and industry professionals. It not only sheds light on the current state of research but also highlights potential directions for future innovations. The insights garnered from this paper are expected to pave the way for subsequent advancements in deep multimodal emotion analysis tailored for real-world deployments.

computer science, artificial intelligence,neurosciences
Research on Chinese Speech Emotion Recognition Based on Deep Neural Network and Acoustic Features

Ming-Che Lee,Sheng-Cheng Yeh,Jia-Wei Chang,Zhen-Yi Chen

DOI: https://doi.org/10.3390/s22134744

2022-06-23

Abstract:In recent years, the use of Artificial Intelligence for emotion recognition has attracted much attention. The industrial applicability of emotion recognition is quite comprehensive and has good development potential. This research uses voice emotion recognition technology to apply it to Chinese speech emotion recognition. The main purpose of this research is to transform gradually popularized smart home voice assistants or AI system service robots from a touch-sensitive interface to a voice operation. This research proposed a specifically designed Deep Neural Network (DNN) model to develop a Chinese speech emotion recognition system. In this research, 29 acoustic characteristics in acoustic theory are used as the training attributes of the proposed model. This research also proposes a variety of audio adjustment methods to amplify datasets and enhance training accuracy, including waveform adjustment, pitch adjustment, and pre-emphasize. This study achieved an average emotion recognition accuracy of 88.9% in the CASIA Chinese sentiment corpus. The results show that the deep learning model and audio adjustment method proposed in this study can effectively identify the emotions of Chinese short sentences and can be applied to Chinese voice assistants or integrated with other dialogue applications.
EEG based Emotion Recognition: A Tutorial and Review

Xiang Li,Yazhou Zhang,Prayag Tiwari,Dawei Song,Bin Hu,Meihong Yang,Zhigang Zhao,Neeraj Kumar,Pekka Marttinen

DOI: https://doi.org/10.1145/3524499

IF: 16.6

2022-03-25

ACM Computing Surveys

Abstract:Emotion recognition technology through analyzing the EEG signal is currently an essential concept in Artificial Intelligence and holds great potential in emotional health care, human-computer interaction, multimedia content recommendation, etc. Though there have been several works devoted to reviewing EEG-based emotion recognition, the content of these reviews needs to be updated. In addition, those works are either fragmented in content or only focus on specific techniques adopted in this area but neglect the holistic perspective of the entire technical routes. Hence, in this paper, we review from the perspective of researchers who try to take the first step on this topic. We review the recent representative works in the EEG-based emotion recognition research and provide a tutorial to guide the researchers to start from the beginning. The scientific basis of EEG-based emotion recognition in the psychological and physiological levels is introduced. Further, we categorize these reviewed works into different technical routes and illustrate the theoretical basis and the research motivation, which will help the readers better understand why those techniques are studied and employed. At last, existing challenges and future investigations are also discussed in this paper, which guides the researchers to decide potential future research directions.

computer science, theory & methods
Research on the Application of Speech Database based on Emotional Feature Extraction in International Chinese Education and Teaching

Xiangli Zhang

DOI: https://doi.org/10.12694/scpe.v25i1.2296

2024-01-04

Abstract:The advanced analysis of the relationship between acoustic and emotional characteristics of speech signals can effectively improve the interactivity and intelligence of computers. Given the current status of speech recognition and the problems encountered in international Chinese education, the study proposes to extract emotional characteristics to achieve speech construction of the database. Based on considering the emotional characteristics of speech, a hybrid algorithm based on spectral sequence context features is proposed. The DBN-BP algorithm is used to process emotional data of different dimensions, and a speech database is constructed. After testing and analyzing the algorithm model, it is found that the dynamic recognition accuracy of the DBN-BP model fused with emotional features is over 90%, and the negative emotion recognition rates in the three databases are all above 60%. At the same time, the accuracy rate of the model in the algorithm comparison experiment remains above 85%, the data information extraction is relatively complete, and the average test time of less than 1s is less than 3%. The speech database based on multi-emotional feature extraction can effectively provide a new reference for the improvement of the quality of Chinese international education and the improvement of the speech recognition system.
A review of EEG features for emotion recognition

Guanhua ZHANG,Minjing YU,Guo CHEN,Yiheng HAN,Dan ZHANG,Guozhen ZHAO,Yong-Jin LIU

DOI: https://doi.org/10.1360/n112018-00337

2019-01-01

Abstract:Emotion recognition is an important research topic in the human-machine interaction field, and it can be applied to medicine, education, psychology, military, and other areas. Electroencephalogram (EEG) signals are mostly used among various indices of emotion recognition. High accuracy of emotion classifiers can be achieved by extracting the most relevant and discriminant features of emotion states. This study surveys EEG features that are extensively used in current emotion recognition studies by introducing EEG features from the following four viewpoints: time domain, frequency domain, time–frequency domain, and space domain. An SLDA algorithm is imported to three public EEG-emotion datasets (SEED, DREAMER, and CAS-THU) to evaluate feature capabilities that distinguish emotion valence. Existing problems and future investigations are also discussed in this paper.

Research Progress of Emotion Processing Based on Speech Signal

Exploring Spatio-Temporal Representations by Integrating Attention-based Bidirectional-LSTM-RNNs and FCNs for Speech Emotion Recognition

Deep Spectrum Feature Representations for Speech Emotion Recognition

Emotion recognition from sound stimuli based on back-propagation neural networks and electroencephalograms

Research And Application Analysis of Multimodal Emotion Recognition Methods Based on Speech, Text, And Facial Expressions

Artificial Emotion and Its Recognition, Modeling and Applications: an Overview

Binding of mammalian ribosomes to ms2 phage rna reveals an overlapping gene encoding a lysis function

The Research Progress of Speech Information Processing

Speech Emotion Recognition Based on Formant Characteristics Feature Extraction and Phoneme Type Convergence.

Research on Speech Emotion Recognition Technology based on Deep and Shallow Neural Network

Speech Emotion Recognition: A Comprehensive Survey

Speech emotion recognition methods: A literature review

Research Progress of EEG-Based Emotion Recognition: A Survey

A Comparison of Methods of Facial Expression Recognition

Research Progresses in Music Emotion Recognition

Recognition of Infant's Emotions and Needs from Speech Signals.

A Review of Key Technologies for Emotion Analysis Using Multimodal Information

Research on Chinese Speech Emotion Recognition Based on Deep Neural Network and Acoustic Features

EEG based Emotion Recognition: A Tutorial and Review

Research on the Application of Speech Database based on Emotional Feature Extraction in International Chinese Education and Teaching

A review of EEG features for emotion recognition