Emotion recognition based on brain-like multimodal hierarchical perception
Xianxun Zhu,Yao Huang,Xiangyang Wang,Rui Wang
DOI: https://doi.org/10.1007/s11042-023-17347-w
IF: 2.577
2023-12-07
Multimedia Tools and Applications
Abstract:Emotion recognition has gained prominence in diverse applications ranging from safe driving and e-commerce to healthcare. Traditional approaches have often relied on single-modal information such as visual, audio, or text, resulting in limitations in both reliability and robustness. To address these shortcomings, we introduce a brain-inspired computing model for emotion recognition that mimics the hierarchical processing characteristics of human cognitive functions. This innovative model accommodates multimodal information cohesively, aiming to emulate the human cognitive process across visual, audio, and text. To gain a better grasp of our brain-like hierarchical perception architecture, we stratify the model into three key layers: feature extraction, fusion, and decision-making. This structure integrates cognitive mechanisms with machine learning algorithms for enhanced performance. Specifically, we begin by extracting deep features that emulate the human brain's perception of emotional cues. These features are then synthesized using a cross-attention mechanism to explore inter-modal correlations. Finally, the aggregated emotional data is categorized and recognized. Experimental results indicate that our approach achieves an average recognition accuracy of across four distinct emotion classifications, showcasing its effectiveness and offering a fresh perspective for multimodal emotion recognition.
computer science, information systems, theory & methods,engineering, electrical & electronic, software engineering