Abstract:Multimodal emotion recognition is a challenging task in emotion computing as it is quite difficult to extract discriminative features to identify the subtle differences in human emotions with abstract concept and multiple expressions. Moreover, how to fully utilize both audio and visual information is still an open problem. In this paper, we propose a novel multimodal fusion attention network for audio-visual emotion recognition based on adaptive and multi-level factorized bilinear pooling (FBP). First, for the audio stream, a fully convolutional network (FCN) equipped with 1-D attention mechanism and local response normalization is designed for speech emotion recognition. Next, a global FBP (G-FBP) approach is presented to perform audio-visual information fusion by integrating selfattention based video stream with the proposed audio stream. To improve G-FBP, an adaptive strategy (AG-FBP) to dynamically calculate the fusion weight of two modalities is devised based on the emotion-related representation vectors from the attention mechanism of respective modalities. Finally, to fully utilize the local emotion information, adaptive and multi-level FBP (AMFBP) is introduced by combining both global-trunk and intratrunk data in one recording on top of AG-FBP. Tested on the IEMOCAP corpus for speech emotion recognition with only audio stream, the new FCN method outperforms the state-ofthe-art results with an accuracy of 71.40%. Moreover, validated on the AFEW database of EmotiW2019 sub-challenge and the IEMOCAP corpus for audio-visual emotion recognition, the proposed AM-FBP approach achieves the best accuracy of 63.09% and 75.49% respectively on the test set.

Content-Based Information Fusion for Semi-Supervised Music Genre Classification

Automatic Music Emotion Classification Using a New Classification Algorithm

Semi-Supervised Classification of Musical Genre Using Multi-View Features.

Image–Music Synesthesia-Aware Learning Based on Emotional Similarity Recognition

Improving Music Genre Classification from Multi-Modal Properties of Music and Genre Correlations Perspective

Music Genres Classification Using Text Categorization Method

Multimodal Music Mood Classification by Fusion of Audio and Lyrics.

Fisher Discriminative Embedding Low-Rank Sparse Representation for Music Genre Classification

Music Classification Via the Bag-of-features Approach.

Boosting for Multi-Modal Music Emotion Classification.

Graph-Based Multimodal Music Mood Classification in Discriminative Latent Space.

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

Music Era Recognition Using Supervised Contrastive Learning and Artist Information

Long Short-Term Memory Recurrent Neural Network Based Segment Features for Music Genre Classification

Convolution channel separation and frequency sub-bands aggregation for music genre classification

A sequential naive Bayes method for music genre classification based on transitional information from pitch and beat

On Feature Combination For Music Classification

Automatic Music Mood Classification by Learning Cross-Media Relevance Between Audio and Lyrics

Course genres classification of music e-learning platform based on deep learning big data intelligent processing algorithm

A Middle-Level Learning Feature Interaction Method with Deep Learning for Multi-Feature Music Genre Classification

Research on Music Emotional Expression Based on Reinforcement Learning and Multimodal Information