Abstract:Comprehending different categories of facial expressions plays a great role in the design of computational model analyzing human perceived and affective state. Authoritative studies have revealed that facial expressions in human daily life are in multiple or co-occurring mental states. However, due to the lack of valid datasets, most previous studies are still restricted to basic emotions with single label. In this paper, we present a novel multi-label facial expression database, RAF-ML, along with a new deep learning algorithm, to address this problem. Specifically, a crowdsourcing annotation of 1.2 million labels from 315 participants was implemented to identify the multi-label expressions collected from social network, then EM algorithm was designed to filter out unreliable labels. For all we know, RAF-ML is the first database in the wild that provides with crowdsourced cognition for multi-label expressions. Focusing on the ambiguity and continuity of blended expressions, we propose a new deep manifold learning network, called Deep Bi-Manifold CNN, to learn the discriminative feature for multi-label expressions by jointly preserving the local affinity of deep features and the manifold structures of emotion labels. Furthermore, a deep domain adaption method is leveraged to extend the deep manifold features learned from RAF-ML to other expression databases under various imaging conditions and cultures. Extensive experiments on the RAF-ML and other diverse databases (JAFFE, CK+$$+$$, SFEW and MMI) show that the deep manifold feature is not only superior in multi-label expression recognition in the wild, but also captures the elemental and generic components that are effective for a wide range of expression recognition tasks.

Real-World Facial Expression Recognition Using Metric Learning Method.

Recognizing Compound Emotional Expression in Real-World Using Metric Learning Method.

A New Deep Learning Method for Multi-label Facial Expression Recognition Based on Local Constraint Features

Efficient Facial Expression Recognition with Representation Reinforcement Network and Transfer Self-Training for Human–Machine Interaction

Real-Time Facial Expression Recognition Based on the Improved Brain Emotional Learning Model

Real-Time facial expression recognition system based on HMM and feature point localization

Face Recognition Based on LDA and Improved Pairwise-Constrained Multiple Metric Learning Method.

Blended Emotion in-the-Wild: Multi-label Facial Expression Recognition Using Crowdsourced Annotations and Deep Locality Feature Learning

DeepEmo: Real-world Facial Expression Analysis Via Deep Learning

Robust Facial Expression Recognition Algorithm Based on Local Metric Learning.

Facial Expression Recognition by Jointly Partial Image and Deep Metric Learning.

Adaptive discriminative metric learning for facial expression recognition

A Scalable Metric Learning-Based Voting Method for Expression Recognition

Spontaneous facial expression recognition: A robust metric learning approach

An Effective and Efficient Real-Time Facial Expression Recognition Method in the Wild

Expression Analysis Based on Face Regions in Real-world Conditions.

Adaptive Metric Learning with Deep Neural Networks for Video-Based Facial Expression Recognition

Identity-aware Facial Expression Recognition via Deep Metric Learning based on Synthesized Images

Facial expression recognition based on deep convolution long short-term memory networks of double-channel weighted mixture

Expression Analysis Based on Face Regions in Read-world Conditions

Real-Time Facial Expression Recognition Based on Attention Mechanism