Abstract:Video-based facial expression recognition has received substantial attention over the past decade, while early expression detection (EED) is still a relatively new and challenging problem. The goal of EED is to identify an expression as quickly as possible after the expression starts and before it ends. This timely ability has many potential applications, ranging from human-computer interaction to security. The max-margin early event detector (MMED) is a well-known ranking model for early event detection. It can achieve competitive EED performance but suffers from several critical limitations: 1) MMED lacks flexibility in extracting useful information for segment comparison, which leads to poor performance in exploring the ranking relation between segment pairs; 2) the training process is slow due to the large number of constraints, and the memory requirement is also usually hard to satisfy; and 3) MMED is linear in nature, and hence may not be appropriate for data in a nonlinear feature space. To overcome these limitations, we propose an online multi-instance learning (MIL) framework for EED. In particular, the MIL technique is first introduced to generalize MMED, resulting in the proposed MIL-based EED (MIED), which is more general and flexible than MMED, since various instance construction and combination strategies can be adopted. To accelerate the training process, we reformulate MIED in the online setting and develop online multi-instance learning framework for EED (OMIED). To further exploit the nonlinear structure of the data distribution, we incorporate the kernel methods in OMIED, which results in the proposed online kernel multi-instance learning for early expression detection. Experiments on two popular and one challenging video-based expression data sets demonstrate both the efficiency and effectiveness of the proposed methods.

Early Facial Expression Recognition Using Hidden Markov Models

Dynamic hand gesture recognition using hidden Markov models

Real-Time facial expression recognition system based on HMM and feature point localization

Multi-Instance Hidden Markov Model For Facial Expression Recognition

Research on Dynamic Facial Expressions Recognition

Non-frontal view facial expression recognition based on ergodic hidden Markov model supervectors

Online Kernel-Based Structured Output SVM for Early Expression Detection

Combining 2D Gabor and Local Binary Pattern for Facial Expression Recognition Using Extreme Learning Machine

Expression recognition from video using a coupled hidden Markov model

Early Expression Detection Via Online Multi-Instance Learning with Nonlinear Extension

Video-Based Facial Expression Recognition Using Histogram Sequence of Local Gabor Binary Patterns from Three Orthogonal Planes

Efficient Facial Expression Recognition with Representation Reinforcement Network and Transfer Self-Training for Human–Machine Interaction

Automatic facial expression recognition on a single 3D face by exploring shape deformation.

Subtle expression recognition-a comprehensive approach

Spontaneous Facial Expression Recognition Based on Feature Point Tracking

A Navel Facial Expression Recognition Method Based On Extreme Learning Machine

A method for recognizing facial expression intensity based on facial muscle variations

Expression Recognition Using 2D Embedded HMM.

Simultaneous Facial Feature Tracking and Facial Expression Recognition.

Non-intrusive gesture recognition system combining with face detection based on Hidden Markov Model

The Human Facial Combined Expression Recognition System