Abstract:Video-based facial expression recognition has received substantial attention over the past decade, while early expression detection (EED) is still a relatively new and challenging problem. The goal of EED is to identify an expression as quickly as possible after the expression starts and before it ends. This timely ability has many potential applications, ranging from human-computer interaction to security. The max-margin early event detector (MMED) is a well-known ranking model for early event detection. It can achieve competitive EED performance but suffers from several critical limitations: 1) MMED lacks flexibility in extracting useful information for segment comparison, which leads to poor performance in exploring the ranking relation between segment pairs; 2) the training process is slow due to the large number of constraints, and the memory requirement is also usually hard to satisfy; and 3) MMED is linear in nature, and hence may not be appropriate for data in a nonlinear feature space. To overcome these limitations, we propose an online multi-instance learning (MIL) framework for EED. In particular, the MIL technique is first introduced to generalize MMED, resulting in the proposed MIL-based EED (MIED), which is more general and flexible than MMED, since various instance construction and combination strategies can be adopted. To accelerate the training process, we reformulate MIED in the online setting and develop online multi-instance learning framework for EED (OMIED). To further exploit the nonlinear structure of the data distribution, we incorporate the kernel methods in OMIED, which results in the proposed online kernel multi-instance learning for early expression detection. Experiments on two popular and one challenging video-based expression data sets demonstrate both the efficiency and effectiveness of the proposed methods.

Early event detection for facial expression based on infinite mixture prototypes

Early Expression Detection Via Online Multi-Instance Learning with Nonlinear Extension

Meta-Learning with Dynamic-Memory-Based Prototypical Network for Few-Shot Event Detection.

Infinite Mixture Prototypes for Few-Shot Learning

Few-shot Event Detection: An Empirical Study and a Unified View

Online Kernel-Based Structured Output SVM for Early Expression Detection

Facial Action Units Detection Aided by Global-Local Expression Embedding

Semi-supervised Early Event Detection.

Efficient Early Event Detector for Streaming Sequence.

Human Detection Method Based on Multi-Part Detector and Multi-Instance Learning

DSFD: Dual Shot Face Detector

Class-Incremental Few-Shot Event Detection

Mixture Distribution Graph Network for Few Shot Learning

Multi-channels Prototype Contrastive Learning with Condition Adversarial Attacks for Few-shot Event Detection

Early Facial Expression Recognition Using Hidden Markov Models

Breaking Immutable: Information-Coupled Prototype Elaboration for Few-Shot Object Detection

One-Shot SADI-EPE: A Visual Framework of Event Progress Estimation

Complex Event Detection by Identifying Reliable Shots from Untrimmed Videos

Matching Compound Prototypes for Few-Shot Action Recognition

Multimodal Channel-Mixing: Channel and Spatial Masked AutoEncoder on Facial Action Unit Detection

Detecting Facial Action Units from Global-Local Fine-grained Expressions