Abstract:Remote photo-plethysmography (rPPG) is a useful camera-based health motioning method that can measure the heart rhythm from facial videos. Many well-established deep learning models can provide highly accurate and robust results in measuring heart rate (HR) and heart rate variability (HRV). However, these methods are unable to effectively eliminate illumination variation and motion artifact disturbances, and their substantial computational resource requirements significantly limit their applicability in real-world scenarios. Hence, we propose a lightweight multi-frequency network named MFF-Net to measure heart rhythm via facial videos in a short time. Firstly, we propose a multi-frequency mode signal fusion (MFF) mechanism, which can separate the characteristics of different modes of the original rPPG signals and send them to a processor with independent parameters, helping the network recover blood volume pulse (BVP) signals accurately under a complex noise environment. In addition, in order to help the network extract the characteristics of different modal signals effectively, we designed a temporal multiscale convolution module (TMSC-module) and spectrum self-attention module (SSA-module). The TMSC-module can expand the receptive field of the signal-refining network, obtain more abundant multiscale information, and transmit it to the signal reconstruction network. The SSA-module can help a signal reconstruction network locate the obvious inferior parts in the reconstruction process so as to make better decisions when merging multi-dimensional signals. Finally, in order to solve the over-fitting phenomenon that easily occurs in the network, we propose an over-fitting sampling training scheme to further improve the fitting ability of the network. Comprehensive experiments were conducted on three benchmark datasets, and we estimated HR and HRV based on the BVP signals derived by MFF-Net. Compared with state-of-the-art methods, our approach achieves better performance both on HR and HRV estimation with lower computational burden. We can conclude that the proposed MFF-Net has the opportunity to be applied in many real-world scenarios.

Multi-hierarchical Convolutional Network for Efficient Remote Photoplethysmograph Signal and Heart Rate Estimation from Face Video Clips.

Non-contact PPG Signal and Heart Rate Estimation with Multi-Hierarchical Convolutional Network

CoSTHR: A Heart Rate Estimating Network with Adaptive Color Space Transformation

Heart Rate Estimation From Facial Videos Using a Spatiotemporal Representation With Convolutional Neural Networks

Information-enhanced Network for Noncontact Heart Rate Estimation from Facial Videos

Deep learning-based remote-photoplethysmography measurement from short-time facial video

Deep-HR: Fast Heart Rate Estimation from Face Video under Realistic Conditions

Convolutional neural network with spatio-temporal-channel attention for remote heart rate estimation

TCNTransNet: A Semi-Supervised Temporal-Spatial Fusion Framework for Heart Rate Estimation from Camera Video

Hyperbolic Embedding Steered Spatiotemporal Graph Convolutional Network for Video-Based Remote Heart Rate Estimation

Robust Heart Rate Estimation with Spatial–Temporal Attention Network from Facial Videos

MSDN: A Multistage Deep Network for Heart-Rate Estimation from Facial Videos

Rppg-Based Heart Rate Estimation Using Spatial-Temporal Attention Network

Non-Contact Heart Rate Estimation from Photoplethysmography Using EEMD and Convolution-Transformer Network

Lightweight and interpretable convolutional neural network for real-time heart rate monitoring using low-cost video camera under realistic conditions

TranPhys: Spatiotemporal Masked Transformer Steered Remote Photoplethysmography Estimation

RhythmNet: End-to-end Heart Rate Estimation from Face via Spatial-temporal Representation

A Feasibility Study of a Video-Based Heart Rate Estimation Method with Convolutional Neural Networks

Heart rate estimation by leveraging static and dynamic region weights

MFF-Net: A Lightweight Multi-Frequency Network for Measuring Heart Rhythm from Facial Videos

Efficient Spatiotemporal Attention Network for Remote Heart Rate Variability Analysis