Abstract:In this article, we presented mmPose-NLP, a novel natural language processing (NLP) inspired sequence-to-sequence (Seq2Seq) skeletal key-point estimator using millimeter-wave (mmWave) radar data. To the best of our knowledge, this is the first method to precisely estimate up to 25 skeletal key points using mmWave radar data alone. Skeletal pose estimation is critical in several applications ranging from autonomous vehicles, traffic monitoring, patient monitoring, and gait analysis, to defense security forensics, and aid both preventative and actionable decision making. The use of mmWave radars for this task, over traditionally employed optical sensors, provides several advantages, primarily its operational robustness to scene lighting and adverse weather conditions, where optical sensor performance degrade significantly. The mmWave radar point-cloud (PCL) data are first voxelized (analogous to tokenization in NLP) and N frames of the voxelized radar data (analogous to a text paragraph in NLP) is subjected to the proposed mmPose-NLP architecture, where the voxel indices of the 25 skeletal key points (analogous to keyword extraction in NLP) are predicted. The voxel indices are converted back to real-world 3-D coordinates using the voxel dictionary used during the tokenization process. Mean absolute error (MAE) metrics were used to measure the accuracy of the proposed system against the ground truth, with the proposed mmPose-NLP offering <3 cm localization errors in the depth, horizontal, and vertical axes. The effect of the number of input frames versus performance/accuracy was also studied for N = {1,2,...,10}. A comprehensive methodology, results, discussions, and limitations are presented in this article. All the source codes and results are made available on GitHub for further research and development in this critical yet emerging domain of skeletal key-point estimation using mmWave radars.

Spatial-temporal Multi-scale Constrained Learning for mmWave-based Human Pose Estimation

Mmskeleton: 3D Human Skeleton Estimation Using Millimeter Wave Radar Sparse Point Clouds

Temporal Constrained Feasible Subspace Learning for Human Pose Forecasting

UWB-Radar-Based 3D Human Pose Estimation Using Micro-Range/Micro-Doppler Images

A Joint Global–Local Network for Human Pose Estimation With Millimeter Wave Radar

Mpose: Environment- and Subject-Agnostic 3D Skeleton Posture Reconstruction Leveraging a Single Mmwave Device

Exploring Severe Occlusion: Multi-Person 3D Pose Estimation with Gated Convolution.

mm-Pose: Real-Time Human Skeletal Posture Estimation Using mmWave Radars and CNNs

ProbRadarM3F: mmWave Radar based Human Skeletal Pose Estimation with Probability Map Guided Multi-Format Feature Fusion

Three-Dimensional Human Pose Estimation from Micro-Doppler Signature Based on SISO UWB Radar

Vulnerable Road User Skeletal Pose Estimation Using mmWave Radars

mmPose-FK: A Forward Kinematics Approach to Dynamic Skeletal Pose Estimation Using mmWave Radars

MMBaT: A Multi-task Framework for mmWave-based Human Body Reconstruction and Translation Prediction

Real-Time Short-Range Human Posture Estimation Using mmWave Radars and Neural Networks

mmPose-NLP: A Natural Language Processing Approach to Precise Skeletal Pose Estimation Using mmWave Radars

RF-based Multi-view Pose Machine for Multi-Person 3D Pose Estimation.

SCRP-Radar: Space-Aware Coordinate Representation for Human Pose Estimation Based on SISO UWB Radar

Human Parsing with Joint Learning for Dynamic mmWave Radar Point Cloud

Multi-Scale Structure-Aware Network for Human Pose Estimation

Real-Time Through-Wall Multiperson 3-D Pose Estimation Based on MIMO Radar

Through-Wall Human Pose Reconstruction Based on Cross-Modal Learning and Self-Supervised Learning