Abstract:3D human reconstruction from RGB images achieves decent results in good weather conditions but degrades dramatically in rough weather. Complementarily, mmWave radars have been employed to reconstruct 3D human joints and meshes in rough weather. However, combining RGB and mmWave signals for weather-robust 3D human reconstruction is still an open challenge, given the sparse nature of mmWave and the vulnerability of RGB images. The limited research about the impact of missing points and sparsity features of mmWave data on reconstruction performance, as well as the lack of available datasets for paired mmWave-RGB data, further complicates the process of fusing the two modalities. To fill these gaps, we build up an automatic 3D body annotation system with multiple sensors to collect a large-scale mmWave dataset. The dataset consists of synchronized and calibrated mmWave radar point clouds and RGB(D) images under different weather conditions and skeleton/mesh annotations for humans in these scenes. With this dataset, we conduct a comprehensive analysis about the limitations of single-modality reconstruction and the impact of missing points and sparsity on the reconstruction performance. Based on the guidance of this analysis, we design ImmFusion, the first mmWave-RGB fusion solution to robustly reconstruct 3D human bodies in various weather conditions. Specifically, our ImmFusion consists of image and point backbones for token feature extraction and a Transformer module for token fusion. The image and point backbones refine global and local features from original data, and the Fusion Transformer Module aims for effective information fusion of two modalities by dynamically selecting informative tokens. Extensive experiments demonstrate that ImmFusion can efficiently utilize the information of two modalities to achieve robust 3D human body reconstruction in various weather environments. In addition, our method achieves superior accuracy compared to that of the state-of-the-art Transformer-based LiDAR-camera fusion methods.

ProbRadarM3F: mmWave Radar based Human Skeletal Pose Estimation with Probability Map Guided Multi-Format Feature Fusion

Mmskeleton: 3D Human Skeleton Estimation Using Millimeter Wave Radar Sparse Point Clouds

UWB-Radar-Based 3D Human Pose Estimation Using Micro-Range/Micro-Doppler Images

Indoor 3D Human Pose Estimation Using Single Millimeter-wave Radar and Conditional Generative Adversarial Network

Mpose: Environment- and Subject-Agnostic 3D Skeleton Posture Reconstruction Leveraging a Single Mmwave Device

Towards Weather-Robust 3D Human Body Reconstruction: Millimeter-Wave Radar-Based Dataset, Benchmark, and Multi-Modal Fusion

Capturing Human Pose Using Mmwave Radar.

mm-Pose: Real-Time Human Skeletal Posture Estimation Using mmWave Radars and CNNs

A Novel Multiperson Activity Recognition Algorithm Based on Point Clouds Measured by Millimeter-Wave MIMO Radar

Vulnerable Road User Skeletal Pose Estimation Using mmWave Radars

Enhancing Vital Sign Estimation Performance of FMCW MIMO Radar by Prior Human Shape Recognition

HuPR: A Benchmark for Human Pose Estimation Using Millimeter Wave Radar

RF-based Multi-view Pose Machine for Multi-Person 3D Pose Estimation.

SCRP-Radar: Space-Aware Coordinate Representation for Human Pose Estimation Based on SISO UWB Radar

mmPose-FK: A Forward Kinematics Approach to Dynamic Skeletal Pose Estimation Using mmWave Radars

mmPose-NLP: A Natural Language Processing Approach to Precise Skeletal Pose Estimation Using mmWave Radars

Real-Time Through-Wall Multiperson 3-D Pose Estimation Based on MIMO Radar

Three-Dimensional Human Pose Estimation from Micro-Doppler Signature Based on SISO UWB Radar

Real-Time Short-Range Human Posture Estimation Using mmWave Radars and Neural Networks

MMBaT: A Multi-task Framework for mmWave-based Human Body Reconstruction and Translation Prediction

Multi-Radar Inertial Odometry for 3D State Estimation using mmWave Imaging Radar