Abstract:3D human reconstruction from RGB images achieves decent results in good weather conditions but degrades dramatically in rough weather. Complementarily, mmWave radars have been employed to reconstruct 3D human joints and meshes in rough weather. However, combining RGB and mmWave signals for weather-robust 3D human reconstruction is still an open challenge, given the sparse nature of mmWave and the vulnerability of RGB images. The limited research about the impact of missing points and sparsity features of mmWave data on reconstruction performance, as well as the lack of available datasets for paired mmWave-RGB data, further complicates the process of fusing the two modalities. To fill these gaps, we build up an automatic 3D body annotation system with multiple sensors to collect a large-scale mmWave dataset. The dataset consists of synchronized and calibrated mmWave radar point clouds and RGB(D) images under different weather conditions and skeleton/mesh annotations for humans in these scenes. With this dataset, we conduct a comprehensive analysis about the limitations of single-modality reconstruction and the impact of missing points and sparsity on the reconstruction performance. Based on the guidance of this analysis, we design ImmFusion, the first mmWave-RGB fusion solution to robustly reconstruct 3D human bodies in various weather conditions. Specifically, our ImmFusion consists of image and point backbones for token feature extraction and a Transformer module for token fusion. The image and point backbones refine global and local features from original data, and the Fusion Transformer Module aims for effective information fusion of two modalities by dynamically selecting informative tokens. Extensive experiments demonstrate that ImmFusion can efficiently utilize the information of two modalities to achieve robust 3D human body reconstruction in various weather environments. In addition, our method achieves superior accuracy compared to that of the state-of-the-art Transformer-based LiDAR-camera fusion methods.

Clip Fusion with Bi-level Optimization for Human Mesh Reconstruction from Monocular Videos

Towards Weather-Robust 3D Human Body Reconstruction: Millimeter-Wave Radar-Based Dataset, Benchmark, and Multi-Modal Fusion

MH‐HMR: Human mesh recovery from monocular images via multi‐hypothesis learning

Marker-Less 3d Human Motion Capture With Monocular Image Sequence And Height-Maps

SelfRecon: Self Reconstruction Your Digital Avatar from Monocular Video

Human Mesh Recovery from Arbitrary Multi-view Images

Mixed Transformer for Temporal 3D Human Pose and Shape Estimation from Monocular Video

STAF: 3D Human Mesh Recovery from Video with Spatio-Temporal Alignment Fusion

HyperGraph Based Human Mesh Hierarchical Representation and Reconstruction from a Single Image.

AdaptiveFusion: Adaptive Multi-Modal Multi-View Fusion for 3D Human Body Reconstruction

Dual-Branch Graph Transformer Network for 3D Human Mesh Reconstruction from Video

DiffMesh: A Motion-aware Diffusion Framework for Human Mesh Recovery from Videos

Human Mesh Recovery from Monocular Images via a Skeleton-disentangled Representation

Synergistic Global-space Camera and Human Reconstruction from Videos

PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos

Temporally Coherent Full 3D Mesh Human Pose Recovery from Monocular Video

VH3D-LSFM: Video-Based Human 3D Pose Estimation with Long-Term and Short-Term Pose Fusion Mechanism.

MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction

Capturing Humans in Motion: Temporal-Attentive 3D Human Pose and Shape Estimation from Monocular Video

SimHMR: A Simple Query-based Framework for Parameterized Human Mesh Reconstruction

Simultaneously Recovering Multi-Person Meshes and Multi-View Cameras with Human Semantics