Abstract:3D human reconstruction from RGB images achieves decent results in good weather conditions but degrades dramatically in rough weather. Complementarily, mmWave radars have been employed to reconstruct 3D human joints and meshes in rough weather. However, combining RGB and mmWave signals for weather-robust 3D human reconstruction is still an open challenge, given the sparse nature of mmWave and the vulnerability of RGB images. The limited research about the impact of missing points and sparsity features of mmWave data on reconstruction performance, as well as the lack of available datasets for paired mmWave-RGB data, further complicates the process of fusing the two modalities. To fill these gaps, we build up an automatic 3D body annotation system with multiple sensors to collect a large-scale mmWave dataset. The dataset consists of synchronized and calibrated mmWave radar point clouds and RGB(D) images under different weather conditions and skeleton/mesh annotations for humans in these scenes. With this dataset, we conduct a comprehensive analysis about the limitations of single-modality reconstruction and the impact of missing points and sparsity on the reconstruction performance. Based on the guidance of this analysis, we design ImmFusion, the first mmWave-RGB fusion solution to robustly reconstruct 3D human bodies in various weather conditions. Specifically, our ImmFusion consists of image and point backbones for token feature extraction and a Transformer module for token fusion. The image and point backbones refine global and local features from original data, and the Fusion Transformer Module aims for effective information fusion of two modalities by dynamically selecting informative tokens. Extensive experiments demonstrate that ImmFusion can efficiently utilize the information of two modalities to achieve robust 3D human body reconstruction in various weather environments. In addition, our method achieves superior accuracy compared to that of the state-of-the-art Transformer-based LiDAR-camera fusion methods.

Ultraman: Single Image 3D Human Reconstruction with Ultra Speed and Detail

Towards Weather-Robust 3D Human Body Reconstruction: Millimeter-Wave Radar-Based Dataset, Benchmark, and Multi-Modal Fusion

Ihuman3d: Intelligent Human Body 3D Reconstruction Using a Single Flying Camera.

High-precision Human Body Acquisition Via Multi-View Binocular Stereopsis

3D Human Reconstruction from A Single Depth Image

GeneMAN: Generalizable Single-Image 3D Human Reconstruction from Multi-Source Human Data

PSHuman: Photorealistic Single-view Human Reconstruction using Cross-Scale Diffusion

3D Real Human Reconstruction Via Multiple Low-Cost Depth Cameras.

Dense 3D Facial Reconstruction from a Single Depth Image in Unconstrained Environment

Enhanced Multi-Scale Attention-Driven 3D Human Reconstruction from Single Image

CrossHuman: Learning Cross-Guidance from Multi-Frame Images for Human Reconstruction

MUC: Mixture of Uncalibrated Cameras for Robust 3D Human Body Reconstruction

ReFu: Refine and Fuse the Unobserved View for Detail-Preserving Single-Image 3D Human Reconstruction

Image-Guided Human Reconstruction via Multi-Scale Graph Transformation Networks

MVP-Human Dataset for 3D Human Avatar Reconstruction from Unconstrained Frames

Deep3DPose: Realtime Reconstruction of Arbitrarily Posed Human Bodies from Single RGB Images

High-Fidelity 3D Digital Human Head Creation from RGB-D Selfies

Detailed 3D Human Body Reconstruction from Multi-view Images Combining Voxel Super-Resolution and Learned Implicit Representation

Video-Based Outdoor Human Reconstruction.

Efficient Neural Implicit Representation for 3D Human Reconstruction

3DPMesh: An enhanced and novel approach for the reconstruction of 3D human meshes from a single 2D image