Abstract:3D human reconstruction from RGB images achieves decent results in good weather conditions but degrades dramatically in rough weather. Complementarily, mmWave radars have been employed to reconstruct 3D human joints and meshes in rough weather. However, combining RGB and mmWave signals for weather-robust 3D human reconstruction is still an open challenge, given the sparse nature of mmWave and the vulnerability of RGB images. The limited research about the impact of missing points and sparsity features of mmWave data on reconstruction performance, as well as the lack of available datasets for paired mmWave-RGB data, further complicates the process of fusing the two modalities. To fill these gaps, we build up an automatic 3D body annotation system with multiple sensors to collect a large-scale mmWave dataset. The dataset consists of synchronized and calibrated mmWave radar point clouds and RGB(D) images under different weather conditions and skeleton/mesh annotations for humans in these scenes. With this dataset, we conduct a comprehensive analysis about the limitations of single-modality reconstruction and the impact of missing points and sparsity on the reconstruction performance. Based on the guidance of this analysis, we design ImmFusion, the first mmWave-RGB fusion solution to robustly reconstruct 3D human bodies in various weather conditions. Specifically, our ImmFusion consists of image and point backbones for token feature extraction and a Transformer module for token fusion. The image and point backbones refine global and local features from original data, and the Fusion Transformer Module aims for effective information fusion of two modalities by dynamically selecting informative tokens. Extensive experiments demonstrate that ImmFusion can efficiently utilize the information of two modalities to achieve robust 3D human body reconstruction in various weather environments. In addition, our method achieves superior accuracy compared to that of the state-of-the-art Transformer-based LiDAR-camera fusion methods.

Computing Thermal Point Clouds by Fusing RGB-D and Infrared Images: from Dense Object Reconstruction to Environment Mapping.

Depth and Thermal Sensor Fusion to Enhance 3D Thermographic Reconstruction

Thermal camera networks for large datacenters using real-time thermal monitoring mechanism

Multi-sensor Spatial Augmented Reality for Visualizing the Invisible Thermal Information of 3D Objects

On-the-fly Extrinsic Calibration of Multimodal Sensing System for Fast 3D Thermographic Scanning

Towards Weather-Robust 3D Human Body Reconstruction: Millimeter-Wave Radar-Based Dataset, Benchmark, and Multi-Modal Fusion

Fusion Algorithm of Laser-Point Clouds and Optical Images

Investigation on performance of RGB point cloud and thermal information data fusion for 3D building thermal map modeling using aerial images under different experimental conditions

3D Fusion of Infrared Images with Dense RGB Reconstruction from Multiple Views -- with Application to Fire-fighting Robots

Combining modern 3D reconstruction and thermal imaging: generation of large-scale 3D thermograms in real-time

Outdoor Scene 3D Reconstruction from Multiple Point Cloud

3D Radiometric Mapping by Means of LiDAR SLAM and Thermal Camera Data Fusion

3D Reconstruction Approach for Outdoor Scene Based on Multiple Point Cloud Fusion

Mobile thermal mapping for matching of infrared images with 3D building models and 3D point clouds

Fusion of Infrared Thermal Image and Visible Image for 3D Thermal Model Reconstruction Using Smartphone Sensors

RGB-D And Thermal Sensor Fusion: A Systematic Literature Review

360-Degree Tri-Modal Scanning: Engineering a Modular Multi-Sensor Platform for Semantic Enrichment of BIM Models

ThermalNeRF: Thermal Radiance Fields

Thermal-NeRF: Neural Radiance Fields from an Infrared Camera

Thermal-Inertial SLAM for the Environments With Challenging Illumination