Abstract:3D human reconstruction from RGB images achieves decent results in good weather conditions but degrades dramatically in rough weather. Complementarily, mmWave radars have been employed to reconstruct 3D human joints and meshes in rough weather. However, combining RGB and mmWave signals for weather-robust 3D human reconstruction is still an open challenge, given the sparse nature of mmWave and the vulnerability of RGB images. The limited research about the impact of missing points and sparsity features of mmWave data on reconstruction performance, as well as the lack of available datasets for paired mmWave-RGB data, further complicates the process of fusing the two modalities. To fill these gaps, we build up an automatic 3D body annotation system with multiple sensors to collect a large-scale mmWave dataset. The dataset consists of synchronized and calibrated mmWave radar point clouds and RGB(D) images under different weather conditions and skeleton/mesh annotations for humans in these scenes. With this dataset, we conduct a comprehensive analysis about the limitations of single-modality reconstruction and the impact of missing points and sparsity on the reconstruction performance. Based on the guidance of this analysis, we design ImmFusion, the first mmWave-RGB fusion solution to robustly reconstruct 3D human bodies in various weather conditions. Specifically, our ImmFusion consists of image and point backbones for token feature extraction and a Transformer module for token fusion. The image and point backbones refine global and local features from original data, and the Fusion Transformer Module aims for effective information fusion of two modalities by dynamically selecting informative tokens. Extensive experiments demonstrate that ImmFusion can efficiently utilize the information of two modalities to achieve robust 3D human body reconstruction in various weather environments. In addition, our method achieves superior accuracy compared to that of the state-of-the-art Transformer-based LiDAR-camera fusion methods.

Multi-Sensor Fusion Based Environment Reconstruction in Integrated Sensing and Communications

Environment Reconstruction based on Multi-User Selection and Multi-Modal Fusion in ISAC

Deep Learning Based Multi-Node ISAC 4D Environmental Reconstruction with Uplink- Downlink Cooperation

Towards Weather-Robust 3D Human Body Reconstruction: Millimeter-Wave Radar-Based Dataset, Benchmark, and Multi-Modal Fusion

Intelligent Reflecting Surface Aided Environment Sensing with CNN-based Correction.

6G Integrated Sensing and Communication - Sensing Assisted Environmental Reconstruction and Communication

3D Environment Sensing with Channel State Information Based on Computational Imaging

Joint Multi-User Communication and Sensing Exploiting Both Signal and Environment Sparsity

Integrated Sensing and Communications Framework for 6G Networks

Multi-View Sensing for Wireless Communications: Architectures, Designs, and Opportunities

Radar and Camera Fusion for Multi-Task Sensing in Autonomous Driving

Multi-Point Integrated Sensing and Communication: Fusion Model and Functionality Selection

Environment Sensing with Beam Sweeping and Non-Uniform Pixelation in Wireless Communication Systems

Networked Integrated Sensing and Communications for 6G Wireless Systems

Outdoor 3D Environment Reconstruction based on Multi-sensor Fusion for Remote Control

Environment Sensing Considering the Occlusion Effect: A Multi-View Approach

Integrated Sensing and Communications for 3D Object Imaging via Bilinear Inference

ISAC 4D Imaging System Based on 5G Downlink Millimeter Wave Signal

Special issue on integrated sensing and communications (ISAC)

Integrated Sensing and Communication for Edge Inference with End-to-End Multi-View Fusion