Abstract:Deep learning is becoming the most widely used technology for multi-sensor data fusion. Semantic correspondence has recently emerged as a foundational task, enabling a range of downstream applications, such as style or appearance transfer, robot manipulation, and pose estimation, through its ability to provide robust correspondence in RGB images with semantic information. However, current representations generated by self-supervised learning and generative models are often limited in their ability to capture and understand the geometric structure of objects, which is significant for matching the correct details in applications of semantic correspondence. Furthermore, efficiently fusing these two types of features presents an interesting challenge. Achieving harmonious integration of these features is crucial for improving the expressive power of models in various tasks. To tackle these issues, our key idea is to integrate depth information from depth estimation or depth sensors into feature maps and leverage learnable weights for feature fusion. First, depth information is used to model pixel-wise depth distributions, assigning relative depth weights to feature maps for perceiving an object's structural information. Then, based on a contrastive learning optimization objective, a series of weights are optimized to leverage feature maps from self-supervised learning and generative models. Depth features are naturally embedded into feature maps, guiding the network to learn geometric structure information about objects and alleviating depth ambiguity issues. Experiments on the SPair-71K and AP-10K datasets show that the proposed method achieves scores of 81.8 and 83.3 on the percentage of correct keypoints (PCK) at the 0.1 level, respectively. Our approach not only demonstrates significant advantages in experimental results but also introduces the depth awareness module and a learnable feature fusion module, which enhances the understanding of object structures through depth information and fully utilizes features from various pre-trained models, offering new possibilities for the application of deep learning in RGB and depth data fusion technologies. We will also continue to focus on accelerating model inference and optimizing model lightweighting, enabling our model to operate at a faster speed.

Deep Learning-Based RGB-D Fusion for Multimodal Condition Assessment of Civil Infrastructure

Deep Learning-Based Damage Inspection for Concrete Structures

Scale Invariant Semantic Segmentation with RGB-D Fusion

Image-based Reinforced Concrete Component Mechanical Damage Recognition and Structural Safety Rapid Assessment Using Deep Learning with Frequency Information

Detecting Internal Defects in FRP-Reinforced Concrete Structures through the Integration of Infrared Thermography and Deep Learning

Multi-Modal Fusion Based on Depth Adaptive Mechanism for 3D Object Detection

Deep Learning-Based Thermal Image Analysis for Pavement Defect Detection and Classification Considering Complex Pavement Conditions

Unveiling the Depths: A Multi-Modal Fusion Framework for Challenging Scenarios

Civil Infrastructure Damage and Corrosion Detection: An Application of Machine Learning

RGB-Fusion: Monocular 3D reconstruction with learned depth prediction

Deep Learning-Based Concrete Surface Damage Monitoring Method Using Structured Lights and Depth Camera

Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images

CMDFusion: Bidirectional Fusion Network with Cross-modality Knowledge Distillation for LIDAR Semantic Segmentation

HFMDNet: Hierarchical Fusion and Multilevel Decoder Network for RGB-D Salient Object Detection

A Depth Awareness and Learnable Feature Fusion Network for Enhanced Geometric Perception in Semantic Correspondence

Vision-based Structural Inspection using Multiscale Deep Convolutional Neural Networks

Optimally leveraging depth features to enhance segmentation of recyclables from cluttered construction and demolition waste streams

RGB-LiDAR fusion for accurate 2D and 3D object detection

Crack Detection and Segmentation Using Deep Learning with 3D Reality Mesh Model for Quantitative Assessment and Integrated Visualization

One-class Damage Detector Using Deeper Fully-Convolutional Data Descriptions for Civil Application