Abstract:Scene graphs have been recently introduced into 3D spatial understanding as a comprehensive representation of the scene. The alignment between 3D scene graphs is the first step of many downstream tasks such as scene graph aided point cloud registration, mosaicking, overlap checking, and robot navigation. In this work, we treat 3D scene graph alignment as a partial graph-matching problem and propose to solve it with a graph neural network. We reuse the geometric features learned by a point cloud registration method and associate the clustered point-level geometric features with the node-level semantic feature via our designed feature fusion module. Partial matching is enabled by using a learnable method to select the top-k similar node pairs. Subsequent downstream tasks such as point cloud registration are achieved by running a pre-trained registration network within the matched regions. We further propose a point-matching rescoring method, that uses the node-wise alignment of the 3D scene graph to reweight the matching candidates from a pre-trained point cloud registration method. It reduces the false point correspondences estimated especially in low-overlapping cases. Experiments show that our method improves the alignment accuracy by 10~20% in low-overlap and random transformation scenarios and outperforms the existing work in multiple downstream tasks.

What problem does this paper attempt to address?

### Problems Addressed by the Paper This paper aims to address the alignment problem between 3D scene graphs. Specifically, the authors view 3D scene graph alignment as a partial graph matching problem and propose a graph neural network-based method to solve this issue. 3D scene graph alignment is the first step for many downstream tasks, such as point cloud registration, stitching, overlap checking, and robot navigation. ### Main Contributions 1. **Subgraph Matching Network (SG-PGM)**: A graph neural network is proposed for partial graph matching to solve the 3D scene graph alignment problem. 2. **Point-to-Scene Graph Fusion Module (P2SG) and Soft Top-k Method**: These modules improve alignment accuracy. 3. **Superpoint Matching Rescoring Method**: This method uses the predicted scene graph node alignment results to guide point matching in point cloud registration. 4. **Revisiting Strategy**: The paper explores how to use 3D scene graph alignment results to improve the performance of downstream tasks. ### Method Overview 1. **Problem Definition**: A 3D scene graph is a graph model containing semantic nodes and edge attributes. The authors formalize the imprecise graph matching problem as an optimization objective function. 2. **Subgraph Matching Network**: First, the semantic node features and edge features of the source and reference graphs are projected into graph embeddings. Then, combined with the geometric embeddings generated by the point cloud encoder, a fused embedding is formed. The AIS module calculates the similarity matrix of node pairs, and the Sinkhorn decoder is used for soft matching prediction. To achieve partial matching, the soft top-k algorithm is introduced. 3. **Point-to-Scene Graph Feature Fusion**: A new P2SG module is designed to aggregate point-level geometric features into object node-level features, forming more distinguishable joint embeddings. 4. **Superpoint Matching Rescoring**: The semantic similarity learned by the scene graph matching network is used to reweight point matching scores, reducing mismatches. 5. **Loss Function**: Negative cross-entropy loss is used to supervise the soft correspondence prediction of graph matching, and mean squared error loss is used to supervise the learning of graph similarity. 6. **Downstream Tasks**: The paper explores improvement strategies for downstream tasks such as overlap checking and point cloud registration. ### Experimental Results 1. **Scene Graph Alignment and Overlap Checking**: Experiments were conducted on the 3RScan dataset, and the results show that this method significantly outperforms existing methods in terms of node alignment accuracy and F1 score, especially in cases of low overlap and random transformations. 2. **Point Cloud Registration and Stitching**: Using scene graph alignment results as priors, the method supports the pre-trained GeoTransformer for point cloud registration and stitching. Experimental results show that this method outperforms SGAligner in multiple metrics and is faster. ### Conclusion This paper proposes an effective subgraph matching network (SG-PGM) that improves the accuracy of 3D scene graph alignment by combining semantic and geometric features. Additionally, the superpoint matching rescoring method further enhances the accuracy of point cloud registration. Experimental results validate the superior performance of this method in various downstream tasks.

SG-PGM: Partial Graph Matching Network with Semantic Geometric Fusion for 3D Scene Graph Alignment and Its Downstream Tasks

Semantic Graph Based Place Recognition for 3D Point Clouds.

GraphAlign: Enhancing Accurate Feature Alignment by Graph matching for Multi-Modal 3D Object Detection

Up-to-Down Network: Fusing Multi-Scale Context for 3D Semantic Scene Completion

GraphAlign++: An Accurate Feature Alignment by Graph Matching for Multi-Modal 3D Object Detection

Graph Matching Optimization Network for Point Cloud Registration

Multi-features Guidance Network for Partial-to-partial Point Cloud Registration

Deep Semantic Graph Matching for Large-Scale Outdoor Point Cloud Registration

Deep Semantic Graph Matching for Large-scale Outdoor Point Clouds Registration

Ensemble Quadratic Assignment Network for Graph Matching

2D-3DMatchingNet: Multimodal Point Completion with 2D Geometry Matching

PointGS: Bridging and Fusing Geometric and Semantic Space for 3D Point Cloud Analysis

Leveraging Commonsense for Object Localisation in Partial Scenes

Joint Graph Learning and Matching for Semantic Feature Correspondence

Semantic-Context Graph Network for Point-based 3D Object Detection

AGO-Net: Association-Guided 3D Point Cloud Object Detection Network

Learning Semantic Alignment Using Global Features and Multi-scale Confidence

Point Cloud Registration Based on Graph Matching Optimization

SGNet: Salient Geometric Network for Point Cloud Registration

3D Scene Graph Generation from Point Clouds

PA-GM: Position-Aware Learning of Embedding Networks for Deep Graph Matching.