Abstract:Graph self-supervised learning (SSL), including contrastive and generative approaches, offers great potential to address the fundamental challenge of label scarcity in real-world graph data. Among both sets of graph SSL techniques, the masked graph autoencoders (e.g., GraphMAE)--one type of generative method--have recently produced promising results. The idea behind this is to reconstruct the node features (or structures)--that are randomly masked from the input--with the autoencoder architecture. However, the performance of masked feature reconstruction naturally relies on the discriminability of the input features and is usually vulnerable to disturbance in the features. In this paper, we present a masked self-supervised learning framework GraphMAE2 with the goal of overcoming this issue. The idea is to impose regularization on feature reconstruction for graph SSL. Specifically, we design the strategies of multi-view random re-mask decoding and latent representation prediction to regularize the feature reconstruction. The multi-view random re-mask decoding is to introduce randomness into reconstruction in the feature space, while the latent representation prediction is to enforce the reconstruction in the embedding space. Extensive experiments show that GraphMAE2 can consistently generate top results on various public datasets, including at least 2.45% improvements over state-of-the-art baselines on ogbn-Papers100M with 111M nodes and 1.6B edges.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the decline in feature reconstruction performance in Graph Self - Supervised Learning (Graph SSL) due to the non - discriminative nature of node features. Specifically, existing graph auto - encoders based on mask prediction (such as GraphMAE) highly depend on the discriminability of input node features when performing feature reconstruction. When node features are not discriminative enough, the model is prone to over - fit to the noise in the input features, thus affecting the learning effect. To solve this problem, the paper proposes the GraphMAE2 framework, which improves the effect of feature reconstruction by introducing a regularization strategy.
### Main Contributions
1. **Multi - view Random Re - mask Decoding**:
- To reduce over - fitting to input features, the paper proposes introducing randomness in the decoding stage. The specific method is to randomly re - mask on the encoded representation and require the decoder to reconstruct the input features from these corrupted representations. This process is repeated multiple times to generate multiple different mask sets, and each set is used to reconstruct the input features. This randomness serves as a regularization method, preventing the network from memorizing unexpected patterns in the input features, thus making the training more robust to perturbations of the input features.
2. **Latent Representation Prediction**:
- In addition to reconstruction in the input feature space, the paper also proposes predicting the representations of masked nodes in the embedding space. The specific method is to use a target generator network to generate potential target representations from the unmasked graph, and then require the model to predict these target representations. This step aims to construct a prediction target that is less directly affected by input features, further improving the generalization ability of the model.
### Experimental Results
- **Benchmark Tests**: The experimental results show that GraphMAE2 consistently outperforms existing graph self - supervised learning baseline methods on multiple public datasets. Especially on large - scale graph datasets (such as ogbn - Papers100M, which contains 111 million nodes and 1.6 billion edges), the performance improvement of GraphMAE2 is significant, with at least a 2.45% increase compared to existing methods.
- **Sensitivity to Feature Discriminability**: Compared with supervised methods, GraphMAE has a more obvious performance decline when using compressed features (reduced to 50 dimensions by PCA). GraphMAE2, by introducing the above - mentioned regularization strategies, reduces the sensitivity to feature discriminability and shows better robustness.
### Extension to Large - scale Graphs
- The paper also explores how to extend GraphMAE2 to large - scale graphs with hundreds of millions of nodes. By generating local and dense sub - graphs through a local clustering strategy, GraphMAE2 can effectively handle the feature mask prediction problem of large - scale graphs. The experimental results show that even on large - scale graphs, GraphMAE2 can significantly improve performance.
In conclusion, by introducing two regularization strategies, multi - view random re - mask decoding and latent representation prediction, GraphMAE2 effectively solves the problem that existing graph auto - encoders are sensitive to the discriminability of input features in feature reconstruction, and improves the performance and robustness of graph self - supervised learning.