Abstract:Graph Masked Autoencoders (GMAEs) have emerged as a notable self-supervised learning approach for graph-structured data. Existing GMAE models primarily focus on reconstructing node-level information, categorizing them as single-scale GMAEs. This methodology, while effective in certain contexts, tends to overlook the complex hierarchical structures inherent in many real-world graphs. For instance, molecular graphs exhibit a clear hierarchical organization in the form of the atoms-functional groups-molecules structure. Hence, the inability of single-scale GMAE models to incorporate these hierarchical relationships often leads to their inadequate capture of crucial high-level graph information, resulting in a noticeable decline in performance. To address this limitation, we propose Hierarchical Graph Masked AutoEncoders (Hi-GMAE), a novel multi-scale GMAE framework designed to handle the hierarchical structures within graphs. First, Hi-GMAE constructs a multi-scale graph hierarchy through graph pooling, enabling the exploration of graph structures across different granularity levels. To ensure masking uniformity of subgraphs across these scales, we propose a novel coarse-to-fine strategy that initiates masking at the coarsest scale and progressively back-projects the mask to the finer scales. Furthermore, we integrate a gradual recovery strategy with the masking process to mitigate the learning challenges posed by completely masked subgraphs. Diverging from the standard graph neural network (GNN) used in GMAE models, Hi-GMAE modifies its encoder and decoder into hierarchical structures. This entails using GNN at the finer scales for detailed local graph analysis and employing a graph transformer at coarser scales to capture global information. Our experiments on 15 graph datasets consistently demonstrate that Hi-GMAE outperforms 17 state-of-the-art self-supervised competitors.

Distill Graph Structure Knowledge from Masked Graph Autoencoders into MLP

Knowledge Distillation Improves Graph Structure Augmentation for Graph Neural Networks

Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework

Edge-free but Structure-aware: Prototype-Guided Knowledge Distillation from GNNs to MLPs

Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation

AdaGMLP: AdaBoosting GNN-to-MLP Knowledge Distillation

Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework

Decoupled graph knowledge distillation: A general logits-based method for learning MLPs on graphs

SA-MLP: Distilling Graph Knowledge from GNNs into Structure-Aware MLP

Hi-GMAE: Hierarchical Graph Masked Autoencoders

Enhanced Scalable Graph Neural Network via Knowledge Distillation

Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference

VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs

UGMAE: A Unified Framework for Graph Masked Autoencoders

Online Adversarial Knowledge Distillation for Graph Neural Networks

What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders

EGNN: Constructing explainable graph neural networks via knowledge distillation

GraphMAE2: A Decoding-Enhanced Masked Self-Supervised Graph Learner

Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification

Where to Mask: Structure-Guided Masking for Graph Masked Autoencoders

Graph Knowledge Distillation to Mixture of Experts