Abstract:Graph Neural Networks (GNNs) rely on the graph structure to define an aggregation strategy where each node updates its representation by combining information from its neighbours. A known limitation of GNNs is that, as the number of layers increases, information gets smoothed and squashed and node embeddings become indistinguishable, negatively affecting performance. Therefore, practical GNN models employ few layers and only leverage the graph structure in terms of limited, small neighbourhoods around each node. Inevitably, practical GNNs do not capture information depending on the global structure of the graph. While there have been several works studying the limitations and expressivity of GNNs, the question of whether practical applications on graph structured data require global structural knowledge or not, remains unanswered. In this work, we empirically address this question by giving access to global information to several GNN models, and observing the impact it has on downstream performance. Our results show that global information can in fact provide significant benefits for common graph-related tasks. We further identify a novel regularization strategy that leads to an average accuracy improvement of more than 5% on all considered tasks.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is an important limitation of Graph Neural Networks (GNNs) when processing graph - structured data: **GNNs cannot effectively utilize the overall structural information of the graph**. Specifically, as the number of GNN layers increases, node representations will become smooth and difficult to distinguish, leading to a decline in performance. Therefore, existing GNN models usually only utilize the information of local neighborhoods and ignore the overall structural information of the graph. This raises the question of whether global structural information is required in practical applications.
To answer this question, the paper experimentally studies the impact of providing global structural information to GNN models. Specifically, the authors propose three different methods to inject global structural information and evaluate the impact of these methods on the performance of common graph - related tasks (such as node classification, graph classification, and triangle counting). The research results show that global structural information can significantly improve the performance of GNN models, and a regularization strategy (RWRReg) based on Random Walk with Restart (RWR) is proposed. This strategy improves the accuracy by more than 5% on average without increasing the model parameters.
### Main contributions
1. **Propose and formalize three ways to inject global structural information**:
- Directly provide the adjacency matrix.
- Provide Random Walk with Restart (RWR) coefficients.
- Combine RWR coefficients and the RWRReg regularization term.
2. **Propose a new regularization strategy RWRReg**:
- This strategy is based on RWR and can significantly improve model performance without increasing the input size or the number of parameters.
- RWRReg does not add additional operations during inference time, nor does it require additional supervision, while maintaining the permutation invariance of GNN models.
3. **Theoretical proof**:
- Prove that the information extracted by RWR can accelerate the 1 - Weisfeiler - Leman (1 - WL) algorithm, thereby improving the ability of GNN models to distinguish non - isomorphic graphs.
### Experimental results
- **Node classification**: On the Cora, Pubmed, and Citeseer datasets, after adding RWR features and the RWRReg regularization term, the performance is improved by 2% to 7.5%.
- **Graph classification**: On the ENZYMES, D&D, and PROTEINS datasets, the performance improvement is more significant, up to 11%.
- **Triangle counting**: On the TRIANGLES dataset, after adding RWR features and the RWRReg regularization term, the performance is improved by 19%, especially when dealing with unseen graph sizes.
### Practical aspects
- **The practicality of RWRReg**: RWRReg does not increase the number of model parameters, does not require additional inference operations, and does not require additional supervision. It is an efficient improvement strategy.
- **Sparsification of the RWR matrix**: To reduce storage overhead, the paper explores the sparsification method of the RWR matrix and finds that even if only the top K highest RWR weights for each node are retained, the performance improvement can still be maintained.
In conclusion, this paper experimentally proves the importance of global structural information for GNN models and proposes a practical regularization strategy to utilize this information, thereby significantly improving the performance of the model.