Abstract:While most network embedding techniques model the proximity between nodes in a network, recently there has been significant interest in structural embeddings that are based on node equivalences, a notion rooted in sociology: equivalences or positions are collections of nodes that have similar roles--i.e., similar functions, ties or interactions with nodes in other positions--irrespective of their distance or reachability in the network. Unlike the proximity-based methods that are rigorously evaluated in the literature, the evaluation of structural embeddings is less mature. It relies on small synthetic or real networks with labels that are not perfectly defined, and its connection to sociological equivalences has hitherto been vague and tenuous. With new node embedding methods being developed at a breakneck pace, proper evaluation and systematic characterization of existing approaches will be essential to progress. To fill in this gap, we set out to understand what types of equivalences structural embeddings capture. We are the first to contribute rigorous intrinsic and extrinsic evaluation methodology for structural embeddings, along with carefully-designed, diverse datasets of varying sizes. We observe a number of different evaluation variables that can lead to different results (e.g., choice of similarity measure, classifier, label definitions). We find that degree distributions within nodes' local neighborhoods can lead to simple yet effective baselines in their own right and guide the future development of structural embedding. We hope that our findings can influence the design of further node embedding methods and also pave the way for more comprehensive and fair evaluation of structural embedding methods.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the insufficient evaluation of structural node embedding methods. Specifically, although most network embedding techniques mainly focus on the proximity between nodes, in recent years, structural embeddings based on node equivalence have attracted significant interest. These structural embedding methods aim to capture nodes with similar roles or functions, regardless of their distance or reachability in the network. However, unlike the proximity embedding methods that have been well - evaluated, the evaluation of structural embeddings is less mature. Existing evaluation methods rely on small - scale synthetic or real - world networks, and the label definitions are not clear enough, and the connection with the concept of equivalence in sociology is also rather vague. Therefore, the goal of this paper is to fill this gap by introducing rigorous intrinsic and extrinsic evaluation methods, as well as carefully designed diverse datasets, to systematically analyze existing structural embedding methods.
### Main Contributions:
1. **Evaluation Methods**: This is the first paper to introduce multiple evaluation methods for unsupervised structural node embeddings. These methods include:
- **Intrinsic Evaluation**: Related indicators based on the definition of equivalence, which help to decouple the effectiveness of the method from the classifier performance in downstream tasks.
- **Extrinsic Evaluation**: Indicators that characterize performance in high - value tasks, rethinking the ground truth used in previous work.
2. **Appropriate Datasets**: Introduce new benchmark datasets and provide methods for obtaining benchmark roles. It is hoped that these datasets can change the way structural embeddings are evaluated.
3. **In - depth Empirical Analysis**: Conduct an empirical analysis of 11 methods on 35 real - world and synthetic datasets for multiple tasks, showing that different methods perform differently under different metrics, label definitions, downstream machine - learning models, or embedding similarity functions. This indicates that there is no single optimal structural embedding method.
4. **New Design Insights**: Discover that the degree distribution of the local neighborhood of nodes is effective as a feature representation in itself and can also serve as the basis for some of the most successful embedding methods. This can influence the design of future structural embedding methods or serve as an independent baseline for structural embedding tasks.
### Research Objectives:
- **Understanding**: Understand which aspects of "structural roles" are captured by node embeddings. Evaluate the effectiveness of each embedding method through the concept of role equivalence in mathematical sociology and network science statistics.
- **Evaluation**: Propose new intrinsic and extrinsic evaluation methods. Intrinsic evaluation directly assesses the geometric structure of the node embedding space, independent of any downstream task or method. Extrinsic evaluation analyzes the performance of downstream tasks using node embeddings.
### Data and Benchmark Roles:
- **Real - world Datasets**: Include datasets such as BlogCatalog, Facebook, ICEWS, Email - 300, Email - 2K, PPI, BR air - traffic, EU air - traffic, US air - traffic, and DD6, covering centrality, equivalence, military and media entities, occupational roles, protein cell functions, the number of flight take - offs and landings, etc.
- **Synthetic Datasets**: Design a large number of synthetic datasets to better control and understand the data structure, especially for showing clear role equivalence.
Through these contributions, the paper aims to provide the research community with more comprehensive and systematic understanding and evaluation tools for structural node embedding methods, thereby promoting the further development of this field.