Abstract:The non-Euclidean geometry of hyperbolic spaces has recently garnered considerable attention in the realm of representation learning. Current endeavors in hyperbolic representation largely presuppose that the underlying hierarchies can be automatically inferred and preserved through the adaptive optimization process. This assumption, however, is questionable and requires further validation. In this work, we first introduce a position-tracking mechanism to scrutinize existing prevalent \hlms, revealing that the learned representations are sub-optimal and unsatisfactory. To address this, we propose a simple yet effective method, hyperbolic informed embedding (HIE), by incorporating cost-free hierarchical information deduced from the hyperbolic distance of the node to origin (i.e., induced hyperbolic norm) to advance existing \hlms. The proposed method HIE is both task-agnostic and model-agnostic, enabling its seamless integration with a broad spectrum of models and tasks. Extensive experiments across various models and different tasks demonstrate the versatility and adaptability of the proposed method. Remarkably, our method achieves a remarkable improvement of up to 21.4\% compared to the competing baselines.
What problem does this paper attempt to address?
The main problem that this paper attempts to solve is that current hyperspherical representation learning models are insufficient in capturing the hierarchical structure of data. Specifically, existing hyperspherical models assume that the underlying hierarchical structure can be automatically inferred and preserved through an adaptive optimization process, but the validity of this assumption is questionable. By introducing a position - tracking mechanism, the author finds that the representations learned by existing models are not ideal and fail to fully maintain the hierarchical structure of the data. To solve this problem, the author proposes a new method - Hyperbolic Informed Embedding (HIE), which improves existing hyperspherical models by combining cost - free hierarchical information extracted from the hyperspherical distance from nodes to the origin (i.e., the induced hyperspherical norm).
### Main Contributions
1. **Proposing a position - tracking strategy**: The author proposes a position - tracking strategy to investigate currently popular hyperspherical models, revealing significant differences between the hyperspherical learning process and the traditional understanding, and providing new insights into the process of hyperspherical representation learning.
2. **Introducing an implicit hierarchical inference method**: The author proposes a novel method that can infer the implicit hierarchical structure from hyperspherical embeddings. This method is cost - free, scalable, and efficient because it directly extracts hierarchical information from the embeddings themselves without requiring additional inputs or annotations.
3. **Proposing a simple and effective improvement method**: The author proposes a simple and effective method that uses the inferred hierarchical structure to improve hyperspherical representation learning. This method can be seamlessly integrated into existing hyperspherical models without introducing any additional model parameters, making it practical and easy to implement.
4. **Extensive experimental verification**: The author conducts extensive experiments on various hyperspherical models, demonstrating the effectiveness of the proposed method. The experimental results show that the method has a significant improvement compared to the baseline models, with a maximum improvement of 21.4%.
### Method Overview
1. **Root - node Alignment**: Calculate the hyperspherical embedding center (HC) as the root node and align it with the hyperspherical origin. HC is calculated by a specific manifold method and can minimize the sum of squared distances to all nodes.
2. **Hierarchical Stretching**: Use the hyperspherical distance to the origin (HDO) to encapsulate implicit hierarchical information. By aligning the hyperspherical embedding center (i.e., the root node) with the hyperspherical origin, HDO more accurately reflects the relative distance of a node with respect to the root node, thus indicating its hierarchical level. By minimizing the loss function \( L_{\text{hyp}} \), nodes are promoted to move away from the origin, and in particular, low - level nodes are given larger weights to ensure that they reach the correct positions.
### Formulas
- **Root - node Alignment**:
\[
\bar{z} = z \oplus_\kappa (-z_c)
\]
where \( \oplus_\kappa \) represents the hyperspherical addition operation.
- **HDO Calculation**:
\[
z_{\text{hdo}} = \frac{1}{|V|} \sum_{i \in V} w_i d_H(\bar{z}_i, o)
\]
where \( w_i \) represents the level of a node in the hyperspherical space, calculated by HDO, i.e., \( w_i = f(d_H(\bar{z}_i, o)) \), and \( f \) is a monotonically increasing function.
- **Hierarchical Stretching**:
\[
L_{\text{hyp}} = \sigma(-z_{\text{hdo}})
\]
where \( \sigma \) is a monotonically increasing function, such as a linear function, tanh, exp, etc.
### Experimental Results
The author conducts experiments on multiple tasks, including link prediction and node classification. The experimental results show that the HIE method exhibits significant performance improvements on various models. For example, in the link prediction task on the DISEASE dataset, with 75% of the training links, the AUC metric is improved by 10.4% and the AP metric is improved by 7.7%; with 25% of the training links, the AUC metric is improved by 21.4% and the AP metric is improved by 20.4%.
In conclusion, this paper...