Abstract:Hyperbolic Neural Networks (HNNs), operating in hyperbolic space, have been widely applied in recent years, motivated by the existence of an optimal embedding in hyperbolic space that can preserve data hierarchical relationships (termed Hierarchical Representation Capability, HRC) more accurately than Euclidean space. However, there is no evidence to suggest that HNNs can achieve this theoretical optimal embedding, leading to much research being built on flawed motivations. In this paper, we propose a benchmark for evaluating HRC and conduct a comprehensive analysis of why HNNs are effective through large-scale experiments. Inspired by the analysis results, we propose several pre-training strategies to enhance HRC and improve the performance of downstream tasks, further validating the reliability of the analysis. Experiments show that HNNs cannot achieve the theoretical optimal embedding. The HRC is significantly affected by the optimization objectives and hierarchical structures, and enhancing HRC through pre-training strategies can significantly improve the performance of HNNs.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: Why are hyperbolic neural networks (HNNs) effective? Specifically, the paper explores the influence of the hierarchical representation capability (HRC) in hyperbolic space on the performance of HNNs, and proposes methods for evaluating HRC and methods for enhancing HRC through pre - training strategies to improve the performance of downstream tasks. ### Background and Problems of the Paper Hyperbolic neural networks (HNNs) perform well in processing hierarchical data, mainly because the hyperbolic space can more accurately preserve the hierarchical relationships of data (i.e., HRC). However, there is currently no evidence that HNNs can achieve theoretically optimal embeddings, causing many studies to be based on imperfect foundations. Therefore, the paper proposes a benchmarking method to evaluate HRC and analyzes why HNNs are effective through large - scale experiments. ### Main Contributions 1. **Proposing the HRC Benchmark (HRCB)** - HRCB is used to evaluate the hierarchical representation ability of HNNs. - Through HRCB, researchers can evaluate the application scope and applicability of HNNs and gain in - depth understanding of their underlying mechanisms. - The analysis results show that the effectiveness of HNNs is affected by multiple factors. 2. **Proposing Pre - training Strategies** - Based on the analysis results of HRCB, multiple pre - training strategies are proposed to enhance HRC. - These strategies further verify the correctness of the HRCB analysis and improve the performance of HNNs within the applicable range. 3. **Extensive Experimental Verification** - Through thousands of experiments, three model structures, three manifold spaces, and eight dimensions are tested. - The reliability of the conclusions is verified using statistical significance tests. ### Experimental Design and Results 1. **HRC Evaluation Metrics** - Four evaluation metrics are proposed: root - node hierarchy metric ($M_r$), origin - of - coordinates hierarchy metric ($M_o$), parent - node hierarchy metric ($M_p$), and sibling - node hierarchy metric ($M_b$). - These metrics, based on the distance relationships between nodes, quantify the influence of HRC under different factors. 2. **Hierarchical Structure Description** - Two metric indicators are proposed: horizontal - level difference (IB) and vertical - degree distribution (ID), which are used to describe and generate different hierarchical structures. - By controlling the probability of generating child nodes and the number of child nodes, hierarchical structures with different IB and ID are generated. 3. **Pre - training Strategies** - Three pre - training strategies are proposed: directly applying the encoder that enhances HRC (EfD), direct application without freezing parameters (ED), and placing the encoder that enhances HRC in front of the downstream - task encoder (EfED). - The experimental results show that these strategies can significantly improve the performance of downstream tasks in some cases. ### Conclusions Through systematic experiments and analysis, the paper reveals that the effectiveness of HNNs depends not only on the HRC of the hyperbolic space but also on the optimization objectives and hierarchical structures. Through the proposed HRCB and pre - training strategies, the performance of HNNs can be improved to a certain extent. However, it should be noted that excessive enhancement of HRC may reduce performance in some cases. This finding provides important guidance for the research and application of HNNs.

Why are hyperbolic neural networks effective? A study on hierarchical representation capability

Hyperbolic Binary Neural Network

Nested Hyperbolic Spaces for Dimensionality Reduction and Hyperbolic NN Design

Hyperbolic Convolutional Neural Networks

Hyperbolic Deep Neural Networks: A Survey

Hyperbolic Representation Learning: Revisiting and Advancing

Hyperbolic Curvature Graph Neural Network

Fully Hyperbolic Neural Networks

Knowledge Graph Representation Via Hierarchical Hyperbolic Neural Graph Embedding

Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers

Hyperbolic Graph Convolutional Neural Networks

Hyperbolic Temporal Network Embedding

Fully Hyperbolic Convolutional Neural Networks for Computer Vision

Embedding Heterogeneous Information Network in Hyperbolic Spaces

MHCN: A Hyperbolic Neural Network Model for Multi-view Hierarchical Clustering

Hyperbolic Neural Collaborative Recommender

Discrete-time Temporal Network Embedding via Implicit Hierarchical Learning in Hyperbolic Space

Hyperbolic Graph Neural Networks: A Review of Methods and Applications