Learning Structured Representations with Hyperbolic Embeddings

Aditya Sinha,Siqi Zeng,Makoto Yamada,Han Zhao

2024-12-02

Abstract:Most real-world datasets consist of a natural hierarchy between classes or an inherent label structure that is either already available or can be constructed cheaply. However, most existing representation learning methods ignore this hierarchy, treating labels as permutation invariant. Recent work [Zeng et al., 2022] proposes using this structured information explicitly, but the use of Euclidean distance may distort the underlying semantic context [Chen et al., 2013]. In this work, motivated by the advantage of hyperbolic spaces in modeling hierarchical relationships, we propose a novel approach HypStructure: a Hyperbolic Structured regularization approach to accurately embed the label hierarchy into the learned representations. HypStructure is a simple-yet-effective regularizer that consists of a hyperbolic tree-based representation loss along with a centering loss, and can be combined with any standard task loss to learn hierarchy-informed features. Extensive experiments on several large-scale vision benchmarks demonstrate the efficacy of HypStructure in reducing distortion and boosting generalization performance especially under low dimensional scenarios. For a better understanding of structured representation, we perform eigenvalue analysis that links the representation geometry to improved Out-of-Distribution (OOD) detection performance seen empirically. The code is available at \url{<a class="link-external link-https" href="https://github.com/uiuctml/HypStructure" rel="external noopener nofollow">this https URL</a>}.

Machine Learning,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper attempts to solve the problem of how to effectively utilize the natural hierarchical structure information between classes to improve representation learning in deep learning. Specifically, most existing representation learning methods ignore the hierarchical relationships between classes and treat labels as permutation - invariant. However, the classes in many real - world datasets have a natural hierarchical structure, which can be cheaply constructed or inferred. Some recent works have attempted to explicitly use this structural information, but using the Euclidean distance may distort the underlying semantic context.

To overcome these problems, the authors propose a new method named **HypStructure**, which accurately captures these relationships by embedding the label hierarchy in the hyperbolic space. HypStructure is a simple regularization method consisting of a hyperbolic - tree - based representation loss and a centering loss. It can be combined with any standard task loss to learn features rich in hierarchical information.

### Main contributions

1. **Proposing the HypStructure method**: This method performs well in supervised hierarchical classification tasks and is effective both when trained from scratch and when fine - tuned under resource - constrained conditions.
2. **Evaluating the learned representations**: The authors evaluate the properties of the learned representations both qualitatively and quantitatively, showing that HypStructure, as a regularizer, not only improves performance but also makes the representations more interpretable and tree - like.
3. **Low - dimensional representation ability**: Research shows that training with HypStructure can learn extremely low - dimensional representations, whose distortion values are even lower than those of high - dimensional Euclidean representations.
4. **OOD detection performance**: The authors find that representations with a hierarchical structure are beneficial not only for in - distribution (ID) classification tasks but also for out - of - distribution (OOD) detection tasks. Experiments prove that the ID representations learned using HypStructure improve OOD detection performance on 9 real - world OOD datasets without sacrificing ID accuracy.
5. **Feature spectrum analysis**: Inspired by the improvement in OOD detection performance, the authors provide a feature spectrum analysis of the structured representations, thus better understanding the behavior of the structured representations.

### Method overview

The HypStructure method mainly consists of two parts:

1. **Hyperbolic Cophenetic Correlation Coefficient Loss (HypCPCC)**: It extends the ℓ2 - CPCC method of Zeng et al. [104] and applies it to the hyperbolic space. The specific steps include:
- Mapping Euclidean vectors to the Poincaré space.
- Calculating class prototypes.
- Calculating CPCC using the Poincaré distance.

2. **Hyperbolic Centering Loss (HypCenter)**: Inspired by Sarkar's low - distortion construction [75], this loss term places the root node of the tree at the center of the Poincaré disk and the child nodes in the area close to the boundary. This is achieved by minimizing the norm of the hyperbolic representation of the root node.

### Experimental results

The authors conducted extensive experiments on multiple large - scale image benchmark datasets, including CIFAR10, CIFAR100, and ImageNet100. The experimental results show that:

- **Hierarchical information distortion**: HypStructure significantly reduces the distortion of hierarchical information, with a maximum reduction of 59.4%.
- **Classification performance**: HypStructure achieves significant performance improvements in both fine - grained and coarse - grained classification tasks, with a maximum improvement of 2.2%.
- **OOD detection**: HypStructure performs excellently in OOD detection tasks, with an average AUROC improvement of 2%.

### Conclusion

By embedding the label hierarchy in the hyperbolic space, HypStructure effectively solves the problem that existing representation learning methods ignore hierarchical information. This method not only performs well in classification tasks but also shows strong performance in OOD detection tasks.