Abstract:Hypercomplex algebras have recently been gaining prominence in the field of deep learning owing to the advantages of their division algebras over real vector spaces and their superior results when dealing with multidimensional signals in real-world 3D and 4D paradigms. This paper provides a foundational framework that serves as a roadmap for understanding why hypercomplex deep learning methods are so successful and how their potential can be exploited. Such a theoretical framework is described in terms of inductive bias, i.e., a collection of assumptions, properties, and constraints that are built into training algorithms to guide their learning process toward more efficient and accurate solutions. We show that it is possible to derive specific inductive biases in the hypercomplex domains, which extend complex numbers to encompass diverse numbers and data structures. These biases prove effective in managing the distinctive properties of these domains, as well as the complex structures of multidimensional and multimodal signals. This novel perspective for hypercomplex deep learning promises to both demystify this class of methods and clarify their potential, under a unifying framework, and in this way promotes hypercomplex models as viable alternatives to traditional real-valued deep learning for multidimensional signal processing.

What problem does this paper attempt to address?

This paper discusses the application of hypercomplex numbers in deep learning and how to utilize specific inductive biases in the hypercomplex domain to enhance the performance of models. Hypercomplex algebra has advantages in dealing with multidimensional signals and can better capture complex structures. The paper proposes a theoretical framework that explains why hypercomplex deep learning methods are successful and demonstrates how to use these biases to handle multidimensional and multimodal signals. Inductive bias refers to the built-in assumptions, properties, and constraints in training algorithms that guide the learning process towards more efficient and accurate solutions. The paper points out that specific inductive biases can be derived in the hypercomplex domain, which are applicable to handle the unique properties of these domains and the complex structures of multidimensional signals. This includes advantages such as saving computational costs, improving generalization capability to unknown data, compact representation of real-valued multidimensional signals, and simulating transformations in high-dimensional spaces. The paper also emphasizes the necessity of defining deep learning models in the hypercomplex domain to reduce neglect of coupling variables in multidimensional input signals. By introducing hypercomplex numbers, the models can better handle spatial relationships and complex rotations. Additionally, the paper analyzes a range of inductive biases that can be beneficial for hypercomplex models and how parameterized hypercomplex models enforce these biases, extending these advantages to a wide range of multidimensional signal processing tasks. Overall, this paper aims to unveil the methods of hypercomplex deep learning, clarify its potential advantages, and advocate for hypercomplex models as viable alternatives to traditional real-valued deep learning, especially in the field of multidimensional signal processing.

Demystifying the Hypercomplex: Inductive Biases in Hypercomplex Deep Learning

Towards Explaining Hypercomplex Neural Networks

Multi-Excitation Projective Simulation with a Many-Body Physics Inspired Inductive Bias

Deep Complex Networks

Theoretical Analysis of Inductive Biases in Deep Convolutional Networks

Inductive Bias of Deep Convolutional Networks through Pooling Geometry

Towards Exact Computation of Inductive Bias

Hyperbolic Convolutional Neural Networks

Hyperbolic Deep Neural Networks: A Survey

Fantastic Biases (What are They) and Where to Find Them

Combinatorial Complexes: Bridging the Gap Between Cell Complexes and Hypergraphs

Gluing Neural Networks Symbolically Through Hyperdimensional Computing

Hyperdimensional computing: A fast, robust, and interpretable paradigm for biological data

A Relational Inductive Bias for Dimensional Abstraction in Neural Networks

A Theoretical Perspective on Hyperdimensional Computing

Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning

The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning

Hyperdimensional computing: a fast, robust and interpretable paradigm for biological data

Classification using Hyperdimensional Computing: A Review