Abstract:In this paper, we propose novel quaternion activation functions where we modify either the quaternion magnitude or the phase, as an alternative to the commonly used split activation functions. We define criteria that are relevant for quaternion activation functions, and subsequently we propose our novel activation functions based on this analysis. Instead of applying a known activation function like the ReLU or Tanh on the quaternion elements separately, these activation functions consider the quaternion properties and respect the quaternion space $\mathbb{H}$. In particular, all quaternion components are utilized to calculate all output components, carrying out the benefit of the Hamilton product in e.g. the quaternion convolution to the activation functions. The proposed activation functions can be incorporated in arbitrary quaternion valued neural networks trained with gradient descent techniques. We further discuss the derivatives of the proposed activation functions where we observe beneficial properties for the activation functions affecting the phase. Specifically, they prove to be sensitive on basically the whole input range, thus improved gradient flow can be expected. We provide an elaborate experimental evaluation of our proposed quaternion activation functions including comparison with the split ReLU and split Tanh on two image classification tasks using the CIFAR-10 and SVHN dataset. There, especially the quaternion activation functions affecting the phase consistently prove to provide better performance.

Learning Algorithms in Quaternion Neural Networks Using Ghr Calculus

Optimization in Quaternion Dynamic Systems: Gradient, Hessian, and Learning Algorithms.

Constrained Quaternion-Variable Convex Optimization: A Quaternion-Valued Recurrent Neural Network Approach

The HR-Calculus: Enabling Information Processing with Quaternion Algebra

Widely Linear Quaternion Unscented Kalman Filter for Quaternion-Valued Feedforward Neural Network.

A Quaternion-Valued Neural Network Approach to Nonsmooth Nonconvex Constrained Optimization in Quaternion Domain

Improving Quaternion Neural Networks with Quaternionic Activation Functions

Quaternion MLP Neural Networks Based on the Maximum Correntropy Criterion

Incremental Quaternion Random Neural Networks.

Quaternion Filtering Based on Quaternion Involutions and Its Application in Signal Processing

Quaternion recurrent neural network with real-time recurrent learning and maximum correntropy criterion

Comments on "the Quaternion LMS Algorithm for Adaptive Filtering of Hypercomplex Processes".

Performance analysis of gradient neural network exploited for online time-varying quadratic minimization and equality-constrained quadratic programming

Quaternion kernel recursive least-squares algorithm

A Neurodynamic Approach to Nonsmooth Quaternion Distributed Convex Optimization With Inequality and Affine Equality Constraints

Quaternion Matrix Optimization and The Underlying Calculus

Properties and Applications of a Restricted HR Gradient Operator

Decomposition Approach to the Stability of Recurrent Neural Networks with Asynchronous Time Delays in Quaternion Field

Exponential Stability of Quaternion-Valued Neural Networks with Proportional Delays and Linear Threshold Neurons

One-Layer Neural Network for Nonlinear Convex Programming with Linear Constraints